16.1 Variable Transformations
Variables can be frustrating at times – they don’t always do what we want them to do! At times, however, we can make a variable easier to visualize, or better-suited for a particular model, with a transformation. Common types of transformations include log transformations and square root transformations – however, there are many more ways in which this can be done.
Sometimes, transformations can be useful for visualizing data that contains outliers. To see this in action, let’s take a look at the dist_spend_data.csv dataset. This dataset contains 60 observations, each of which is based on a Lobster Land season pass-holding household. Its two variables are ‘dist’, which shows the distance in miles between the household’s primary address and Lobster Land, and ‘season_spend’ which shows the household’s discretionary spending at the park during a particular year.
When we view a histogram of distance, we see that nearly all distances fall within fewer than 500 miles from Lobster Land, but that there are two huge outliers, whose distances exceed 1000 miles. By default, the histogram shows the full distribution of the data, so it cannot simply ignore these data points – but the resulting plot then contains a lot of empty “real estate” and can make it harder to meaningfully interpret the distribution around the more dense part.
Similar issues arise with the scatterplot shown below. Including all the points forces the graph to stretch its x-axis nearly four times longer than would be the case without the outliers.
Attempting to model this data shows a poor linear fit.
Below, we perform a log transformation on the ‘dist’ variable, converting the original data points to their natural log values. The data now appears to be far better suited for a linear fit than was the case before.
The linear regression model built with the transformed data shows a far better fit now, compared with the situation beforehand.
So what is this really doing to the data? By default, numpy’s log() function takes the natural logarithm of any value passed to it. Replacing any value x with its natural logarithm is like saying “If we used a base of 2.71828, to what exponent would we have to raise that base in order to get back to x?” If that all sounded a bit strange and abstract, let’s see it with a real numeric example:
The most immediate effects that a log transformation will have on most datasets are to condense its overall range and to reduce the impact of outliers.
Keep in mind that log transformations will only work when the values are positive.
Also, depending on your audience, variable transformations could make it harder for you to explain your results.