1.18 Summary Stats: Concentration and Dispersion
A variable’s standard deviation tells us how much, on average, we can expect a variable’s value to differ from its mean.9 When summarizing a numeric variable, it is often helpful to express mean along with standard deviation in order to establish a sense of what should be “expected” when working with its values.
To see how standard deviation is calculated, let’s use the following six observed totals of daily soda sales from the Snack Shack at Lobster Land:
| Sodas Sold | |
| Day 1 | 453 |
| Day 2 | 351 |
| Day 3 | 401 |
| Day 4 | 397 |
| Day 5 | 478 |
| Day 6 | 598 |
| Sodas Sold | Mean Soda Sales | Sales – Mean | (Sales-Mean)^2 | |
| Day 1 | 453 | 446.33 | 6.67 | 44.49 |
| Day 2 | 351 | 446.33 | -95.53 | 9125.98 |
| Day 3 | 401 | 446.33 | -45.33 | 2054.81 |
| Day 4 | 397 | 446.33 | -49.33 | 2433.45 |
| Day 5 | 478 | 446.33 | 31.67 | 1002.99 |
| Day 6 | 598 | 446.33 | 151.67 | 23003.79 |
As a next step, we will sum all of the values in the right-most column. That sum is 37,665.51.
37,665.51 divided by the number of records, n, is 37,665.51 / 6 = 6277.585. That value tells us the variance for daily sodas sold. Finally we take the square root of that figure to arrive at the standard deviation: 79.23.
Knowing the standard deviation helps us to contextualize the mean statistic of 446.33. The mean itself can be seen as an “expected” value – all else equal, that’s what we should forecast for any particular day’s soda sales. When we know the standard deviation as well, we can better understand just how unusual some particular value is.
Thankfully, we can do this much more quickly in Python, as shown below. The slight difference in the value shown here is due to rounding (in the by-hand calculation, the figures were rounded to only include two decimal places).
Note that the formula shown above is only used with a population standard deviation. If we are instead using a sample of data, taken from an entire population, we would make one adjustment – we would use n-1 in the denominator, rather than n (all of the examples in this chapter will be based on populations, rather than on samples).
While standard deviations are an essential element of the EDA process, bear in mind that it can be hard to meaningfully compare standard deviations across variables. Variables that are measured in larger units should be expected to have larger standard deviations, even if their relative movement is the same. Thankfully, we can remedy that by using a statistic called the Coefficient of Variation (CV).
The CV is found by dividing a variable’s standard deviation by its mean. CV serves a valuable purpose – it enables us to compare dispersion across multiple variables, which may be expressed in different units, or on different scales.
From the 2021 park data, we can see that Lobster Land’s daily average count of unique visitors is 3758. Let’s imagine that for a 10-day period, we compare the Lobster Land average visitor count to that of Disney’s Magic Kingdom, whose mean daily visitor tally across the entire season is 57,515. 10
| Lobster Land Visitor Total | Magic Kingdom Visitor Total | |
| Day 1 | 3,851 | 58,914 |
| Day 2 | 3,712 | 62,841 |
| Day 3 | 3,816 | 63,312 |
| Day 4 | 4,218 | 54,016 |
| Day 5 | 4,391 | 53,012 |
| Day 6 | 4,815 | 51,124 |
| Day 7 | 3,912 | 57,812 |
| Day 8 | 3,483 | 57,415 |
| Day 9 | 3,801 | 58,912 |
| Day 10 | 3,944 | 61,117 |
When we compare the standard deviations of these two samples, we see that Magic Kingdom has a higher standard deviation, as shown below. However, we can also see that Magic Kingdom’s visitor counts are measured on a higher scale – if the relative movement of each group were the same, then we should expect Magic Kingdom to have a higher standard deviation.
The last two lines of code in the screenshot below show the CV values for each park. Lobster Land’s CV of .09, compared with Magic Kingdom’s CV of .067, shows that Lobster Land’s attendance is actually more subject to variation, relative to the size of its mean.
The skewness of a variable tells us about the relationship between its mean, median, and standard deviation. Knowing how the data points fall within a variable helps us decide how we treat a variable in our analysis. For example, if we knew that a variable was normally distributed, we would be able to apply the empirical rule to our calculation; if an element we needed as a predictor variable in a linear regression model was left or right skewed, we could improve the model’s performance by applying a log transformation to the variable.
While there are multiple methods for calculating skewness, one of the most straightforward approaches is the one shown below:
In the formula above, x-bar is the variable’s mean, while the x with the tilde symbol above it is the variable’s median. The ‘s’ in the denominator is the standard deviation. The standard deviation will never be negative, so this statistic’s sign (positive or negative) is determined by the relationship between the mean and the median.
A perfectly symmetric variable, whose mean and median are the same, would have a skewness of 0.
With a right-skewed, or positively-skewed, variable, the mean is larger than the median. When visualized with a histogram, a right-skewed distribution will have a “tail” going off to the right. Right-skewness is often found with variables that are zero-bound on the low side, but with outliers occurring on the high side. Incomes tend to be right-skewed.
With a left-skewed, or negatively-skewed, variable, the mean is smaller than the median. Left-skewness is observed when the “tail” of the distribution goes off to the left, with the bulk of the data on the right.
Left skew could occur if a professor gives a very easy quiz to a class of 50 students. If nearly all students attain scores of 18, 19, or 20, but a handful of students earned scores of 6 and 7, then we should expect to see a median value that exceeds the mean.
In Python, we can calculate skewness with the skew() function from scipy.stats. In the example below, we will look at the skewness of a set of 15 quiz scores, which were graded on a scale from 1 to 20.
The method used by scipy for calculating skew is detailed in the package documentation. The number shown above is slightly different from the one that you would obtain using the formula shown at the top of this section.
However, the truly essential thing remains unchanged – for data distributions whose mean is greater than their median, we can expect to see positive skewness, or right skew. Conversely, when the median is greater than the mean, we can expect to see negative skewness, or left skew.
9 Alternatively, we could use another metric, Mean Absolute Deviation, which is the arithmetic mean of all absolute differences between measured values and the variable mean. The standard deviation is more sensitive to outliers, and is far more commonly used in statistical practice.
10 This is just a hypothetical example – these are not actual Disney park statistics.