1.11 Summary Stats: Establishing a Baseline
Before we can meaningfully interpret any statistic that we encounter, we need to first acquire some contextual information, starting with the way that some specific value compares to an average. This context often comes to us with the help of exploratory plots, domain knowledge familiarity, and functions like describe() from pandas.
Before seeing the table above, how would you react if a Lobster Land manager told you that the park’s gross revenue for some particular day was USD $165,000? Most likely, you would have no idea how to react – should you congratulate him, or console him?
However, after seeing the describe() results shown above, you might (quite reasonably) surmise that the manager would be thrilled to see this daily revenue figure.
Uber drivers and movies are both rated on five-star scales, with 5 representing the best rating, and 1 representing the worst. While that basic domain knowledge is a helpful start for someone analyzing movie and Uber driver ratings, it does not enable any meaningful interpretation of either type of value – what is a “good” Uber driver rating? And what is a “good” movie rating? Are these the same, or different?
As it turns out, Uber driver ratings are considerably left-skewed. Drivers are expected to maintain ratings of 4.6 or higher. A rider who is brand new to the service could (understandably) be excited to see that a 4.35-star driver is on his way to pick him up; however, a more experienced Uber user might cancel the ride and just decide to walk!
Movie ratings, on the other hand, tend to be much more normally-distributed. Even though they are based on the same scale (and even the same symbol – stars), they are simply not comparable to Uber driver ratings. A movie with a 4.35-star rating is most likely an excellent film, and well worth a viewer’s time.
Since there is no universal standard, or baseline, that we can apply to ratings systems, there is no automated, shortcut way to assess a single statistic, offered in isolation. Instead, we must simply embrace the EDA process, and always remember to examine datasets with summaries and plots before we begin to model with it, and/or to draw any meaningful conclusions from it.