11.1 Limitations of Forecasting

In this chapter, we will explore several methods for forecasting, ranging from the shockingly simple to the devilishly complex.  No matter where they fall along that continuum, they all have something very important in common:  they are all based on the past.  

Outside of tightly-controlled laboratory experiments, in which the risks of chaotic, unpredictable changes to model inputs can be reduced or even eliminated, any real-world phenomenon is subject to exogenous shocks, which is really just a fancy way of saying “unexpected stuff.”  

Exogenous shocks guarantee that there is no forecasting method that can actually predict the future with complete accuracy.  Even if there were such a method, its very existence – once exploited – would reduce its efficacy.  To see why, think about the example below.

Let’s say you built a prediction model that could perfectly predict the next day’s movement of the S&P 500.  In other words, this model could flawlessly indicate not just whether the market would go up or down, but could also pinpoint the exact magnitude of the change.  After testing it out a few times – and then pinching yourself to make sure that you’re not dreaming – you might be tempted to profit from the model.  As your profits increase, so would the scale of your operation.  Soon, the size of your own trades would begin to have a meaningful impact on the market, and whatever edge you had earlier would be eroded.