Consensus or Individual Forecasts?

Silver goes over the theoretical and practical aspects of forecasting in a wide range of fields, from metereology or earthquake prediction to baseball and – you guessed it – economics. Be warned: economists don’t come out looking very flattered in this book. In spite of increasingly sophisticated models, for all our efforts, economic forecasting has not improved greatly in the past years. In the book, Jan Hatzius, Chief Economist at Goldman Sachs and a very respected professional in the field puts it bluntly – if perhaps a little dramatically: “nobody has a clue”.

The truth of the matter is that, as Hatzius explains and as any economist worth his salt knows, macroeconomic forecasting is very complicated. Anyone claiming differently is either a fool or a liar. Predicting the behaviour of something as complex and enormous as an entire economy is a daunting task for a number of reasons; some are rather evident like the fact that economies are in perpetual change, therefore one cannot always extrapolate behaviours and relationships from past business cycles into the next one. Some are not. For instance, the sheer size of raw economic data available is something that most would consider an advantage, a sort of commodity that would allow us to test models massively in order to reach increasingly accurate results. Instead, the thousands of economic indicators available produce a vast amount of statistical noise, making the establishment of meaningful relations of causation between variables a serious challenge.

So is this to say there is no hope to obtain success when trying to forecast economic variables? Not exactly. Individual forecasts can and will achieve a certain level of success, but they can be improved upon. How? This is when consensus forecasts kick in.

My friends at FocusEconomics have been publishing consensus forecasts reports for over 15 years, so they have a massive database of economic projections that comes in very handy in order to illustrate the point.

The charts above show forecasts for GDP growth for the four largest economies in the Euro area. These forecasts were collected during March 2012 (and published at the start of April, hence the chart notes), exactly one year before the publication of the final 2012 GDP figures for those economies. This period is an arbitrary choice made by me, but a one-year horizon can generally be considered a reasonable timeframe to evaluate the accuracy of an economic forecast.

The blue dots represent the individual forecasts made by banks, financial institutions and research institutes across the world, while the red dot at the right end represents the consensus forecast for each country. The green line marks the final GDP result published one year later: -1.4% GDP growth in Spain, +0.7% in Germany, 0.0% in France and -2.4% in Italy.

These charts allow us to evaluate the effectiveness of the consensus approach quickly and intuitively. Spain and Germany come up as nothing short of a smashing success, with the consensus forecast accurately predicting GDP growth in both countries. The case of Spain is particularly compelling, as not a single panelist correctly predicted the final outcome, yet the consensus forecast was right on the money.

France shows a more typical scenario, where the consensus forecast is beat by some of the panelists but manages to be more accurate than most. Finally, Italy presents us with a case of a deeper-than-expected downturn, where the final GDP result overshoots panelists expectations, including the consensus forecast (arithmetically, it couldn’t be any other way).

Thus, a quick overlook already shows that the consensus forecast approach appears to be more reliable than a large number of individual projections, with a few exceptions in the overall sample. However, it is when we apply a standard statistical procedure to evaluate the accuracy of those forecasts that the results are extraordinarily revealing. Calculating the root mean square error (RMSE) – a frequently used system to calculate prediction errors in statistics – we obtain that the aggregate forecast is a whopping 98% more accurate than the average individual forecast in the case of Germany and 92% more accurate in the case of Spain. These extraordinary results, however, are the product of the consensus forecast predicting GDP growth with pinpoint accuracy in both those countries, a result that unfortunately is not all that frequent, therefore it was to be expected that calculations would yield an enormous difference against the individual forecasts.

For France, the consensus forecast is 52% more accurate than the average individual one, still an excellent result. However, it is the Italian case that provides possibly the most revealing result of all. As we saw before, Italy suffered a much larger contraction in GDP than anticipated by the market, therefore the consensus forecast fell quite far from the actual outcome and was actually beat by more than half of the panelists. You could therefore think that individual forecasts beat the aggregate one in this case. You would be wrong. When applying RMSE, the consensus forecast still was 7% more accurate than the average individual one. And this proves the central point of this thesis: the consensus forecast will not correctly predict the final outcome each time; the consensus forecast will not beat all individual panelists each time – in fact a certain number of individual forecasts will beat the aggregate one most of the time – BUT the consensus forecast will be more accurate than the average individual projection. Perhaps the consensus forecast won’t be able to predict the next global crisis, but the point is, it will still be able to give you a better projection than your average individual forecast.

Admittedly, my sample is limited to only four countries, one economic variable and one time frame, so those specific error levels cannot be extrapolated to a wide scale (for what it’s worth, I later run the same test for 2012 GDP projections in another 10 Euro area countries and found the consensus forecast to be on average 29% more accurate than the individual ones). Still, it’s a small sample for projections limited to one year, but whether these specific error numbers are significant enough or not, the key issue is that empirical studies show that the phenomenon of aggregate forecasts beating individual ones appears to happen in nearly every field under study.

As I mentioned before, a certain number of individual forecasts will likely beat the consensus any given time, so one may be tempted to try to choose one among those instead in order to find guidance for future results. The problem with this approach is that it has been widely documented that the past accuracy of a forecaster bears almost no weight regarding its future performance. In other words, if you decide to trust someone else’s view based on getting it right once, you’ll do so at your own risk.

Facing an uncertain, unknown future, the best you can do is try to minimize the risk and reduce the margin for error, and that’s where the consensus forecast is a reliable tool. Some of the differences between forecasts and the actual outcome that we saw before may seem trivial to the untrained eye, but, for instance, in a one trillion euro economy like Spain, a 1% error in predicting GDP growth translates into a sizable 10 billon euro difference. Whether you are a policy maker planning the government’s budget or a financial officer attempting to anticipate future demand for a product, wouldn’t you rather be on the right end of that projection?

Ángel Talavera is an economist based in London who specializes in macroeconomic analysis and country risk. Any views or opinions in this entry are solely those of the author and do not necessarily represent those of the company.