The end of forecasting?

Or could we simply be at the end of forecasting’s beginning – with unforeseen potential ahead?

By Thornton May

Researchers at the IT Leadership Academy have studied many soon-to-be-great enterprises. Their analysis shows that those exceptionally performing organizations are in the final stages of early experiments that have profoundly transformed the relationships they have with their customers. In other words, many businesses are undergoing a paradigm shift when it comes to how they think about customers.

Business models appear to be migrating from the make-and-sell mode of the industrialized past, through the sense-and-respond methods of the “Webified” present, and are now shifting toward the anticipate-and-lead ecosystems of the not-so-distant future.

In an environment where the ability to anticipate and lead is a critical determinant of success, it stands to reason that forecasting and forecasters will rise in prominence and importance. However, while forecasting is increasingly viewed as a key ingredient of strategic success, forecasting techniques and practitioners have never been so much under the magnifying glass as they are today. The powers that be are seriously rethinking forecasting.

A millennium-plus of ‘smartness’ comes to an end
From the Dark Ages through the Renaissance and the Age of Reason/Age of Enlightenment, right on up through the Industrial and Information Ages, the history of man has been one of ever-increasing knowledge. A thousand-plus years of eliminating ignorance, getting smarter and solving an ever expanding array of important problems might explain the broadly held belief in the decades leading up to the start of this century that there was not much in this world that was unknowable.

In the face of ever-improving, massively powerful and omni-connected computers, who could blame many for believing that someday real-world systems like the economy, the weather and human health would become truly and totally predictable?
 
Is the future truly unknowable?
The first decade of the 21st century has been a humbling one for forecasters. A new subcategory of management literature has emerged addressing and questioning the extent to which what comes next might be knowable.

For example, Michael Raynor says, “The future is deeply unpredictable.”¹ Nassim Taleb asserts, “Our world is dominated by the extreme, the unknown, and the very improbable … the future will be increasingly less predictable.”² And David Orrell says, “Studies have shown that social forecasting, scientific or otherwise, is about as accurate as random guessing, despite the vast numbers of highly paid experts employed to do it.”³

Following the fall of the Berlin Wall in 1989, Francis Fukuyama, Professor of International Political Economy at the School of Advanced International Studies of the prestigious Johns Hopkins University, published The End of History and the Last Man (New York: Avon Books, 1998). In that work, he boldly prophesied that liberal democracy may constitute the “end point of mankind’s ideological evolution.” He essentially stated that we were at the end of history. Subsequent events in the Middle East, Asia and Russia demonstrated that Fukuyama’s forecast was materially off the mark.

However, Fukuyama’s response to having his forecast proved wrong provides a role model for all of those who would forecast. Rather than retreat from the public realm or move into denial or excuse-mongering, this stellar academic redoubled his efforts to understand uncertainty. His most recent book Blindside: How to Anticipate Forcing Events and Wild Cards in Global Politics (Washington, DC: Brookings Institution Press, 2007) is my favorite treatment of the question of what “knowability” is.

Blindside courageously stares in the face of forecast error and asks why. Fukuyama explains that “the past decade and a half has demonstrated that nothing is as certain as uncertainty … As the famous scatological bumper sticker suggests, bad things happen.”

Business forecasters face tumult – or new opportunity
Those of us in the business intelligence community need to realize that important conversations about what we do and how we do it are taking place at the highest levels of the enterprise today. If we do not involve ourselves in those conversations and compellingly portray the vital role we play in making good things happen and preventing bad things from happening, we may indeed be at the end of forecasting as we know it.

But being at the end of the beginning for forecasting technologies doesn’t mean that our skills have become obsolete. In my last column, I spoke of paradigm shifts – a period of crisis that follows a period of relative normalcy when the old ways of doing things stop working, marking a shift as we search for a new, more effective way to solve problems.

Forecasting may simply be facing such a time of crisis. As old reasons for forecasting and techniques of forecasting become obsolete, those of us who are practitioners need to be open to new ideas and approaches, including: 

• Understanding which situations are most appropriate for forecasting.
• Learning how to minimize volatility and randomness.
• Basing forecasts on marketplace realities, not your best hopes.

Ultimately, these and other lessons can help us from becoming obsolete ourselves.