The Future of Forecasting Software

With advanced technologies, companies are poised to benefit more than ever from a forecasting strategy.

by Michael Gilliland & Michael Leonard, SAS

Forecasting is a serious business, but not all parties to the history of forecasting can be taken seriously. In the past, prophets, oracles, astrologers and psychics provided us with guidance about the future – from when to plant crops to when to go to war. Today, we find no shortage of self-proclaimed "seers" and "futurists" who will gladly dispense guidance for a fee, but we also have access to terabytes of data, powerful computers, statistical software and elaborate acronymed processes. This article looks at where the forecasting software industry may take us in the future.

A lot of intelligent people have worked on the forecasting problem for many years and still have found no "magic formula." Simply put, no software or statistical method can guarantee our forecasts will be as accurate as we desire them to be, and it is implausible that forecasting software will ever be able to guarantee an arbitrary level of accuracy. However, significant progress is taking place in the areas of automation, scalability and the incorporation of structured judgment.

The following are some thoughts on trends in the business forecasting problem and the future direction of forecasting software:

• The range of business forecasting problems is increasing. Forecasting is not just for supply and demand planning anymore.  Elements of forecasting are incorporated in a broad range of business problems, covering every industry. Automotive manufacturers and consumer products companies care about warranty claims and returns. Phone companies and credit card issuers care about the "churn" of their customer base. Lenders care about risk and bad debt provisioning. Airlines care about reservation center and flight crew staffing. Manufacturers care about predictive maintenance in their factories. Good forecasting allows all these kinds of organizations to operate more efficiently – better satisfying their customers, and increasing their profit.
• The scale of business forecasting problems is increasing. One example of this is in retail, where more rigorous forecasting and planning processes and statistical tools are starting to replace (or at least augment) the "art" of merchandising. Retailers wish to know what will sell at which price points, what promotions will be most effective and the best clearance strategy when a product is out of season. These questions all have a basis in forecasting. Add to these revenue optimization issues the more fundamental question of how retailers should stock and replenish their stores, and the challenge is huge. Large retailers have tens of thousands of items, sold in hundreds or thousands of stores. The need for millions of forecasts is not uncommon.
• It is crucial to distinguish the "high-value" forecasts for special attention while automating the "not-as-valuable" forecasts. No organization has the luxury of hiring hundreds, or even thousands, of analysts to individually model and forecast each series. It is important to distinguish the high-value forecasts (such as the $5,000 plasma TV at an electronics manufacturer) from the vast majority of items of lesser importance and apply the appropriate forecasting approach to each. Large-scale automation can be used to forecast the products and locations of lesser importance, while analysts must still have access to sophisticated tools for high-value forecasts. Software must be able to handle both kinds of needs, delivering quality forecasts in an automated mode, while still meeting the requirements of the savvy statistical analyst.
• There is a need to handle the "continuously evolving product." With the proliferation of new products and shorter product life cycles, it may be impossible to obtain as much history as the forecast analyst needs for good model building. In personal computers and consumer electronics, for example, product life cycles are typically three to 18 months. Fashion apparel items will sell for just one season. Software should provide tools for the analyst to "string together" the history of current items with their predecessors to leverage the demand history for like items from the past.
• One-model forecasting is dead. Rather than rudimentary "pick best" selection from a handful of models prespecified by the analyst, forecasting software must be able to consider thousands of models from a variety of model families. Today's computational power permits this. For large-scale forecasting automation to succeed, the system must be able to consider the types of models most appropriate to the types of demand patterns that will be encountered.
• Quantity and quality of data will continue to increase. Perhaps more significant than improvements in statistical forecasting models will be an increase in the kind and amount of data available for use in forecasting. Retail point-of-sale (POS) data is now widely available, making product consumption directly visible to consumer product manufacturers. Web site traffic is another new data point, and several services now provide weather-related data, economic indicators and other types of information that forecasting models can readily use. Given all of the new data, however, it is still necessary for the software to automatically distinguish the useful variables from the extraneous ones.
• Use of structured judgment enhances collaboration. Elaborate forecasting processes, such as Collaborative Planning, Forecasting, and Replenishment (CPFR), have evolved over the past several years. CPFR facilitates the input of new participants – both internal (sales, marketing, finance, operations, etc.) and external (customers and suppliers). Today's software is much better at handling these multiple process inputs, and several vendors now offer special collaborative and sales-and-operations planning modules. However, fundamental questions remain: Do these additional participants add any value by making the forecast better? Or do they just let more bias and politics infiltrate what should be an objective and scientific process? Structured judgment techniques will enhance standard collaboration by providing feedback on process participants, identifying sources of chronic bias and allowing for bias corrections. Simply adding more opportunities for people to "touch" a forecast may make it worse more often than making it better. Forecasting software will need to incorporate the tracking of process steps and participants, identifying the "value added" by these efforts in terms of improved accuracy and reduced bias. The result will be a "lean" forecasting process that has been stripped of all wasted activities, providing a forecast that is as good as one can reasonably expect it to be, as efficiently as possible.

What it all means
While data, software and forecasting methods will continue to improve, what is less clear is how much forecast error will ultimately be reduced. Two contrary forces are evident right now: The tools to forecast are getting better, but the demand patterns to be forecast are frequently getting worse! Smooth, long-running and repeating patterns can be forecast quite well with simple techniques. Wild, erratic, highly volatile and short lifecycle patterns, with longer lead times, are inherently difficult to forecast – even with the most powerful techniques. Unfortunately, the trend is for business forecasters to have to deal with more of the latter type of series. The "forecastability" of a demand pattern then becomes a more important consideration than the means we use to forecast it.

Demand will always have an essentially random component. Although our software may be able to extract the level, trend, seasonality and even a long-term cyclical component from historical demand, some element of "noise" (randomness) will always remain. Ultimately, it is the amount of noise in the demand that determines the upper limit of its forecastability.

In short, we have better data, computers, software and statistical methods than ever before, and these will continue to improve. These improvements let us take on even more difficult forecasting challenges, on a scale that would never have been considered without the advent of large-scale automated techniques. Perhaps the only sure way to eliminate forecast error is to eliminate the need to forecast – by perfecting the flexibility and responsiveness of our demand fulfillment capabilities. Until that happens, those of us in "the world's second oldest profession" should have a busy and exciting future.