Cheaper than printing it out: buy the paperback book.

Out of Control
Chapter 22: PREDICTION MACHINERY

If silicon chips are enough of a crystal ball to help steer a superarmy war, and algorithms coursing through small computers are enough predictive technology to outguess the stock market, then why not reconfigure a supercomputer to predict the rest of the world? If human society is just a large distributed system of agents and machines, why not construct an apparatus to forecast its future?

Even a cursory study of past predictions shows why not. On the whole, cultural predictions historically have been worse than random guesses. Old books are a graveyard of prophesied futures that never came to pass. A few prophecies hit the bullseye, but there is no way to discern beforehand the rare right one from the plentiful wrong ones. Since predictions are so often wrong, and since believing erroneous predictions is so tempting and so misleading, some professional futurists avoid predictions altogether on principle. To emphasize the corrupting unreliability of trying to prophesy, these futurists prefer to state their prejudice in deliberate exaggeration: "All predictions are wrong."

They have a point. So few long-term predictions prove correct that statistically they are all wrong. Yet, by the same statistical measure, so many short term predictions are right, that all short-term predictions are right.

There is nothing more certain about a complex system than to say it will be just like it is now a moment later. This observation is nearly a truism. Systems are things that keep persisting; so it is only tautological that from one moment to the next a system -- even a living thing -- doesn't change much. An oak tree, the post office, and my Macintosh hardly change at all from one day to the next. I offer an easily guaranteed short-term prediction for complex things anywhere: tomorrow will be mostly like today.

Equally true is the cliché that things occasionally do change from one day to the next. But can these immediate alterations be predicted? And if they can, could you stack up a series of predictable short-term changes into a probable medium-range trend?

Yes. While long-range predictions will remain essentially unpredictable, short range predictions for complex systems are not only possible, they are essential. Furthermore, some types of mid-range predictions are quite feasible, and becoming more so. For reasons I will explain below, the human ability to forecast aspects of our society, economy, and technology will steadily increase despite the Alice-in-Wonderland strangeness that dependable predictions will have upon present actions.

We have the technology now to forecast many social phenomena, if we can catch them at the right moment. I follow the work of Theodore Modis, whose 1992 book, Predictions, nicely sums up the case for utility and believability of predictions. Modis addresses three types of found order in the greater web of human interactions. Each variety forms a pocket of predictability at certain times. He applies his research to the domain of economics, social infrastructure, and technology, but I believe his findings apply to organic systems as well. The three pockets of Modis: Invariants, Growth Curves, Cyclic Waves.

Invariants. The natural and unconscious tendency for all organisms to optimize their behavior instills in that behavior "invariants" that change very little over time. Humans in particular are certified optimizers. Twenty-four hours of time per day is an absolute invariant, so over decades people, on average, tend to spend a remarkably constant amount of time on such chores as cooking, traveling, cleaning -- although the distance or what they accomplish during that time might change. If new activities (say 0201483408 flight instead of walking) are reformulated into elemental dimensions for analysis (how much time is spent in daily moving), the new behaviors often exhibit a continuous pattern with the old that can be extrapolated (and predicted) into the future. Instead of walking a half hour to work, you now drive a half hour to work. In the future, you may fly a half hour to work. Marketplace pressures for efficiency are so relentless and unforgiving that they inevitably push human-made systems in a single (predictable) direction toward optimization. Tracing an invariant optimization point can often alert us to a clean pocket of predictability. For instance, improvement in mechanical efficiency is very slow. No system is yet over 50 percent efficient. A projected system operating on 45 percent efficiency is possible, but one that requires 55 percent is not. Therefore one can safely make a short-term prediction about fuel efficiency.

Growth Curves. The larger, more layered, more decentralized a system is, the more it takes on aspects of organic growth. Growing things share several universal characteristics. Among them are a lifespan that can be plotted as an S-shaped curve: slow birth, steep growth, slow decline. The worldwide production of cars per year or the lifetime production of symphonies composed by Mozart both fit an S-curve with great precision. "The predictive power of S-curves is neither magical nor worthless," writes Modis. "What is hidden under the graceful shape of the S-curve is that fact that natural growth obeys a strict law." This law says that the shape of the ending is symmetrical to the shape of the beginning. The law is based on empirical observations of thousands of biological and institutional life histories. The law is closely related to the natural distribution of complex things as expressed in a bell curve. Growth is extremely sensitive to initial conditions; the first data points on a growth curve are almost meaningless. But once a phenomenon is on a roll, a numerical snapshot of its history can be taken and flipped over to predict the phenomenon's eventual limits and demise. One can extract from the curve a cross-over point with a competing system, or a "ceiling" and a date when the ceiling essentially flattens out. Not every system exhibits a smooth S-curve lifespan; but a remarkable variety and number do. Modis believes that more things adhere to the laws of growth then we suspect. If such growing systems are examined at the right time (midway in their history), then the presence of local order -- summed up by the S-curve law -- affords yet another pocket of predictability.

Cyclic Waves. The apparent complex behavior of a system is partly a reflection of the complex structure of the system's environment. This was pointed out over 30 years ago by Herbert Simon, who used the journey of an ant over the ground as an illustration. The ant's jig-jagging path across the soil reflected not the ant's complex locomotion but the complex structure of its environment. According to Modis, cyclic phenomenon in nature can infuse a cyclic flavor to systems running within it. Modis is intrigued by the 56-year economic cycles discovered by economist N. D. Kondratieff. In addition to Kondratieff's economic waves, Modis adds similar 56-year cycles in scientific advances described by himself, and 56-year cycles in infrastructure replacement studied by Arnulf Grubler. The causes of these apparent waves have been hypothesized by various other authors as coming from 56-year lunar cycles, or every fifth 11-year sunspot cycle, or even from the every-other cycle of human generations -- as each 28-year generational cohort swings away from the work of its parental cohort. Modis argues that primary environmental cycles trigger many secondary and tertiary internal cycles in their wake. Seekers who uncover any fragments of these cycles can use them to predict pockets of behavior.

Together, these three modes of prediction suggest that at certain moments of heightened visibility, the invisible pattern of order becomes clear to those paying attention. Like the next beat of a drum, its future can almost be heard. A moment later, the pattern is gone, muddied and overwritten by noise. Pockets of prediction won't keep away big surprises. But local predictability does point to methods that can be improved, deepened, and lengthened into bigger things.

The long odds against successful big predictions haven't discouraged hordes of amateur and full-time financial chartists attempting to extract longwave patterns from past stock market prices. Any external cyclic behavior is fair game for a chartist: the length of women's hemlines, the age of presidents, the price of eggs. Chartists are forever chasing the mythical "leading indicator" that will predict the destiny of stock prices as a number they can bet on. For many years chartists were ridiculed for their vaguely numerological approach. But in recent years academics such as Richard J. Sweeney and Blake LeBaron have shown that chartist methods often do work. A chartist's technical rule can be stunningly simple: "If the market has been going up for a while, bet that it will continue to go up. If it's on a downward trend, bet it will continue downward." Such a rule reduces the high dimensionality of a complex market into to the low dimensionality of this simple two-part rule. In general, this kind of pattern-seeking works. The "up-up, down-down" pattern performs better than random chance, and thus better than the average investor. Since stasis is the most predictable thing about a system this pattern of order should not come as a surprise, even though it does.

In opposition to chartism, other financial forecasters rely on the "fundamentals" of the market in an effort to predict it. Fundamentalists, as they are called, attempt to understand the driving forces, the underlying dynamics, and the fundamental conditions of a complex phenomenon. In short they seek a theory: f=ma.

Chartists, on the other hand, seek a pattern from the data without concern for whether they understand why the pattern is there. If there is order in the universe, then somewhere, somehow, all complexity will disclose -- at least momentarily -- order that reveals its future path. One merely needs to learn what signals to disregard as noise. Chartism is organized induction in Doyne Farmer's mode. Farmer admits that he and his fellows at the Prediction Company are "statistically rigorous chartists."

In another fifty years, computerized induction, algorithmic chartism, and pocket predictionism will be respectable human endeavors. Forecasting stock markets will remain an oddball case because, more than other systems, stock markets are built out of expectations. In an expectation game, accurate predictions offer no opportunity for money-making if everyone shares the prediction. All the Prediction Company can really own is lead time. As soon as Farmer's group makes much money exploiting a pocket of predictability, others will rush in, somewhat clouding the pattern, but mostly leveling the opportunity to make any money. In a stock market, success stirs up strong self-canceling feedback currents. In other systems, such as a growing network, or an expanding corporation, anticipatory feedback is not self- canceling. Ordinarily, feedback is self-governing.

continue...