The Predictors by
Thomas A. Bass
Henry Holt, 1999, $25.00
Review score: **** out of *****
We attempt to impose order in the world around us. We build social structures and develop and use technology. The world around us is viewed through a veil of order, that helps hide the fact that the universe is a strange and chaotic place. Our view of order is an illusion.
Small events can produce large changes in complex dynamical systems (e.g., chaotic systems). Sometimes small events are dramatic. While rushing to catch the trolley someone gets hit by a car. Their life is forever altered in an instant. Other changes are more subtle. In my case, reading Thomas Bass' book The Predictors was a catalyst for a change that had been waiting to happen. The Predictors changed the lives of my wife and I.
The Predictors tells the story of The Prediction Company which was founded by Doyne Farmer and Norman Packard. The Prediction Company is located in Santa Fe, New Mexico, an area that has been somewhat hopefully labeled the Silicon Mesa (see also InfoMesa links from www.bioreason.com). The company develops financial trading software that is used by the United Bank of Switzerland (UBS AG). The roots of The Prediction Company lie in chaotic dynamical systems, which exist in both the physical and economic worlds.
Science in the twentieth and twenty-first centuries is based on the recognition that the universe is not as orderly as we might wish. Heisenberg uncertainty and quantum mechanics include limits on what we can know. Physics, chemistry and biology have all undergone revolutions in their structure and thought. Theories that were described with linear equations have been replaced with more exact models that use non-linear dynamic equations that cannot be solved without a computer.
The intellectual trajectories of Farmer and Packard has mirrored the evolution of physics. Thomas Bass went to school with Farmer and Packard when they were graduate students at the University of California at Santa Cruz. Bass' earlier book, The Eudaemonic Pie (now out of print), tells the story of an earlier Farmer and Packard venture to build a miniaturized computer that could be used to beat casino roulette. Roulette is a complicated Newtonian system, consisting of the moving ball, friction, gravity, metal deflectors and the moving wheel. Farmer, Packard and their colleagues built the roulette computer and data entry system into clothing and shoes and took it into Nevada casinos to give them an edge over roulette. Packard and Farmer seem to have always had a strong entrepreneurial streak and the roulette venture was structured as a company, although a very egalitarian one. From the physics of roulette, Farmer and Packard moved on to non-linear dynamics, chaos and the founding of the Prediction Company.
The move from theoretical physics to a market modeling software is not an obvious path. Like physical systems, markets are highly chaotic and most markets are best modeled by dynamical systems. The realization that this is the case has been relatively new. Until recently the "dismal science", economics, has escaped the modern revolution of uncertainty and non-linear dynamics. Economics has resisted change and has remained in the Victorian world until the latter part of the twentieth century.
In September 1987 Doyne Farmer and Norman Packard took part in a conference titled The Economy as an Evolving Complex System in Santa Fe, organized by the Santa Fe Institute. This conference brought together physicists, chaos theorists, biologists, computer scientists and economists. Most of the scientists attending the conference viewed the universe through the filter of complex systems and uncertainty. This was not a common view among economists. In The Predictors Thomas Bass describes this conference:
When they were published in 1988, the conference proceedings described the event as "a serious dialogue between disparate, and at times hostile, communities of scholars." The physicists thought the economists were bluffing, and the economists thought the physicists were suffering from the Tarzan syndrome. The physicists harried the economists in the question-and-answer periods, and the economists got huffy in defending their discipline. In spite of the heat that was generated with the occasional flashes of light, Doyne and Norman came away from the meeting excited about the problems in economics and convinced they had some fresh ideas on how to solve them.
For most of the latter half of the twentieth century the central theme of economic thought has been market equilibrium. In this theoretical view of markets, all market "actors" have perfect knowledge of the market and interpret this knowledge in similar fashions, working to maximize their profit. Any mispricing is immediately acted on by the market players, returning market price to its proper value. One of the implications of this view of markets, beloved by the editorial page of The Wall Street Journal, is that the market always settles on the best solution. As Thomas Bass puts it
The theory [of market equilibrium] celebrates Adam Smith's claim that the "invisible hand" of free market enterprise will allocate resources efficiently, even in the absence of government intervention.
Economists who have suggested that the free market does not always produce the best of all possible worlds, as Brian Arthur has, have not always been warmly received by their peers. Conservatives dislike this view, because they see it as opening the door to the evils of government regulation.
There is an obvious attraction to perfect market theory. Securities and commodities regulations in the United States make it illegal to trade on inside information, which helps assure that market players have access to similar information. With the Internet and modern telecommunications, market information is rapidly delivered to the market players. A "level playing field" and rapid information dissemination are the prerequisites for efficient markets that reach equilibrium in academic theory. One of the arguments for market equilibrium has been the observation that few people seem to be able to win "excess profits" in markets. For example, in The Wall Street Journal's column which pits mutual fund stock pickers against stocks chosen by throwing darts at the stock market listings, the darts frequently win. Academic theory tells us that markets may rise and fall, but the best we can hope is to be buoyed along with the tides. As a result, we should only invest in index funds. Anyone who buys actively managed mutual funds, according to this theory, is a sucker, paying high fees for a service (better market performance) that is never delivered. This is the view eloquently expressed in A Random Walk Down Wall Street by Burton G. Malkiel, an economics professor at Princeton. Stock prices move, but their movement is "a random walk" and can't be predicted.
Although the theory of market equilibrium is popular in academia and among those selling index funds, it is less popular with traders on Wall Street and in the Chicago commodities markets. Instead of being the efficient markets of academic economics, markets exhibit speculative bubbles and crashes. Markets are described in financial publications as if they were moody individuals in the grips of euphoria, depression, calm and worry. Markets may fluctuate around an equilibrium line, but few markets seem to reach ever reach equilibrium.
The moods and ballistic behavior of market price have suggested patterns to many people. Technical traders (sometimes called chartists) follow market movements attempting to derive trading rules from market patterns. The language used to describe price movement and the superstitious behavior of some chartists make them easy targets for ridicule. However, some chartists make money and they are employed by Wall Street trading firms.
In the last forty years people have started to take a more scientific approach to looking at the movement of market prices. This has included the application of chaos theory and fractals. For example, Benoit Mandelbrot published Fractals and Scaling in Finance: Discontinuity, Concentration, Risk, a collection of his papers that look at market price patterns from the point of view of fractal mathematics.
Chaos theory and dynamical systems started Doyne Farmer and Norman Packard on the course from physics to economics and financial markets. The path to founding The Prediction Company was not immediate. Along with work on artificial life and self organizing systems, Farmer and Packard authored papers on markets. Both Packard and Farmer were very successful in the academic research communities. However, the pressure to constantly write research proposals wears on any senior researcher. Their entrepreneurial streak and the dream of the freedom that money can buy seems to have given them the final push away from theory into commerce.
Ironically, markets do not seem to have attractors as chaotic systems do. In fact, chaos theory does not seem to be a good model for markets. Instead, markets resemble the predator-prey interactions described by the Lotka-Volterra equations, although markets are far more complex. According to The Predictors, Farmer, Packard and colleagues worked for about three years before creating models that yielded consistent profits. Models, even models with predictive value in markets, are not enough to create the cash machine that the founders of The Prediction Company dreamed of.
The Prediction Company was founded by physicists, who are notorious for writing poor software. This is understandable, since software engineering is a huge field of study itself. Few people are able to master math, physics, quantitative finance and software engineering. There is a huge amount of nuts and bolts work that goes into creating a trading system that can turn a promising trading model into a trading system that can make money. Farmer and Packard were wise enough to realize that they needed to hire professional software engineers and today about half of the technical staff at Prediction are software engineers.
The path from the egalitarian and hierarchy free "company" dedicated to constructing the software and hardware to profit from roulette described in The Eudaemonic Pie to the company described in The Predictors is fascinating. Someone once described The Predictors as the story of the loss of innocence of Doyne Farmer and Norman Packard. From reading the book and interviewing at The Prediction Company, the egalitarian ideals of the company still seem to be there, although they have been modified by the realities of running a small business. The book never directly describes the changes Farmer and Packard went through. There are hints, but nothing definite. Norman Packard is now the CEO of The Prediction Company. The company does have heirarchy, as any successful company does. How does Norman feel about all this? The book gives the impression he simply slipped into this role. My experience is that management is one of the hardest tasks next to being a parent. So I find it unlikely that becoming a CEO was such an effortless process. There is a certain breathless quality to The Predictors. Everyone involved with the Prediction Company is described as brilliant. But even in the case of the central characters we never see a picture of the complex personality. Politics plays a part in any group of people larger than one. Yet in The Predictors there are only a few brief references to any conflict or interpersonal difficulties.
After reading The Predictors one of the things that fascinated me about The Prediction Company was that academic theory says that what they are doing (building models with predictive value in markets) is impossible (or based on blind luck, which will soon run out). But they seem to have succeeded in making money for a number of years.
Before reading The Predictors, I blindly followed the theory of market equilibrium. After reading The Predictors and a number of books and articles on markets as dynamic systems, I don't hold this view any longer. But the ability to build a model of a market based on a dynamical system does not necessarily mean that that model has any predictive value. This is the secret art of those who build quantitative trading systems. Little if any information is available on this in the academic literature.
As I mentioned at the start of this review, reading The Predictors changed my life. It was a small event that came at the right time to trigger large changes.
One day I got a call one day from a Microsoft recruiter who was recruiting people for the Microsoft compiler group. With some reluctance, I agreed to go up to Redmond to interview. It was hard to imagine moving out of Silicon Valley. I had just finished The Predictors and I was fascinated by the hints it contained about the work being done at The Prediction Company. I took a look at the Prediction Company web page and noticed that they were looking for software engineers. At first I was put off by Thomas Bass' description of everyone involved with The Prediction Company as effortlessly brilliant. I consider myself to be a top notch software engineer, but I'm not a world class scientist. As I started to think about the possibility of moving out of Silicon Valley, I thought I might as well send Prediction a resume.
The interview at Microsoft was very unpleasant. Microsoft has an excellent engineering staff and at least when it comes to software development tools, the products are great. But their reputation for arrogance is entirely deserved. I spent two days interviewing in Redmond with two different groups. After the first day I felt like flying home. The interview process consists of endlessly working through problems. In many cases this involved writing code on a white board. Writing even the simplest algorithms under pressure can be very intimidating. I was asked little about the software projects I have worked on or the compilers I have implemented. Apparently everyone Microsoft hires is expected to think rapidly under pressure.
The Prediction Company was, in some sense, the anti-Microsoft. They asked about what I had done, in depth. I was treated very well and it was a positive experience. As it turned out, the staff of The Prediction Company is very bright, but not everyone there is a world class scientist or a genius. The Prediction Company offered me a job and I accepted.
Another view of The Predictors after spending two years at Prediction Company.
Cracking Wall Street, by Kevin Kelly, Wired Magazine. A shorter version of this appeared in Chapter 22 of Kevin Kelly's book Out of Control
Market Force, Ecology and Evolution by J. Doyne Farmer
Modeling Markets by Cosma Rohilla Shalizi, a draft of an article published in the Santa Fe Institute Bulletin vol. 15, No. 1, Winter 2000
This is a draft of a profile of Doyne Farmer, by Cosma Shalizi, a researcher at the Santa Fe Institute. This is the only part of this book review that I've updated since I wrote it (I wanted to put this reference somewhere, and this seemed like a good place).
My book review of Increasing Returns and Path Dependence in the Economy by W. Brian Arthur
Some vignettes on quantative finance and market prediction in Way Off Wall Street, Fortune Magazine
This book review was written while I was still living in Silicon Valley and employed by Quickturn Design Systems. When this review was written, I had interviewed at Prediction and accepted their job offer. As far as I know, I learned nothing that was proprietary during the job interview. I've read The Eudaemonic Pie, The Predictors and a number of articles by Doyne Farmer and others who follow the "Santa Fe" approach to economics. This review is drawn from that material and does not contain any "inside information". I still know nothing about market prediction and I will not update this review with any information I learn while employed at The Prediction Company (so don't send me e-mail asking about this). I own the bearcave.com domain and only my views are reflected here, not those of The Prediction Company or any of its other employees.
Ian Kaplan - July, 2000