Return to Military Theory page

Return to Military Intelligence page

DARPA's Policy Analysis Market for Intelligence: Outside the Box or Off the Wall?

Strategic Insights, Volume II, Issue 9 (September 2003)

by Robert Looney

Strategic Insights is a monthly electronic journal produced by the Center for Contemporary Conflict at the Naval Postgraduate School in Monterey, California. The views expressed here are those of the author(s) and do not necessarily represent the views of NPS, the Department of Defense, or the U.S. Government.

Click here for a PDF version of this article.

"The difficulty lies not so much in developing new ideas as in escaping from old ones."

"Practical men, who believe themselves to be quite exempt from any intellectual influences, are usually the slaves of some defunct economist."

John Maynard Keynes

Introduction

The Defense Advanced Research Projects Agency (DARPA) was born in the uncertain days after the Soviets launched Sputnik in 1958. Its mission was to become an engine of technological change that would bridge the gap between fundamental discoveries and their military use (Bray, 2003). Over the last five decades, the Agency has efficiently gone about its business in relative obscurity, in many cases not getting as much credit as it deserved. The Agency first developed the model for the internet as well as stealth technology. More recently, DARPA innovations have spanned a wide array of technologies. To name a couple: computers that correct a user's mistakes or fix themselves when they malfunction and new stimulants to keep soldiers awake and alert for seven consecutive days.

Because DARPA is mandated to take on risky projects, failures have occurred. For the most part, however, the agency's low profile has protected it from inaccurate scare stories cropping up in the popular press. However, in 2003 DARPA has managed to make the front pages twice, both times with disastrous results. Earlier in the year Congress moved to scale back the agency's Terrorism Information Awareness Program (TIA). In an effort to spot patterns of terrorist activity, this program proposed the development of advanced computer systems capable of scanning commercial databases containing information on millions of Americans.

Then, in late July, the Agency backed off a plan to set up a kind of futures market (Policy Analysis Market or PAM) that would allow investors to earn profits by betting on the likelihood of such events as regime changes in the Middle East. Critics, mainly politicians and op-ed writers, attacked the futures project on the grounds that it was unethical and in bad taste to accept wagers on the fate of foreign leaders and the likelihood of terrorist attacks. The project was canceled a day after it was announced. Its head, retired Admiral John Poindexter, has resigned.

The debate over the Policy Analysis Market has been quite contentious, but there have been few answers to several critical questions: How were the markets supposed to work? What were PAM's underlying theoretical and empirical assumptions? What was PAM supposed to produce in the way of intelligence? As the title of this essay asks, was the project an innovate way of thinking outside the box or just an off-the-wall idea?

How the Policy Analysis Market (PAM) Would Have Worked

With the development during the last several decades of well functioning futures markets for many commodities, private sector analysts often use the prices from these markets as indictors of potential events. The use of petroleum futures contract prices (Looney, Schrady, Brown, 2001) is an example of the manner in which traders gauged the likely outcome of events such as the U.S. Naval response to Iraq's invasion of Kuwait in 1990. In a like manner, the movement of petroleum futures prices in late March 2003, after the recent Iraq war began, reflected the implications traders drew concerning the outcome of the conflict—falling rapidly in the first few days of the conflict, but rising again after it became apparent the Iraqi regime would not fall in a matter of days. Before the Iraq war began, oil prices, incorporating a war premium, suggested there was a very high probability of a conflict (Leigh, Wolfers and Zitzewitz, 2003).

In a similar fashion, the proposed Policy Analysis Market would have provided the U.S. Intelligence Agencies access to a wide variety of markets in various events. Trading in these events, as in the case of petroleum futures, would produce price movements that could be easily translated into the likely occurrence of future incidents, such as the likelihood of a coup in Yemen.

The presumption was that in many cases, intelligence derived in this manner would be more accurate than that obtained through traditional means (see Figure 1, from the original PAM web site). Initially the site was to be confined to political economic, civil and military futures of the key Middle Eastern countries of Egypt, Jordan, Iran, Iraq, Israel, Saudi Arabia, Syria, and Turkey, and the impact of U.S. involvement with each. A typical bet would involve issues such as whether the United States would pull its troops out of Saudi Arabia (Coy 2003), or whether the Egyptian currency was likely to fall by 20% by the end of the year. Assassinations, the most controversial feature of PAM and the most publicized, were not officially listed as a likely market.

Figure 1
DARPA's Vision of the Contribution of Markets to Intelligence

Operationally, PAM planned to offer three types of futures contracts:

  • Quarterly contracts based on data indices that track economic health, civil stability, military disposition, and U.S. economic and military involvement in Egypt, Iran, Iraq, Israel, Jordan, Saudi Arabia, Syria, and Turkey
  • Quarterly contracts that track global economic and conflict indicators such as the likely occurrence of a regime change in Syria
  • Specific possible events (e.g., U.S. recognition of Palestine in the first quarter of 2005)

At the expected start of trading (October 1, 2003), there were to be contracts of the first two types. These were scheduled to mature at the end of the 4th quarter 2003, 1st quarter 2004, 2nd quarter 2004, and 3rd quarter 2004. On January 1, 2004, contracts were to be issued that matured at the end of the 4th quarter 2004. In this way, the forward view of PAM was to be maintained at one year. The plan was to add contracts of the third type as relevant situations presented themselves.

Another design innovation permitted traders to take positions based on interrelated issues. For example, the economic health of a country may affect civil stability in the country, and the disposition of one country's military may affect the disposition of another country's military. The trading process at the heart of PAM allowed traders to structure combinations of futures contracts (see Figure 2, from the original PAM web site). Such combinations were to be structured to represent predictions about interrelated issues of critical interest to the intelligence community. The idea here was to create chains of events leading up to the activity of main concern. It was anticipated that trading in event structured derivatives of this type should result in a substantial refinement in predictive power. In effect this process would be similar to the Bayesian probabilistic forecast techniques used from time to time by intelligence agencies such as the CIA (Schweitzer, 1978). In short, PAM's conditional hedges would have made these markets much more comprehensive and detailed than current online futures exchanges such as www.tradesports.com.

Another limitation (from the intelligence community's perspective) of other on-line sites is that markets in specific events of national security interest (or the derivative driven scenarios noted above) are difficult to start because most players would be unwilling to take an initial position—that is, to be a market maker, willing to accrue potential losses as the markets subsequently moved to new equilibriums. To overcome this, the government would have been the market maker in PAM, implying an on-going stream of public subsidies throughout the market's existence (Norris 2003). As Coy notes "By giving odds to well-informed bettors, the Pentagon would lose money on average. But with bets limited to $100, it would have been a small price to pay for a snapshot of expert opinion".

Finally, as a means of ensuring the smooth and efficient start of live operations, registration was to be limited initially to 1000 traders. As system operations were tuned to the trading load, this limit was to be increased. The plan was to have at least 10,000 traders by January 1, 2004.

Theory Underlying PAM

The Pentagon contended that the system was soundly based on economic theory—specifically, the theory of efficient markets and market discovery. Here the Pentagon cited the fact that many futures markets have a history of predicting events better than individual experts do. In this sense the Pentagon may have lagged a bit behind the times. In the early 1980s, the efficient market theory was widely accepted by academic and Wall Street economists. The thrust of this view stressed the fact that securities markets were extremely efficient in reflecting information about individual stocks and about the stock market as a whole.

The main mechanism driving market prices is the rapid dissemination of reliable and accurate information. Acting on this information, traders in effect immediately incorporate it into the prices of securities. Information driven prices equilibrate so fast that neither technical analysis (the study of past stock prices) nor even fundamental analysis (the examination of financial information such as company earnings and asset values) would enable an investor to achieve returns greater than those that could be obtained by holding a randomly selected portfolio of individual stocks, at least not for shares with comparable risk (Malkiel 2003). One major assumption of the efficient market theory is that if the flow of information is unimpeded and information is immediately reflected in stock prices, then tomorrow's price change will reflect only tomorrow's news and will be independent of the price changes today.

The efficient market theory was largely developed to explain price movements in the U.S. stock market. However, PAM was not going to trade in highly liquid securities where millions of highly sophisticated traders made their daily livings. Instead, DARPA assumed the efficient market theory could be easily extended to other types of markets. The belief is that any market with many hundreds or thousands of participants, making many tens of thousands of decisions, generates an equilibrium price that will hold more information than any one individual expert or team of experts. As John Delaney, chief executive of Tradesports.com, notes "it is reasonable to assume that if you have 20,000 people from 11 countries all trading on whether Saddam will be caught by the end of September, that's probably going to give you a very good real-money predictor." (Blahnik, 2003). How is this possible? In the simplest terms, you might say that markets are collectively intelligent even when the actors who make up those markets are individually dumb—the "dumb agent theory" of market discovery.

Examples of the dumb agent theory abound in the literature. James Surowiecki (February 17, 2003), provides several excellent examples: (1) Every year Michael Mauboussin, an investment strategist at Credit Suisse First Boston teaches a class at Columbia Business School. On the first day of class, he passes out a form and asks the students to estimate IBM's assets at the end of 1989 (not a number that you would expect even business students to know exactly). Every year, without fail, the mean of all the responses is within 5 percent of the actual number. (2) Mauboussin also assembles a good-sized group of people (100-125 people) and gives them a ballot for the Oscars. On one side are the six most popular categories—Best Picture, Best Actress, Best Actor, Best Supporting Actor, Best Supporting Actress, Best Director—and on the other are six more esoteric categories. To play, each participant chips in a dollar and then guesses who will win the Oscar in each category. Obviously, some of the participants know a lot about the movies and about the Oscars, and some know very little. But without fail, the group's mean response across the 12 categories does better than any single human. Two years ago, the group got 11 out of 12 right, while the best human only got nine right.

From these examples Surowiecki (2003) concludes: "I don't know about you, but I find these stories absolutely eerie and absolutely instructive. Taken together, what they seem to suggest—I want to say 'prove', but I'll refrain—is that the collective response of a group to any question of knowledge is going to be both the best response possible (the Oscar example) and a remarkably accurate response as well (the IBM example)."

Another example of market discovery concerns the stock market reaction to the 1986 Challenger crash. In examining the market's response to the crash, Maloney and Mulherin (2003) find abundant evidence of the speed and accuracy of the market in finding the source of the shuttle's demise. While the event was widely observed, it took several months for an esteemed panel to determine which of the mechanical components failed during the launch. By contrast, Maloney and Mulherin find in the period immediately following the crash, securities trading in the four main shuttle contractors seemingly singled out the firm that manufactured the faulty component. Further, they show that price discovery occurred without large trading profits, as the efficient market theory would predict.

While the movement of Morton Thiokol's stock would seem to be a strong verification of market efficiency—the notion that stock prices quickly and accurately respond to new information—the authors correctly don't take this to mean that we should scrap commissions and instead simply look at the market movements when disasters involving publicly held companies occur (Gross, August 8, 2003). The problem is Maloney and Mulherin still cannot figure out why the market knew to blame Morton Thiokol. They pretty much rule out insider information, but they conclude they don't have any other good explanation: "while markets appear to work in practice, we are not sure how they work in theory." Clearly statements like this leave doubts about placing excessive reliance on the ability of markets to quickly and unambiguously generate accurate intelligence.

Still, DARPA saw the dumb agent theory as key in revolutionizing intelligence because its use in markets would presumably uncover and aggregate diverse pieces of information to an extent not previously possible because there would be no bureaucratic or political factors influencing participants' decisions. In effect this system would eliminate many of the hurdles that impede the flow of information within organizations. "That's especially important in the case of the intelligence community because we know that, for example, in the case of 9/11 there was lots of valuable and relevant information available before the attack took place. What was missing was a mechanism for aggregating that information in a single place. A well-designed market might have served as that mechanism." (Surowiecki (2003). Clearly the operative phrase is "a well-designed market."

This concern about the usefulness of markets in providing national security related intelligence was born out by the recent Columbia shuttle disaster, where the market's response was not so accurate. Among the publicly traded NASA contractors, the biggest loser was Alliant Techsystems Inc, the current owner of Thiokol, which made the shuttle's booster rockets. Alliant's stock fell almost exactly the same amount that Morton Thiokol did after the earlier Challenger crash—about 11.66 percent. Boeing, which now owns Rockwell International, a major NASA contractor, fell 1.5 percent and Lockheed Martin fell about 3 percent (Gross, August 8, 2003). Gross (August 8, 2003) notes that, "The market —perhaps remembering Thiokol's implication in the prior disaster—swiftly punished Alliant. Wrongly, it seems. Thus far, attention has focused on the performance of foam insulation lining the external fuel tanks, which were made by the Michoud unit of Lockheed Martin. The Market may be efficient. But it can also be emotional. Did traders with long memories rush to sell Alliant disproportionately because Morton Thiokil was deemed responsible for the Challenger disaster? Almost certainly."

In sum, the efficient market theory underlay the design of PAM. The idea of efficient markets was in vogue mainly in the 1970s, and while still a formidable theory, it has lost much of its luster—no doubt partially the reason so few prominent economists came to the program's defense. The Columbia shuttle case above provides an instructive note of caution in placing exclusive stake in market derived intelligence.

Arguments Against the Funding and Development of PAM

As noted, the main public criticisms of PAM came from politicians and editorial writers, who somewhat unfairly focused on the potential use of the Policy Analysis Market to predict terrorist acts and assassinations—markets unlikely to ever be included in the PAM . Here the main objections to the program were based either on (a) ethical/moral grounds—critics branded the program as in "bad taste" or "ghoulish," and suggested that it encouraged "betting on death," or that it was a government sponsored "betting parlor;" or (b) doubts about its make-up and main premise. Here, the program was, without much elaboration, characterized as "bizarre" or "harebrained." Although PAM was a very minor budgetary item by Washington standards (perhaps less than $1 million to launch) it was also dubbed a "waste of money". More sophisticated arguments came from several noted economists, including a former Nobel Prize winner, Joseph Stiglitz (Berkeley Economist Against Empire 2003). Here the main points of contention were that:

The system creates a strong incentive for someone to buy futures in a violent act and then carry out the act—the insider information problem. This possibility already exists, with terrorists suspected of shorting on airline and hotel stocks right before 9/11. The stock market fell 2.8 percent following President Kennedy's assassination; it fell 4.9 percent following the attacks on the World Trade Center; and the oil markets gyrate wildly with each assassination attempt in the Middle East (Wolfers and Zitzewitz July 31, 2003). Some intelligence analysts contend Saddam speculated on oil futures—going long and then menacing Kuwait or issuing some bombastic threat, and then pocketing a nice profit for his efforts. Given the limits on the size of trades conducted on PAM, it is highly unlikely terrorists betting on their own operations could make a great deal of money.

Clearly, terrorists determined to profit from their actions can easily buy derivatives to cash in on their actions. Of course, they will leave behind a paper trail. But more to the point, all that these prediction markets will do is make the information content of such trading more transparent to policymakers (Wolfers and Zitzewitz July 31, 2003). In any case, by being willing to lose money on the market, the market maker, the Defense Department, would be able to get information at a price.

Markets may not be efficient in the short-run. Here, the argument is that the static value of a contract wouldn't communicate any information; only changes in prices would be of interest. Basically this argument assumes that the bulk of fluctuations in prices on stock and futures markets are driven not by the emergence of new information but only through herding. Thus it would be difficult to extract a signal (i.e. information) about the change in the likelihood of particular events from such a market. "This is more the case over the short-term than the long-term. But it is short-term changes in the likelihood of events that are of the greatest interest."

This criticism may be more on the mark for certain types of PAM markets, especially ones where the participants are not particularly knowledgeable or where public information is rather limited. As noted above, the efficient market theory came under attack starting in the 1980s when sophisticated research found a number of facts at odds with what efficient markets would have produced. For example, stocks that performed poorly in one period typically did better on average over time. If markets were truly efficient, well-informed investors should have sprung into immediate action and bought these poorly performing stocks well before anyone had the chance to make a big profit (Madrick,2003).

In short, it increasingly became clear, even to economists, that human psychology and behavior—sometimes irrational—affected stock prices. The newer theories of behavioral finance appear to explain a number of recent market phenomena as well as or better than the efficient market hypothesis.

Similarly with the dumb agent theory, most of the successful experiments noted above were undertaken with participants voting in private and then their answers presented to the group as a whole. If each participant had shouted out or registered his "vote" in public there is sufficient reason to believe this information would have influenced the position of subsequent votes.

It is also easy to see how emotional or psychological factors could easily affect the outcome in, say, the market for terrorism futures—several near-spaced terrorist attacks would no doubt dramatically raise the market price on terrorist futures even though there might not be any connection. Even in something like oil, it has taken us some time for traders not to act on emotion, but instead to realize that prices have normal ceilings and floors over time based on supply and demand adjustments. Nothing like this appears to encumber terrorist attacks.

In sum, the market failure objection to PAM does have some validity. PAM's market efficiency premise is no longer the dominant market theory as in the 1970s. Since then a number of behavioral theories have increasingly gained acceptance. In short, DARPA may have put too much faith in a theory that is being increasingly questioned in the economics profession. Operationally, the dumb agent theory often cited as producing striking results in private surveys would also, when applied in an on-line environment, come under the came criticisms.

The system would create incentives for participants to manipulate the market by spreading rumors about the likelihood of events. This would further decrease the signal from the market and increase the amount of noise heard by intelligence agencies attempting to monitor threats in a more conventional way.

Here one can simply counter that rumors are a part of any market—the wise investors who make money in the longer term know what they need for verification. Those not so wise are soon out of the market and are not able to bias it in the wrong direction any longer.

The fact that governments would use the markets to predict events and then take action to see that they don't occur would mute any signal. What if the value of the futures contract on the assassination of a friendly foreign leader were to rise rapidly? Clearly, action would soon take place to increase protection of the leader—in essence reducing the likelihood of the event, and the value of the contract. Because market participants recognize this, the market would never show the initial steep rise in the value of the contract.

This is pretty much a straw man argument—focusing on a market unlikely to be offered on the proposed PAM. Of the markets mentioned on the original PAM site, it is hard to see how any would meet the conditions where events could be easily altered. Those that might, like U.S. policy responses, could be handled with conditional datives, like the Pakistan foreign assistance scenario suggested at the end of this essay.

Participants would have doubts that anyone could ever cash in on a futures contract, as it is likely that public outrage would not allow it. It is hard to imagine, for example, Congress allowing anyone to collect on a futures contract betting that the September 11 attacks would occur. Consequently, there will be even less signal in market fluctuations, and changes in market prices will reflect perceptions of the likelihood of a payoff being sanctioned

This is another straw man based on a market not likely to be offered on PAM. Besides, once the government reneged on paying off a bet of this sort, there would be no more action on the PAM market—hardly in DARPA's best interest.

Future contracts can only be written for events that are explicitly anticipated. However, many events of interest, e.g. an attack on the World Trade Center via hijacked passenger planes, are unlikely to be anticipated. Again, this is beside the point, since these contracts were not likely to be offered. PAM was interested in a whole spectrum of more general background type issues.

Other arguments against PAM included:

PAM is a gambling parlor. Some people consider gambling immoral. For one thing, PAM would have had a special legal status, through negotiations with the SEC and IRS resulting in limitations like the $100 investment ceiling that would put it above the usual gambling laws. Of course one can also ask "what is gambling?" There are speculators in all markets—individuals that believe they know more than other participants in the markets do. Generally speculation is thought of as gambling. Sometimes the level of speculation is much greater than the hedging transactions (purchasing the security to hedge a risk), yet we do not treat this activity under the gambling laws (Khallow, 2003).

PAM would be inequitable. Professor Stiglitz (2003) has noted that terrorism futures allow only a sophisticated and wealthy elite to hedge against terrorist attacks, thereby "leaving the rest of America fully exposed." Given the limits on betting size, this eventuality would be highly unlikely. In any case the rich use insurance to protect themselves against losses of this sort.

Assessment

By focusing on "terrorist attacks" as the main events traded on the PAM, critics made it appear preposterous that any good information could be discovered that the FBI and CIA didn't already have. As noted above, most of PAM's contracts would have had nothing to do with terrorist attacks. Instead, markets were to be created in more mundane events like "How fast will Saudi Arabia's non-oil gross domestic product grow next year?" or "What chance does Prince so and so have by the end of the year in ascending to the throne in Country X?" For these areas, there is no doubt considerable information out there that the United States does not have. This information could be converted at relatively low cost into accurate intelligence through something like the Policy Analysis Market.

On the other hand DARPA clearly oversold the plan by assuming its markets would function as predicted by the efficient market theory. If PAM were to focus more on speculative events (like terrorist acts), the critics of PAM's main assumptions are on firmer, but certainly not completely solid ground. For many of these issues the debate between the efficient market vs. behavioral finance schools of thought would be more in favor of the behavioralists. Schiller's (2003) studies show that when participants see a trend in markets devoid of much hard information, they tend to extrapolate. These speculative event markets in something like PAM would no doubt tend to be driven much more by emotional and psychological factors than would the more rational economic/political markets. Again, however, direct markets in terrorist acts were not foreseen as part of the PAM markets.

Clearly, PAM's dilemma is that it would work best in areas of general interest, but would be of much less use in areas of direct and immediate concern to national security. Even if we had information on general environmental factors such as economic political and social conditions, it is not at all clear that markets would have the expertise to convert this information into more accurate assessments of future terrorist acts.

Assuming DARPA's ultimate aim is to derive better forecasts of political instability, terrorist attacks and the like, one approach might entail gathering (in addition to normal sources of intelligence) relevant information available on sites like Tradesports.com. Then to bridge the gap between background intelligence and the prediction of terrorist acts, an internal, interagency version of PAM traded on exclusively by governmental analysts (and perhaps a select outside group of academics, businessmen, and contractors) could be developed.

As an example, using a problem from an earlier Strategic Insight concerning the effectiveness of U.S. assistance to Pakistan in combating terrorism (Figure 3), information flow and policy response/impact can be broken down into four main blocks. The first block —diversionary society—is the basic starting institutional setting in Pakistan. Here the main components, corruption and the like, can be quantified sufficiently to assign rough magnitudes of change over time. Betting on changes in these factors would produce conditional input into the second, environmental block—low economic freedom (an index for which exists in the literature), productivity and religious school expansion.

At this point, the exercise would shift from public on-line sites to secure internal sites with only "experts" participating. Their assessment and votes on the effectiveness of proposed and discussed U.S. aid/assistance responses would produce a final set of conditional probabilities that in turn would drive, through bets, an overall terrorism activity index that could be monitored and revised on a daily basis.

Which agencies appear to have the best information? The best analysts? At a minimum, the system would assist in cutting through bureaucratic and institutional barriers to allow a better flow of information between agencies and even within agencies.

Conclusions

Although the Policy Analysis Market appears to be a dead issue, it did break new ground in the country's search for better intelligence. The PAM idea embodied a solid body of theory and proven empirical capability. While one can quibble about how closely PAM markets would approximate the efficient market hypothesis, there is no doubt trading on many future events would come close enough to provide valuable intelligence. Thus, while it was a public relations disaster, some version of the program will likely be introduced on a restricted basis, perhaps along the lines suggested above, in an attempt to better tap the country's disperse knowledge base, human insight, and analytical expertise. This solution is far from perfect, not allowing realization of the full potential of the program. Lou Dobbs (2003), has perhaps best summed up this unfortunate episode:

We will never know if the Policy Analysis Market would have been successful. But if there were even a small chance that it could have been a useful tool, there should be, at a minimum, further discussion of the idea. This is, after all, not a matter of just partisan politics but one of national security. And forcing the resignations of those involved with the planning is a strong deterrent to progressive thinking, of which we have no surplus.

For more insights into contemporary international security issues, see our Strategic Insights home page.

To have new issues of Strategic Insights delivered to your Inbox at the beginning of each month, email ccc@nps.navy.mil with subject line "Subscribe". There is no charge, and your address will be used for no other purpose.

References

"You Bet Your Life," Financial Times, August 4, 2003.

A Berkeley Economist Against Empire—Blog July 29, 2003.

Blahnik, Mike,"Pentagon Drops Plans for Terrorism Futures Exchange," Minneapolis Star Tribune, July 30, 2003.

Bray, Hiawatha, "In Defense of DARPA Programs," Boston Globe, August 3, 2003.

Brenner, Reuven, "A Safe Bet," Wall Street Journal, August 1, 2003.

Brown, Jeff, "Lack of Intelligence True Demise of terrorism Futures," Houston Chronicle, August 2, 2003.

Coy, Peter, "Betting on Terror: PR Disaster, Intriguing Idea," Business Week, August 25, 2003, p. 41.

Dobbs, Lou, "Deep-sixing a Bright Idea," US News and World Report, August 11, 2003.

Gross, Daniel, "Bookmakers for the Bomb-Makers," Slate, July 29, 2003.
Gross, Daniel, "The Disaster Market: Can Wall Street Figure Out the Cause of a Space Shuttle Crash Faster than NASA's Experts?," Slate, August 8, 2003.

Heuer,Richards J., "Adapting Academic Methods and Models to Governmental Needs" in Richards J. Heuer, ed., Quantitative Approaches to Political Intelligence: The CIA Experience (Boulder,Co: Westview Press 1978), pp. 1-10.

Khallow, "The Policy Analysis Market: Why it is a Great Idea," (August 14, 2003).

Leher, Eli, "The Terror Gamble," National Review Online, July 30, 2003.

Leigh, Andres, Justin Wolfers and Eriz Zitzewitz, "What do Financial Markets Think of War with Iraq?," Stanford University, March 17, 2003.

Looney, Robert, "Oil Prices and the Iraq War: Market Interpretations of Military Developments," Strategic Insight, April 2003.

Looney, Robert, "Market Effects of Naval Presence in a Globalized World: A Research Summary," in Sam J. Tangredi ed., Globalization and Maritime Power (Washington, NDU Press, 2003), pp.103-132.

Looney, Robert, David Schrady and Ron Brown, "Estimating the Economic Benefits of Forward-Engaged Naval Forces," Interfaces 31:4 July/August 2001, pp. 74-86.

Looney, Robert, "Failed Economic Take-Offs and Terrorism: Conceptualizing a Proper Role for U.S. Assistance to Pakistan," Strategic Insight, February 1, 2003.

Madrick, Jeff, "Market Messes Happen: And Inefficiencies Have Consequences," New York Times, August 3, 2000, c.2.

Malkiel, Burton G., "The Efficient Market Hypothesis and Its Critics," Journal of Economic Perspectives 17:1 Winter 2003, pp. 59-82.

Maloney, Michael and J. Harold Mulherin, "The Complexity of Price Discovery in an Efficient Market: The Stock Market Reaction to the Challenger Crash," Journal of Corporate Finance 9:4, September 2003, pp. 453-479.

Mickolus, Edward F., "An Events Data Base for Analysis of Transnational Terrorism" in Richards J. Huer, ed., Quantitative Approaches to Political Intelligence: The CIA Experience (Boulder,Co: Westview Press 1978), pp. 127-163.

Norris, Floyd, "Betting on Terror: What Markets Can Reveal," New York Times, August 3, 2003.

Pethokoukis, James M., "The Big Ideas of Robin Hanson Chief Economist at Starfleet Command," U.S. News and World Report July 30, 2003, p. 1.

Pinkerton, James, "In Terrorism Game, Intelligence is Big Loser," Newsday, July 31, 2003, p. A44.

Schweitzer, Bayesian, "Analysis: Estimating the Probability of Middle East Conflict," in Richards J. Huer, ed., Quantitative Approaches to Political Intelligence: The CIA Experience (Boulder,Co: Westview Press 1978), pp. 11-30.

Shiller, Robert J. "From Efficient Markets Theory to Behavioral Finance," Journal of Economic Perspectives 17:1 Winter 2003, pp. 83-104.

Surowiecki, James, "The Dumb Smart Market," Slate, February 17, 1999.

Stiglitz, Joseph, "Terrorism: There's No Futures in It," Los Angeles Times, July 31, 2003.

Wolfers, Justin and Eric Zitzewitz, "The Furor over Terrorism Futures," Washington Post, July 31, 2003.