Return to Contents

Directorate of Intelligence	Notes on Analytic Tradecraft	Product Evaluation Staff
		June 1995
	Note 4
	Outlook
	This is the fourth in a series of Product Evaluation Staff notes to clarify the standards used for evaluating DI assessments and to provide tradecraft tips for putting the standards into practice.

This Note on Outlook continues coverage of DI standards for helping policymakers, warfighters, and law enforcers manage uncertainty regarding threats to and opportunities for US security interests. (Please see Note 3, Articulation of Assumptions.)

The outlook sections of DI assessments must address not only what analysts believe to be the most likely course of development but also important alternative outcomes that could confront US officials. This concern for the unlikely or unexpected reflects an experience-based appreciation of (1) the fallibility of analytic judgment, (2) the special analytic requirements of US officials for addressing long-shot threats and opportunities, and (3) the adversarial character of the policymaking process.

The perils of analytic judgment include:

• Limitations in the evidence and in the analysts' understanding of complex issues that can lead to judgments structured around either the wrong drivers (key variables) or incorrect linchpin assumptions.
• The impact of essentially unpredictable events on sound analytic judgments; for example, changes in political stability caused by assassination of a president or the sudden advent of a charismatic opposition leader.

Policymakers and other consumers who carry the burden of executing US policy often have a different attitude than analysts toward unlikely events.

• Officials who are charged with advancing a long-shot objective may probably see a 20 percent likelihood of success as promising opening odds for arranging a negotiated settlement of a longstanding civil war or diverting an entrenched nuclear weapons program.
• Officials also require an understanding of unlikely developments to prepare contingency plans against low-probability, high-impact dangers (such as protecting Persian Gulf oil suppliers from another Iraqi military attack on Kuwait or an Iranian blockade of the Straits of Hommuz).

Finally, the outlook on important issues can be subject to deep-seated disagreements - among and between analysts, Administration officials, and congressmen - that generate suspicions about the objectivity of DI memoranda and briefings.

• At such times, DI credibility can depend more on explanation of why the outlook - for example, for Cuba after Castro or for Russia after the next election cannot be predicted with high confidence than in an attempt to trivialize uncertainty by "just making the call."

General Rules

As indicated in previous Tradecraft Notes, each DI assessment represents a distinctive challenge to analysts on how to assure credibility with the consumers who count most in planning and executing US policy. The general rules that follow should weigh heavily as analysts choose the most appropriate approach.

1. In all cases, the analysts' judgment on the most likely outcomes must follow clearly from the marshaling of evidence and articulation of assumptions.

• The more complex and controversial the issue, the more analytic effort is required to ensure that drivers and linchpin assumptions are precisely stated and well defended.

2. When the US stake is high (for example, political stability of an important ally or adversary) and plausible alternative outcomes are judged to be in the range of for instance, 20 percent or greater, the outlook section should also structure these potentially important outcomes. Policymakers and other consumers should get enough understanding of these alternatives to decide whether to review contingency plans, ask for more evidence, or levy additional tasks on analysts.

Analysts should identify:

• The factors at play that could lead to unexpected developments. What alternative key variables could drive the outcome in a different direction (perhaps the economy, or growing corruption)? Or what plausible reversal of linchpin assumptions about correctly defined drivers could turn things around (perhaps the Egyptian military will not remain loyal to the Mubarak government)?
• Signposts or indicators of change. What intermediate developments would indicate that events may not be working out as originally expected? For example, such negative signposts for regime stability as the advent of coup plotting or capital flight.
• Events that could trigger a major shift in direction. What essentially unpredictable events could precipitate the unexpected outcome? For example, a crop failure, or a mishandled natural disaster, or anti-government demonstrations that get out of control.

3. On issues vital to US security (for example, military intentions of adversaries), analysts must help policymakers and warfighters engage in contingency planning by:

• Addressing all important factors at play that could influence subsequent events.
• Ranking their relative importance.
• Relating them to a range of plausible outcomes, including alternatives analysts deem to be remote possibilities against which policy planners may nonetheless decide to deploy resources.

Tradecraft Tips

1. Analysts should avoid unrealistic precision in citing odds. Weather forecasters may have sufficient data to determine that the chances of rain are slightly better than even. Analysts rarely have such luxury in assessing the political or economic outlook.

2. Also avoid phrases that compound unavoidable uncertainty with unnecessary confusion: for example, real possibility, good chance.

3. As a rule, use constructions that tie the outcome to the driver and linchpin assumptions, rather than flat predictions.

• Castro is unlikely to be forced out of office over the next six months unless his security forces mishandle disturbances and trigger prolonged violent riots.

4. To minimize misinterpretation when making important judgments, combine probabilistic phrases and rough numerical odds.

• On balance, Castro is unlikely to be forced from office in the next six months (less than a one-in-five chance).

5. To understand the often intense concern of hands-on policy officials about a 20 percent likelihood of a disastrous turn of events happening on their watch, analysts should consider whether they would voluntarily get on an airplane that had a one-in-five chance of crashing.

6. To deal rigorously with unlikely developments, analysts should switch their focus from whether something will happen to how it could happen. Some techniques for generating credible alternative scenarios by structuring the available evidence differently:

• Who and What Counts. Generate lists of the key individual and group players that can determine the outcome of an issue (for example, whether Egyptian politics will continue to simmer or will explode), and critically examine the factors most likely to determine their behavior (such as institutional loyalties, personal rivalries, family interests).
• Thinking Backwards. Start with the assumption that the targeted danger or opportunity has occurred. For example, revolutionary violence undermines the Castro government! Then generate one or more plausible "how-it-could-happen" scenarios, by postulating who would have to do what, when, how, and why.
• Judgments Discarded. List assumptions that fell by the wayside in reaching the bottom line. Regarding the judgment that the military will continue supporting the Egyptian government, for example, revisit any available data or supportive reasoning indicating such support may be weakening, for example, among junior officers.

Forward to "Note 5"

Return to Contents