Return to Open Source Intelligence topic area
Ninety percent of intelligence comes from open sources. The other ten percent, the clandestine work, is just the more dramatic. The real intelligence hero is Sherlock Holmes, not James Bond. -- Lieutenant General Sam Wilson, USA Ret. former Director, Defense Intelligence Agency
Former Ambassador to Algeria L. Craig Johnstone (presently State Department Director of Resources, Plans and Policy) recently told a Washington conference that during his assignment in Algeria, he bought and installed a satellite dish enabling him to watch CNN so he could have access to global news. He recalled:
The first week I had it running was the week of the Arab League summit in Algiers and, for whatever reason, the Department was interested in finding out whether Yasser Arafat would attend the summit. No one knew, and the day of the summit Washington was getting more frantic. We in the Embassy were banned from the summit site so there was no way we could find out whether or not Yasser Arafat would show. Finally, at about noon I was home for lunch and watching CNN when the office of the Secretary of State called. The staffer on the other end asked if there was anything at all he could tell the Secretary about Arafat's participation. And just then, on CNN I saw a live picture of Yasser Arafat arriving at the conference. "He is definitely at the conference," I reported. The staffer was ecstatic and went off to tell the Secretary. The next day I received a congratulatory phone call from the NEA bureau for pulling the rabbit out of the hat. How did you find out, they asked? The secret was mine. But I knew then and there that the business of diplomacy had changed, and that the role of embassies, how we do business in the world, also had to change.
Ambassador Johnstone's story provides an example of the value of information from open sources. Allen W. Dulles, when he was Director of Central Intelligence, acknowledged to a congressional committee, "more than 80 percent of intelligence is obtained from open sources." Whether the amount of intelligence coming from open sources is 90 percent, 80 percent, or some other figure, experienced intelligence professionals agree that most information processed into finished intelligence may be available from open sources. This essay explores the significance of a trend toward increased recognition of the role of open source information and discusses what this may mean for intelligence consumers at every level.
The use of information from open sources (OSINT) for intelligence production is not a new phenomenon. Intelligence services in most nations have always made use of OSINT obtained by working with scholars in academia, debriefing business travelers and tourists, and examining foreign press and broadcast media. Intelligence prepared from sources available to the general public draws from books, periodicals, and other print publications such as catalogues, brochures, pamphlets, and advertisements. Also included are radio and television broadcasts and a more recent technological innovation, the Internet. Collectively, these are frequently referred to as open media resources.
Intelligence--information and analysis that is not available to the public--is prepared for use by policymakers and military leaders inside the government. Intelligence is categorized customarily according to the source from which it is obtained. Today, five sources are recognized:
While most discussions of open source intelligence seem to concentrate on intelligence collection, it is important to view intelligence trends in conjunction with developments in its traditional components. These components are:
• Costs. With decreasing national security budgets, government leaders are having to examine their infrastructure. As military forces become more dependent on off-the-shelf commercial technology, intelligence organizations appear headed toward greater reliance on open source intelligence.
• Sources. Cost-driven decisions dictate that a significant quantity of intelligence requirements can be filled by a properly designed comprehensive monitoring of open sources, either by the intelligence establishment itself or by private organizations. A particular advantage of open source intelligence is that the product can be maintained at a low level of classification required for these sources and methods. This outcome allows relatively wide dissemination and distribution when compared with material from other sources. This characteristic of open source intelligence is particularly important in coalition operations.
• Methods. It has been demonstrated many times that good intelligence production relies on all-source assessment. Traditional intelligence structures and methods have been optimized for designated core or central missions, and today many of these remain structured to meet Cold War requirements and scenarios. Current and likely future contingencies seem less likely to involve major hard military net assessments and diplomatic intelligence than was the case between 1945 and 1991. Current and future contingencies probably will continue a trend toward soft analyses of complex socioeconomic, technological, and political problems, and of issues that will include such items as international organized crime, information warfare, peacekeeping operations, and activities associated with special operations and low-intensity conflict.
• Targets. Intelligence targets of greatest concern to US leaders have changed since the collapse of the Soviet Union, the accompanying geopolitical upheavals (such as political deterioration in the Balkans), and changes in Western perceptions of global security interests (e.g., the significance of the Middle East). Intelligence agencies must now focus their activities on a far broader range of targets and potential targets than was common in the Cold War era. Today, intelligence professionals have to be concerned with terrorism, major international crime, and arms proliferation, including programs in some areas to produce weapons of mass destruction. They have to be prepared for possible military intervention on short notice in overseas conflicts or for humanitarian relief. Some of these targets require constant scrutiny in substantial depth; for others, broad general surveillance will suffice--provided a reserve or surge capability is maintained.
Although many aspects of intelligence work are changing, for the near term the preponderance of them will probably remain familiar. Today's emerging main problem is how to deal with new and indistinct boundaries among and between intelligence organizations and functions, and increasing ambiguity in roles and missions. Any intelligence officer who has ever worked at a senior level knows that senior policymakers and government officials abhor ambiguity; they want timely, accurate intelligence. As Peter Schwartz, a recognized futurist, founding member of the Global Business Network, and author of The Art of the Long View, told his audience at the Colloquium on the 21st Century, "We will see not only changing rules of the game, but new games. There is an emerging competitive information marketplace in which non-state intelligence will be `cheap, fast, and out of control.'"
Enthusiastic proponents of open source intelligence argue that the information revolution is transforming the bulk of any nation's intelligence requirements and reducing the need to rely upon traditional human and technical means and methods. But Robin W. Winks, distinguished Yale University historian who served in the Office of Strategic Services during World War II and in its successor, the Central Intelligence Agency, concluded, "Research and analysis are at the core of intelligence . . . . [Most] `facts' are without meaning; someone must analyze even the most easily obtained data."
The emerging debate between investing in technology and developing competent analysts concerns itself basically with the value and role of open source intelligence. To understand some of the forces that are shaping the debate, we need to weigh the relative benefits of primary and secondary sources, two discrete subsidiary classes of open source material. Primary sources, generally taken to include print and electronic media, have always provided information of value to the intelligence community in current intelligence, indications, and warning as well as background information used by analysts in their work. What the so-called information revolution has done is to increase the ability of users to gain access and to manipulate the information, and although most intelligence managers do not believe that the number of primary sources has expanded greatly, the number of secondary sources has increased exponentially. To compound the analyst's problem, the objectivity and reliability of many secondary sources are often questionable. We will need more experience before we can accept expansion of secondary sources as a benefit to the management of national security.
The largest general open source collection in the world is the Library of Congress. To replace the original library, which was destroyed during the War of 1812, Congress in 1815 purchased the private library of former President Thomas Jefferson, greatly increasing the collection's size and scope. The Library of Congress now includes works in more than 450 languages and comprises more than 28 million books, periodicals, and pamphlets as well as manuscripts, maps, newspapers, music scores, microfilms, motion pictures, photographs, recordings, prints, and drawings. The library's services also include research and reference facilities, which coordinate with or amplify local and regional library resources.
There are also several thousand databases available from commercial organizations; LEXIS/NEXIS, Dialog, Reuters, and The New York Times come to mind. Any discussion of contemporary open sources must now include the Internet and the World Wide Web (WWW). The World Wide Web (developed in 1989) is a collection of files, called Web sites or Web pages, identified by uniform resource locators (URLs). Computer programs called browsers retrieve these files.
The term "Internet" describes the interconnection of computer networks, particularly the global interconnection of government, education, and business computer networks, available to the public. In early 1996, the Internet connected more than 25 million computers in more than 180 countries. The Internet provides an immense quantity and variety of open source information and must be increasingly looked upon as a source for intelligence purposes.
The Internet and the World Wide Web exemplify technology that is not yet mature. One hallmark of immature technology is an underlying anarchy and a potential for disinformation. In October 1938, when radio broadcasting was emerging as a reliable source of information, producer-director Orson Welles, in his weekly radio show Mercury Theater, presented a dramatization of an 1898 H. G. Wells story, War of the Worlds. The broadcast, which purported to be an account of an invasion of earth from outer space, created a panic in which thousands of individuals took to the streets, convinced that Martians had really invaded Earth. Orson Welles later admitted that he had never expected the radio audience to take the story so literally, and that he had learned a lesson in the effectiveness and reach of the new medium in which content was struggling to catch up to technology.
Recent examples with the Internet and its spin-offs suggest that e-mail abuses, careless gossip reported as fact, and the repeated information anarchy of cyberspace have become progressively chaotic. This does not mean that the Internet and the Web cannot be considered seriously for intelligence work, but it does mean that intelligence officers must exercise a vigilant and disciplined approach to any data or information they acquire from on-line sources.
In December 1997, senior officials from Germany, Canada, France, Italy, Japan, Britain, Russia, and the United States (the Group of Eight industrialized nations) gathered in Washington to explore the transnational nature of computerized crime, with specific attention to opportunities for criminals to exploit the Internet's legal vacuum. Among the facts presented to the officials were these:
The principal significance of these facts for the intelligence officer is that Internet sources are subject to manipulation and deception. Consequently, counterintelligence and security processing will henceforth have to include cyberspace during analysis.
Perhaps the greatest value to military organizations in this array of adjustments following the end of the Cold War and the proliferation of technologies is freedom from confinement to a fixed geographic site for ready access to basic unclassified references. Modern communications will free deployed military from the need to transport large quantities of reference material (classified and unclassified) during operations. Military forces in the field can now tap into an immense quantity of information resources in near real-time. Four relevant types are:
Since the quantities of information available are great and much of the information is often irrelevant, staffs of deployed units may find it difficult to use the information productively. Deployed organizations may well have to establish forward and rear intelligence activities. The threat of information warfare will have to be taken into account in planning and executing split-echelon operations.
Providing unclassified information to the general public as well as to officials is the objective of democratic governments in their declarations of open and immediate reporting. Even the tabloid press has never advocated a freedom that would deliberately compromise national security or put the lives of service members at risk, yet there can be unintended consequences from such expanded openness. The British government learned this during the 1982 Falklands campaign when a BBC reporter inadvertently revealed operational plans for what proved to be a costly assault at Goose Green by the Parachute Regiment: the enemy was listening. During Operation Desert Storm, the US government and its coalition partners would encounter other problems. While CNN was reporting directly from the theater of operations, government control of mass communications was in effect in Israel, Jordan, and Saudi Arabia, as it was in Iraq. The sites of SCUD attacks on Israel were quickly cordoned off by the authorities; media representatives were granted access only after a response had been determined by the Israeli government. The state-owned Iraqi media not only repeatedly told its citizens they were winning the struggle, but it manipulated reporting of the use of the Patriot missile against the SCUD, ensuring that CNN and others reported only what the Iraqi government wished. Coalition anti-SCUD measures soon were placed under direct control of Washington.
Intelligence consumers, government officials, and policymakers have not been complaining about a shortage of information; they are suffering from a saturation. The flood of mass-produced data now available and the ensuing overload means that collection is no longer the principal problem. The greater challenge facing intelligence organizations is analysis, consolidation, and timely dispatch of data and results to the individuals who need it. Effectiveness in this process will depend upon allocation of human resources among those responsible for analysis and others responsible for its transmission. An information management executive will consider any increase in volume as proof that information is being managed better, even more efficiently. But the information manager is not in the business of analysis, so he or she is not interested in how well or poorly the information is interpreted, or even if it contains disinformation or inaccuracy. One cannot equate increased throughput to improved situation awareness within a theater of operations.
Nevertheless, the quantitative arguments of information managers recently have become more effective than those of the intelligence community with respect to open source policy. The last time a similar contention occurred, the proponents of technical intelligence argued that they had the key to ultimate wisdom. As the late Ray Cline, one-time Deputy Director of Intelligence at CIA and later Director of the Department of State's Intelligence and Research Bureau observed,
The technical miracle has greatly reduced the burden on the secret agent. Lives need not now be risked in gathering facts that can easily be seen by the eye of the camera. . . . Instead the agent concentrates on gathering ideas, plans, and intentions, all carried in the minds of men and to be discovered only from their talk or their written records. Nobody has yet taken a photograph of what goes on inside people's heads. Espionage is now the guided search for the missing links of information that other sources do not reveal. It is still an indispensable element in an increasingly complicated business.
Claims of open source enthusiasts need to be examined in context. Those making extravagant claims sometimes have little vested interest in the role and value of open source materials, or even the knowledge or experience to make reliable judgments about the broader issue of multidisciplined all-source analysis by skilled intelligence analysts.
The communications revolution is presenting intelligence organizations with a new challenge far beyond that of mass production. Like other enterprises, intelligence now faces competition from directions believed to have been impossible only a few years ago. As has been true with commerce and industry, intelligence will have to remodel its organization, form new associations, tailor or customize its products, and question its fundamental missions. So long as there are nations led by aggressive totalitarian rulers inclined toward terrorism, or there are fanatics equipped with lethal weapons, democracies will continue to need effective secret services.
1. Reported by David Reed, "Aspiring to Spying," The Washington Times, 14 November 1997, Regional News, p. 1.
2. Remarks at opening session of the Conference Series on International Affairs in the 21st Century, US State Department, Washington, D.C., 18 November 1997.
3. US military operations in Somalia, Haiti, and Bosnia are examples of requirements of a different nature.
4. It is important to keep in mind an old intelligence maxim: "You can't surge HUMINT!"
5. Address, Washington, D.C., 21 October 1997.
6. Robin W. Winks, Cloak & Gown: Scholars in the Secret War, 1939-1961 (2d ed.; New Haven, Conn.: Yale Univ. Press, 1996), p. 62.
7. One source estimates the current total to be more than 8000 such databases.
8. The Internet was initially developed in 1973 and linked computer networks at universities and laboratories in the United States. This was done for the US Defense Department's Advanced Research Projects Agency (DARPA). The project was designed to allow various researchers to communicate directly in connection with their work. It was also developed with the idea in mind that it could provide a nuclear survivable communications system.
9. Current estimates suggest that around 30 million individuals and more than 40,000 networks are connected, numbers which appear to be increasing rapidly. The quantity of data on the Internet is huge. One estimate is total content between two and three terabytes. (A terabyte is a million megabytes.) A typical public library of some 300,000 books has about three terabytes of data. Rajiv Chandrasekaran, "In California, Creating a Web of the Past," The Washington Post, 22 September 1996, p. H1. An essay by James Kievit and Steven Metz, "The Internet Strategist: An Assessment of On-line Resources," Parameters, 26 (Summer 1996), 130-45, available on the Internet, is an excellent introduction and guide.
10. Ray Cline, "Introduction," in The Intelligence War (London: Salamander Press, 1984), p. 8. Emphasis added.
The Reviewer: Colonel Richard S. Friedman (USA Ret.) served in the European, African, and Middle Eastern theaters in World War II as an intelligence NCO in the Office of Strategic Services. After the war, he was commissioned from Army ROTC at the University of Virginia, where he received a law degree. He subsequently served in a variety of intelligence and special forces positions, including an assignment as the senior US intelligence officer at NATO Headquarters in Brussels. Since retiring from the Army, he has worked for the Central Intelligence Agency as a senior analyst, assistant national intelligence officer, and staff operations officer. Colonel Friedman was the lead author of Advanced Technology Warfare (1986) and contributed chapters to The Intelligence War (1984) and U.S. War Machine (1987). As with all Parameters articles and reviews, the views expressed herein are those of the author; they do not represent Department of Army policy or that of the Central Intelligence Agency or any other agency of the US government.
Return to top