Click HERE to return to the Plain English intro page.
Testing Your Documents
Testing your documents should be an integral part of your plain language writing process. It should not just be something you do after the final version to see if it worked. This is especially important if youre writing to hundreds, thousands or even millions of people. The information gained in testing can save time in answering questions about your document later.
Types of Testing
You can do either qualitative (What do people think about it?) or quantitative (Do the numbers show that this is a success?) testing. Protocol testing and focus groups produce qualitative data; control groups are used to produce quantitative data.
Focus Groupsare conducted with a small group of people (usually 8-12). Focus groups are very valuable in gathering information about the way people feel about a given document. Participants will tell you if they like or dislike something. They will tell you if they think they understand what you are doing. And they may even tell you a better way to do something.
Prepare a moderator's guide (a list of questions or script for the moderator) in advance. Design the questions to generate a discussion among the participants about the document they are reviewing. Beobjective; don't ask leading questions that will bias the answers. In professional settings, you can use a one-way mirror for observers (usually those with a stake in the project) to watch the participants. This is valuable because it prevents observers' expressions from swaying the participants. Focus groups can be taped to allow the moderator to write an accurate final report.
While focus groups are very valuable in some situations, they are not usually an effective way to test the usability of a letter, or to learn how well an individual really understands what you have written.
Protocol Testingis an interview technique that tests the usability of a document. Protocol tests will tell you what a reader actually thinks it means. It is extremely valuable in determining if the reader is interpreting your message the way you intended.
Testing involves a one-on-one interview with a reader. You should conduct 6 to 9 interviews on each document. Ask the reader to read to a specific cue (usually a dot identifying a stopping point). Each time the reader reaches the cue, ask for an explanation of what that section means. At the end of the document, ask additional questions, such as:
You should use a different type of protocol testing when evaluating long documents, like booklets and regulations. In this type of protocol test, not only do you test for comprehension, but you are also make notes about the way the reader uses the document. For instance you would note how often a reader has to flip from page to page, to find references. In other words, you test the document as a whole, not just individual paragraphs.
Protocol testing is rather time consuming. Is the time invested in this technique worth the effort? The answer is a resounding "yes." Tested letters eliminates needless phone calls and correspondence asking for clarification of the original letter.
Remember, the goal of protocol testing is to ensure that your readers understand your letters, and therefore won't have to call you for an explanation. Although this technique is very valuable, it probably isn't worth the time to test letters that go to only one or a very few people.
Control Studieswill allow you to collect quantitative data on how well the general public uses the final document you've produced. Control studies can be done in several different ways, but they all have similar characteristics. Before you do a control study, it is important that you know what type of results you will consider a success. For instance,
Having answers to these questions will help you determine whether your document is successful.
Send a small test group of people the new version of your document. Let's say you're sending the new version to 700 people. You should also send 700 people, your control group, the old document. Track the responses to all 1400 documents and compare the results. Note that it is significantly easier to test results when people return a written response than when you try to track the number of phone calls you receive. (If you have a statistician or actuarial staff, they can tell you how many people you should use to make your study scientifically valid. If your agency doesnt have an expert on staff to help you, statistics books will give you a formula to determine a good sample size for your study.)
There are numerous other ways of collecting quantitative data. For instance, you can record what percentage of your "before" letters generates correct responses compared to your "after letters," or what percentage of each letter results in your customer calling you asking for an explanation. This approach is much easier when testing phone responses.
You should use control studies after your qualitative testing is completed and you believe you have the best possible document. That's because control testing will tell you if the new document is a success, but it won't tell you why it is or isn't a success.When to Use What Tool
Like any good tool, focus groups, protocol tests and quantitative data collection are most successful when used for their intended purpose. The chart below will show you the best times to use different methods of testing documents. Focus Groups and Control Groups are optional depending on what type of document you are rewriting. But protocol testing is an essential tool to help the writer know where changes should be made in their document. The chart below shows the most effect times to use each tool.