Click HERE to return to the Plain English intro page. banner.gif (20700 bytes)

Testing Your Documents

Testing your documents should be an integral part of your plain language writing process. It should not just be something you do after the final version to see if it worked. This is especially important if you’re writing to hundreds, thousands or even millions of people. The information gained in testing can save time in answering questions about your document later.

Types of Testing

You can do either qualitative (What do people think about it?) or quantitative (Do the numbers show that this is a success?) testing. Protocol testing and focus groups produce qualitative data; control groups are used to produce quantitative data.

Focus Groups are conducted with a small group of people (usually 8-12). Focus groups are very valuable in gathering information about the way people feel about a given document. Participants will tell you if they like or dislike something. They will tell you if they think they understand what you are doing. And they may even tell you a better way to do something.

Prepare a moderator's guide (a list of questions or script for the moderator) in advance. Design the questions to generate a discussion among the participants about the document they are reviewing.  Beobjective; don't ask leading questions that will bias the answers.  In professional settings, you can use a one-way mirror for observers (usually those with a stake in the project) to watch the participants. This is valuable because it prevents observers' expressions from swaying the participants. Focus groups can be taped to allow the moderator to write an accurate final report.

While focus groups are very valuable in some situations, they are not usually an effective way to test the usability of a letter, or to learn how well an individual really understands what you have written.

Protocol Testing is an interview technique that tests the usability of a document. Protocol tests will tell you what a reader actually thinks it means. It is extremely valuable in determining if the reader is interpreting your message the way you intended.

Testing involves a one-on-one interview with a reader. You should conduct 6 to 9 interviews on each document. Ask the reader to read to a specific cue (usually a dot identifying a stopping point). Each time the reader reaches the cue, ask for an explanation of what that section means. At the end of the document, ask additional questions, such as:

    • What would you do if you got this document?
    • Do you think the writer was trying to help you?

You should use a different type of protocol testing when evaluating long documents, like booklets and regulations. In this type of protocol test, not only do you test for comprehension, but you are also make notes about the way the reader uses the document. For instance you would note how often a reader has to flip from page to page, to find references. In other words, you test the document as a whole, not just individual paragraphs.

Protocol testing is rather time consuming. Is the time invested in this technique worth the effort? The answer is a resounding "yes." Tested letters eliminates needless phone calls and correspondence asking for clarification of the original letter.

For instance
Veterans Benefits Administration tested a letter in which readers appeared to understand every word. However, when asked what they would do if they got this letter, most people said they would call VBA's toll-free number. The letter was about a replacement check sent because the original check was now out of date. The letter said, "You will receive the new check shortly." Readers indicated that they would call if they didn’t receive the check the same time as the letter. Changing the sentence to show an approximate date they would receive the check eliminated countless phone calls.
In another situation, some readers were confused by the VA term "service-connected disability." To VBA it means that a veteran has a disability that can be traced back to his/her time in military service." Protocol tests showed that one veteran thought it meant a disability that happened at work. Another was injured while in the military, but not while on duty, and did not know if he had a service-connected disability. When each reader was asked a general question about understanding the letter, they all said that it was clear. Yet several would have done something different than what VBA wanted because they had a different definition of "Service-Connected." The solution was to explain the phrase so that everyone was working from the same definition.

 Remember, the goal of protocol testing is to ensure that your readers understand your letters, and therefore won't have to call you for an explanation. Although this technique is very valuable, it probably isn't worth the time to test letters that go to only one or a very few people.

Control Studies will allow you to collect quantitative data on how well the general public uses the final document you've produced. Control studies can be done in several different ways, but they all have similar characteristics. Before you do a control study, it is important that you know what type of results you will consider a success. For instance,

  • Do you want more calls regarding a certain program?
  • Do you want fewer calls asking for clarification?
  • Do you want more people to return an application or a payment?
  • Are there certain parts of the application that you would like to be completed more accurately?

Having answers to these questions will help you determine whether your document is successful.

Send a small test group of people the new version of your document. Let's say you're sending the new version to 700 people. You should also send 700 people, your control group, the old document. Track the responses to all 1400 documents and compare the results. Note that it is significantly easier to test results when people return a written response than when you try to track the number of phone calls you receive. (If you have a statistician or actuarial staff, they can tell you how many people you should use to make your study scientifically valid. If your agency doesn’t have an expert on staff to help you, statistics books will give you a formula to determine a good sample size for your study.)

There are numerous other ways of collecting quantitative data. For instance, you can record what percentage of your "before" letters generates correct responses compared to your "after letters," or what percentage of each letter results in your customer calling you asking for an explanation. This approach is much easier when testing phone responses.

You should use control studies after your qualitative testing is completed and you believe you have the best possible document. That's because control testing will tell you if the new document is a success, but it won't tell you why it is or isn't a success.

When to Use What Tool

Like any good tool, focus groups, protocol tests and quantitative data collection are most successful when used for their intended purpose. The chart below will show you the best times to use different methods of testing documents. Focus Groups and Control Groups are optional depending on what type of document you are rewriting. But protocol testing is an essential tool to help the writer know where changes should be made in their document. The chart below shows the most effect times to use each tool.

Method of Testing When to Use It What You will Get
Protocol Test (qualitative) After completing a final draft of your document
  • Specific information about what the reader thinks your document means.
  • Information about what they will do with the document when they receive it.
  • Observations about how they read your document and how well they can follow the format.

This should be the basis of any revisions needed to the document.

Focus Groups (qualitative) Before rewriting an old, usually lengthy, document

 

 

 

 

 


_______________

After rewriting to compare the format of different versions of a document

  • Information about how readers feel about the old document; what they like and don’t like.
  • What information they need that they don’t have.

This can be a basis for how you rewrite the new document.

___________________

When used to decide on format, the content should already be protocol tested. And the content should be the same for each version.

Control Group (quantitative) After protocol testing and revising a document
  • Data about how many people did what you wanted.
  • A comparison between the old document and the new document

This information can help demonstrate the success of the rewrite.