TripleScoop Tip #7: 3 Steps to More Effective Reviewing and Revising Questionnaire Drafts

When I was on the client-side, one of my least favorite tasks was to review the questionnaire draft from my research partner. I always started the review process with the best of intentions--to simply review it against our objectives. But inevitably, within moments of reading through the draft, some grammar mistake or the way a question or answer option was worded, would trigger my Evil-Red-Pen-in-Hand-Editor mode, and I would start editing and rewriting the draft in great detail. I am certain it drove my partners crazy. I know it frustrated me with all the time and energy I put into it. And, most importantly, that focus on the details often overshadowed the need to evaluate whether the draft was meeting our overarching objectives.

This despite the great example my first manager set for me. I loved how he edited my drafts of surveys, proposals, or reports. He would just make broad stroke comments like "I don't think this word means what you think it means," or "this slide doesn't seem to be going where you want it to go" and then let me figure out the details to revise it. Not only were the results a hundred times better than when managers full-on edited working documents, I learned so much more in the revision process.

However, most times when researchers review a questionnaire draft, no matter what side of the client-supplier desk they occupy, the compulsive need to edit the draft into submission is strong among researchers as The Force is with Luke. So here is a 3-step process that I guarantee will lead to more effective questionnaire draft feedback and less editor-in-chief, self-induced stress and grief.

Photo by Matt Artz on Unsplash

1. First, Role Play

Start by taking the survey in role play mode, optimally verbally. If it helps, have someone on your team administer it to you. While you are going through it, make notes about things you notice--whether grammar, style, missing answer options, etc. But don’t edit or rewrite at this point. I find it best to have printed the survey so I can use a pen or pencil (not red ink though!) to make notes in the margins. This also helps me avoid rewriting or getting too in the weeds, as it is harder to write long-hand compared to typing in changes. These notes can be broad stroke thoughts like "this reads awkward," or "didn't understand the question," or you might highlight or underline industry jargon, misspellings, or other grammar issues. The point is that anything you notice that needs help, whether broadly or in detail, gets noted and not forgotten without having to take on the time-consuming task of editing. You can do that later, if you so desire.

2. Second, Track Against Objectives

Now that you have gone through the whole survey and have a good feel for it, go back through and match up questions or sections with your study objectives. I recommend using the printed version and a pen or pencil for this as well. Maybe I am old school, but I feel that this is visually better and more effective than comments on a Word Doc. I also like to note when a question exists that doesn’t address any stated objectives and whether it is a nice-to-know question, something that drives additional objectives we didn’t consider initially, or is just a meta-question--necessary to the administration of the survey whether to qualify respondents or make the flow of questions work. I also like print up the objectives and list the questions that address each one and a sort of qualitative note about how well they do so. With notes on the draft and the objectives listing, I then have two perspectives that make it easier to see where I am not meeting the objectives or where I might be beating an objective like the proverbial horse.*

3. Last, comment and/or edit

Now is the time to add your comments to the survey. I recommend using the Review feature and inserting comments. But it is less time-consuming and far easier to simply scan or take pictures of your now annotated printouts of the survey. (Assuming you feel your written comments are readable and understandable.)

And, if you really, really want to edit the survey, this is the most effective time to do that. Your editing will be much more informed because you are now familiar with the survey flow and cognizant of the objectives. The edits that are going to be most helpful will be centered on:

  • Identifying and marking any language that might be insider jargon

  • Missing answer options or redundant options

  • What missing questions need to address--not necessarily writing the questions.

  • Issues of flow

  • Misspellings or grammar mistakes you catch. (And if you prefer the Oxford comma or the other way of using commas, let your partner know.)

If you feel the style or voice of the survey is not what you wanted, rather than editing for that, I recommend you let your partner know and then describing as well as you are able what the style or voice of the survey should reflect. Then let your partner do the heavy lifting to integrate rewrite for style.

From my experience both on the client and the partner side, I much prefer to get broad stroke comments to set direction on revisions as opposed to edited and rewritten questions. As the initial writer, I am going to be much more familiar with the questionnaire details and better able to edit it fresh with those comments in mind than to try and decipher a whole series of edits and new questions and evaluate those against the objectives. (Yes, this is how scope creep often happens.) And, as a client, it is so much easier and less frustrating to just comment and provide good direction rather than spend all that extra time doing rewrites.

I do so hope this process makes your draft/revise cycle for questionnaires more effective and more enjoyable.

*No real horses were beaten in the writing of this tip.