Survey Tips

Sample size

• Keep sample small
• Can tell respondents that they are part of a select group (appeals to scarcity)
• If using a sample of the population, you can administer multiple surveys, using different people, so that folks don’t get tired of being surveyed.

How many times do you contact folks?

• Send at least 3 reminders
• Change the day of the week and time of day that you send reminders, because students and professors tend to have times that they are more likely to respond.
• Be sure to remove those who have already responded from your reminder list.

Use a mixed-method of contact

• Use of both paper and email contacts has been found to be effective. For example, send a letter from the provost before the survey.
• Follow-up with a postcard
• A study has shown that sending a post-card before the survey has increased response rate by 9% (Public Opinion Quarterly)

Structure of email

• Subject line
  • Don’t use the word “survey”
  • Mention EMU to make your email stand out from SPAM
  • On the final reminder, one study found that having the subject line blank got the most responses.
• Body of email
  • Keep it short and use text only (not html because some SPAM filters look for html)
  • Put the hyperlink near the top of the email.
  • Include contact information out of courtesy.
• Techniques
  • Tell respondents that they are part of a select group.
  • Give a deadline; limited amount of time to complete.
  • Ask for their help (i.e., appeal to their helping tendencies).
  • Tell them to let you know if they don’t want to be contacted again.
  • Give a rough estimate of completion time.
  • Promise confidentiality, if you can. But, don’t overemphasize it. Overstating confidentiality can decrease response rates and leads to suspiciousness.
  • Make sure that you can defend the content of your email to the Provost and President.
• Things that typically cause emails to go into ‘Junk Mail’ as SPAM
  • Capital letters
  • “from” name contains underlines or numbers (e.g., sally_52)
  • A large number of blank lines or spaces in the body of the text
  • “Click here” or “Click below”; it is better to use the word “submit”

Response Rate

• Rule of thumb – at least 50%
• If your response rate is low, check to see how the demographics of your sample compare to the population.
• Send follow-up emails not more than 1 week apart.
• After the first 2-3 days, the response rate typically tends to drop off.
• Once the response rate drops off, send another follow-up email.

Web Design

• Appearance
  • Have a short introduction at the top
  • Avoid drop-down boxes (they are easy to miss); use radio buttons or check boxes instead.
  • Make sure that the layout is vertical and that respondents don’t have to scroll horizontally.
  • Use plenty of white space and headings.
  • Number your questions so that people don’t get lost.
  • Use yes/no instead of having them check all that apply (will be easier to use the data and forces respondents to read all of the options).
    Example:

				Yes		No
		Library?
		Athletic Facilities?
		Dining Hall?

• Length
  • Don’t ask demographic information if you already have it.
  • If it is long use multiple pages (downloads quicker).
  • Don’t force responses.
  • Keep the survey under 20-30 minutes to complete.
• Salience of the survey topic
  • This is one of the best predictors of survey response.
  • Put interesting questions near the top and boring questions near the end.
• Open-ended questions
  • Use of “other” often indicates that you have not formulated your responses well.
  • Keep short, for your own sake.
• End of survey
  • Make sure the “submit” button is visible.
  • Have a “thank you” page after the submit button.
    • Lets them know the survey is over.
    • Can also use it to direct them to another website, or collect another survey.
• Timeline
  • Start early, at least a month before you want to the survey to go out.
  • Developing a survey is a long process and there are often many refinements along the way.
  • Allow time for pilot testing.
  • Take feedback seriously.

Resources

• Don Dillman is the guru in this area and has published studies on web surveys.
• Public Opinion Quarterly (Hartzler Library has 1983 – present.)
• Web survey methodology website
• Social Science Computer Review (Hartzler Library does not subscribe.)
• Groves et al. (1992) “Understanding the decision to participate in a survey.” Public Opinion Quarterly, 56:475-495.

Modified from Steven Porter’s presentation for an Academic Impressions webinar on February 15, 2005.