Guide to Designing a Questionnaire

22 Oct 2009

A short guide, produced by Eoin O’Mahony, Social Researcher at the Bishops’ Council for Research & Development, to help parish and community groups with designing questionnaires

22 October 2009 | Irish Catholic Bishops’ Conference


Guide to Designing a Questionnaire
Suggestions or comments about this guide can be sent to [email protected]

Full text:

Introduction

Every year, there are a number of requests for technical assistance with a questionnaire or another kind of survey. While I am happy to assist in any way that I can, many of the same ideas arise repeatedly. A short list of these same features and ideas can be compiled.

In this short guide, you will find a number of steps, suggestions and tips when setting out a questionnaire. These are not so much tricks of the trade but first steps in putting together a project that should serve your parish or community group. There is a little bit of jargon involved but like any language, once you become familiar it will seem easier in the end.

Eoin O’Mahony
Social Researcher
October 2009

What is a questionnaire?

A questionnaire is a list of questions that can be used to gather lots of information (or data) about people, their opinions and their views on a subject of relevance. Questionnaires are generally in paper form but are increasingly to be found on the internet as well. As questionnaires gather data, they are tools by which people’s views and attitudes can be measured using numbers and text.

A questionnaire can be handed out to a limited number of people (e.g. doctor’s surgery) or to many millions (e.g. a census). In both cases, there is a single questionnaire that is given to everyone and all of the questions listed on the questionnaire will be the same for everyone who fills it in. That is not to say that everyone who receives the questionnaire will answer every question (or read it in the same way) but we can deal with that later. When a questionnaire is filled out, it is called completing the questionnaire. The person who completes a questionnaire is called a respondent.

Samples and representativeness

Very often, a survey will not ask everyone within an area or a community to answer a questionnaire. Using probability theory (the detail is unnecessary here), you can distribute a limited number of questionnaires to people and, within reason, expect that this smaller group’s views will be accurately reflective of (or representative) of the population as a whole. Here’s an example:

St Mary’s parish is interested in what the people in the parish think about the new Mass times. The times were changed at the request of the priest although not everyone liked these new times. To get some feedback on the changes, a questionnaire was distributed to the people of the parish. As there are over 4,000 people in the parish that could give a view on changed Mass times and the funds for this project are limited, the parish council decide to distribute 800 copies of a short questionnaire. This is about 20% of the total population. But which 800 does the council pick? If it picks all men then only men’s views will be counted; if it picks people from only one side of the parish, that does not give an accurate representation of the views of all of the people in the parish.

In selecting people to ask, the parish council can attempt to make the 800 people as representative of the entire parish of 4,000 as it can. For example, it should contain approximately 51% female recipients, 25% aged between 15 and 25 etc. if these are the proportions that are present in the 4,000 people in the parish. Because the parish council can never be sure that this is the right 800, there will always be a margin of error. What is this?

In opinion polls published in newspapers and other mass media, we sometimes notice that a poll has a margin of error of plus or minus 3% (or +/-3%). A political party might have a 46% opinion poll rating and this might be good news or bad news. In a poll with a margin of error of plus or minus 3% this means that the actual poll result could be as high as 49% or as low as 43%. Why is there a margin or error at all?

A margin of error is a statistical measurement of how close the sample that has been chosen is to the entire population. There are formulas available to calculate the margin of error but a general rule of thumb is:

The larger the sample drawn from the whole population, the lower the margin of error.

For all samples drawn, margins of error increase when the sample is small. This is represented in the table below:

Sample size Margin of error
96 10%
384 5%
600 4%
1,067 3%
2,401 2%

Source: en.wikipedia.org/wiki/File:Marginoferror95.PNG (last accessed August 31st 2009)

The margin of error can never be 0 unless you ask the entire population to complete the questionnaire. By asking everyone of the 4,000 in the parish to complete the questionnaire you have eliminated all margin for error that the results do not adequately reflect the views of the population.

You might also notice that decreases in the margin of error are negligible for larger samples. A large increase in sample size will not yield a significantly lower margin of error. In the real world, you (and those who read the results) only need be aware that this margin exists. A 3% margin of error is generally acceptable and should be aimed for in a survey, if the budget allows it. This means giving the questionnaire to at least 1,067 people – stratified in the same way as the population as a whole of course.

Even if you ask over 1,000 people to complete the questionnaire you will not get 1,000 returned to you. The rate of return does not affect the margin of error, as you are only responsible for distributing the questionnaire, not its return. Of course if you provide an addressed and stamped return envelope it does help the return rate but the margin of error is not affected.

Questionnaire design

Now that you have decided who will receive the questionnaire, you need to figure out what you want to ask people. There are many different types of questions available and these types can all be placed within one questionnaire. A general rule for putting together a good questionnaire can be remembered if you employ the general rule of RIRO: Rubbish In? Rubbish Out. If you are interested in how people have reacted to a change in the types of services provided by the parish then ask them directly about these changes, as succinctly as you can.

With the group who is compiling the questionnaire, there are a few important questions to answer before you begin to write questions for your questionnaire. Below are a few of these questions about question length and style.

Purpose

What kind of information do you want to collect? If you are interested in assessing people’s opinions about an event or an occurrence, ask directly about them. The answers you receive will only be as good as the questions you ask.

Example: avoid “what do you think of event X?” and go instead for something like “Event X took place on March 20th. Please tick one of the options below to indicate your opinion of this event.”

Clarity

How can you make your questions as clear as possible? Phrase your questions with the least amount of ambiguity. If you are unclear in phrasing your questions, respondents are less likely to want to answer it and just move on or give up entirely.

Example: avoid “do you like chocolate?” and go instead for something like “On the scale below, please indicate how much you like this type of chocolat

Leading questions

If respondents feel like they are being led along a certain way of questioning, they are less likely to complete the question. There is an old example of a leading question that is still doing the rounds.

Example: avoid “In what way did you blackmail your work colleague

Needless to say, your own questions will be less dramatic than this but here is a further example of a leading question:

Example: “How good do you think the parish is at preparing children for the sacraments?” This might be rephrased to be asked as “When it comes to preparing children for sacraments, how would you rate the parish?” Respondents would then be presented with a number of choices from which to pick

Question Flow

Respondents are more likely to complete all of the questions when the questions that require more consideration appear later in a questionnaire. Beginning a questionnaire with some simpler questions like their banded age, their gender or their marital status is often better than asking questions that need more thought.

Example: avoid “Q1. The government has cut the education budget by 10% for 2010. Do you agree or disagree with this? Q2. How would you describe the area that you live in?” and go instead for easier demographic questions before asking the more thoughtful questions later on.

Redundancy

Do not ask any questions for which you do not need to know the answer. Asking redundant questions makes a questionnaire unnecessarily long and only frustrates respondents. Think about how much you like to complete questionnaires.

Privacy and anonymity

It is important to respondents that the information they give is treated with the respect with which they gave it. People give their views and opinions voluntarily and in offering their own thoughts, care should be taken in how these are handled. Every questionnaire drafted should contain an assurance that respondents decide what they complete and what they do not. They should also be assured that all of the information they give will be treated in private and in confidence. It helps to state that only you, as the survey’s coordinator, will view and have access to the views given.

The principle of informed, active consent is important at this stage. Although a potential respondent may be sent a questionnaire for completion, it does not mean that they must complete and return it. All respondents should be made aware that their cooperation is voluntary, actively given and private. Respondents should also be made aware that any personal information (addresses, names, dates of birth) that is given in the questionnaire is bound by the relevant Data Protection legislation. Details of Data Protection legislation can be found at:

For the Republic of Ireland

http://www.dataprotection.ie/ViewDoc.asp?fn=%2Fdocuments%2Fresponsibilities%2FOrganisationsHome.htm&CatID=3&m=y

For Northern Ireland

Question types

Based on the principle of RIRO above, the types of questions you ask will be determined by what exactly you would like to find out. If you are interested in the ways in which a change in Mass times has changed people’s views, then you may not need to know people’s age. However, if your project is about how men and women may react to this in different ways then you will need to ask people if they are male or female.

Phrase questions concisely and with accuracy, remembering the guidelines from above. It is also important to state that the overall length of your questionnaire will affect the response rates. Shorter is better of course but make the questionnaire as long as it needs to be to find out all you need to know. There is a careful balance to be struck between brevity and finding the ‘right’ kinds of information.

Bear in mind too how you would like to analyse the data. Data that can be coded numerically is generally easier to analyse than text based data. If you have collected numeric and basic text data, it can be analysed using a programme called SPSS – a facility available from the Council for Research & Development in Maynooth.

Below is a list of sample question types with an example provided in each case.

1. Simple Yes / No
Are you Female or Male?

2. Filtered Yes / No
Were you born in Ireland?
If No, in what country were you born?

3. List
Are you

18 or under
19 to 40
41 to 60
61 or more

4. Likert scale
The parish council is proposing that the 9.30am Mass will be replaced by a 10am Mass. Do you:

Strongly disagree Disagree Don’t know Agree Strongly agree

5. Scalar
The parish council is proposing that the 9.30am Mass will be replaced by a 10am Mass. Please mark the number that corresponds to your level of agreement / disagreement with this proposal.

Agree Disagree
1 2 3 4 5 6 7 8 9 10

6. Free text

The parish council is proposing that the 9.30am Mass will be replaced by a 10am Mass. Please let us know what you think of this proposal.

Suggestions or comments about this guide can be sent to [email protected]