In this paper  you'll see how to design and conduct surveys. You will also be introduced to the principles of questionnaire design and layout.

On completion of this section you should be able to:

-        describe the principles of survey development

-        list the major types of survey that can be used to collect social and economic data

-        explain the purpose and structure of pilot surveys

-        outline the principles governing the design of questionnaires

-        describe the major components of questionnaires, and how they can be layed out to create effective questionnaires

One major problem that we face in developing and managing a survey is that there are key points in the process where we may 'lose control' over the data collection process. An ill-conceived survey is frequently caused by lack of control during the development process. There are a number of critical stages that we go through before making observations or asking questions.

The exact goals of the survey should be clear at the start of the development process; no matter how well we design the 'measurement instrument' (such as a questionnaire), we will probably not achieve those goals.

This process should be made easier if the survey is part of a hypothesis-testing program, using the deductive methodology discussed in earlier chapters. Just as in experiments the ability to properly test the hypothesis depends on making the right measurements with the right instrument, in surveys we have to make the right observations, or ask the right questions. In a complicated survey this might not be easy to guarantee. If we are carrying out a survey on behalf of a client, we should also make sure we are able to collect the specific data the client wants from the survey.

How do we choose the right survey method? One thing that affects the choice of survey method is the need to make the data being collected match data already available. If we have secondary data that we want to use to provide verification for the primary data, we must collect specific data to support this.

When we choose a method, we have to take into account how we are going to actually carry out the survey. In principle we choose a particular method (the 'right' one) and then do it; in practice we often 'cut the suit to match the cloth'. It may be necessary to use a less desirable technique on the grounds of feasibility and practicality. To know which type of survey to use we need to be fully aware of their comparative advantages and disadvantages.

In direct observation we record events during an experiment or record behaviour. In the observation of human attitudes and behaviour (as in psychology or sociology) this process is complicated by the need to inform people of their participation in the process (we have no right to 'spy' on people). If we make direct observations of people without their knowing it then we are doing an experiment upon them without their agreeing to participate. Unfortunately, it is very likely that, when people are aware of the observation process, they will make subtle changes (often unconsciously) to their 'normal' activities.

Diaries

A tempting form of data collection, particularly when carrying out longitudinal surveys, is a diary system. Over a period of time (usually days or weeks, but in some cases over a period of years) the respondents themselves record data about what they are doing, or about what they watch on television, or what they buy. Until recently all television and radio ratings were conducted using diaries. The system is easy to administer (hand out the diary and collect it some time later) and hence cheap, but is largely restricted to behavioural (as opposed to attitudinal) information.

The major problem with using diaries is that it is very difficult to obtain any independent verification of what is recorded in the diary; only the respondent knows whether or not it is accurate. Some may give answers that they think will please the organisation conducting the survey (particularly if they fear or respect it), whilst others may give incorrect information specifically to spite the organisation. Many organisations that use diaries in this way rely on incentives (such as free gifts) to persuade respondents to provide accurate entries.

A widely used survey technique in social surveys is the interview. There are basically two types of face-to-face interview: the impromptu and the scheduled (we will deal with surveys which are based on interviews 'at a distance' shortly).

Impromptu interviews can take place wherever there are people who are likely to have an opinion of, or knowledge about, the survey topic. Shopping surveys might be done in a shopping centre, recreation surveys at the beach, and so on. As well as the right location, we should also consider the time of day at which such interviews are conducted, as the availability of the right type of respondent will vary during the day (or night).

Scheduled interviews are usually conducted at home or at work. If possible it's preferable to use scheduled interviews: people often feel uncomfortable when stopped in the street, but are more uncomfortable in their own home. The general principle is that if you get people on their own 'home ground' they will answer a larger number of questions, and give more detailed answers. On the other hand, such interviews usually demand greater mobility on the part of the interviewer, and hence are more costly to administer on a 'per interview' basis.

Interviews are not necessarily conducted by asking the interviewee a series of questions in strict order (question one, followed by question two, and so on). An interviewer can often get more useful information if he or she continually reorganises the sequence of questions to react to the interviewee's responses. The interviewer has to be prepared (and allowed) to blend the questions into a less structured discussion, but this can normally only be done when interviewing people at home or at their place of work. To generate such a discussion the interviewer must understand the questions completely and know them extremely well. The recording process must also be unobtrusive; it may be more effective (if the respondent agrees) to record the discuss on tape and transcribe it later, rather than ritually filling in a printed survey form.

There are a number of problems that may affect the usefulness of interview results. These are the most significant:

There may be bias (conscious or unconscious) in the selection of interviewees. Given that all participants will be volunteers, the interviewer may be less encouraging to some potential respondents than to others, or act in a more 'threatening' manner, or be less persuasive when requesting an interview. Male interviewers may 'intimidate' some female candidates, thus reducing the likelihood that they will agree to participate. All of these circumstances may change the characteristics of the sample; quota sampling is often employed to try to minimise these effects.

There may be bias in the question process. The interviewer may present certain questions in a way that is likely to elicit a particular answer. When recording a response the interviewer may 'filter' the response in a manner that changes the meaning the interviewee intended it to have. The commonest response to this problem is to give the interviewer little or no latitude (in wording, phrasing or inflection) in how questions are asked. The use of closed questions (the design of which you'll examine in a later chapter) is considered likely to reduce recording bias.

There is a recognised tendency for interviewees to give the answer that they think the interviewer would like to hear. Again, the use of more structured interviewing techniques and application of closed questions are thought to act as a counterbalance to this problem.

The interviewer may make straightforward errors in asking the questions or (more commonly) recording the answers. Several measures can be taken to minimise this problem:

training interviewers intensively in the general principles of interview conduct

ensuring that they have an understanding of the overall structure and aims of the survey

ensuring that they are fully briefed on the detailed format of the questions that need to be asked

applying appropriate measures during the pre-processing phase of data analysis (which we will cover in a later chapter) to cross-check results in order to look for possible errors

Interviews, like all voluntary survey techniques, are prone to variable response rates. The response rate of any survey is the proportion of the total number of people who were approached to participate, and those that actually participated; it is usually expressed as a percentage. Again, quota sampling is often employed to deal with this problem (keep asking until enough agree to participate), but at considerable potential cost.

As already suggested, face-to-face interviews can be a time-consuming (and hence costly) approach. Scheduled interviews, for example, have to include the time taken to arrange the interview, the travel time to and from the location, any time spent waiting for the interview to be available, and the actual interview time. In some cases it also includes the time required to re-visit the interviewee to check some answers, resolve some inconsistencies, or complete the survey if the first interview had to be abandoned for some reason. On average, a complex survey employing a scheduled interview system might only obtain one completed interview per interviewer per day.

Despite these problems, interviews are widely used where more complex issues than 'what soap powder do you buy?' are being investigated.

If we consider efficiency in surveying to mean 'completing as many surveys as possible in a given period of time' it is undoubtedly true that the most efficient forms of survey are mail and telephone surveys. Not surprisingly, many organisations use such surveys, and they are used to collect all types of data, from political opinions to shopping patterns. In most cases this is because the efficiency factor is considered to outweigh the well-known deficiencies of these techniques.

Mail surveys are conducted, as the name suggests, using the post. The organisation supplies potential respondents with a printed survey form and they (hopefully) fill it in and send it back. The simplest (and least expensive) version is where the organisation posts out the surveys and the respondents post them back, invariably in a pre-paid or stamped envelope supplied with the survey.

Alternatively, a member of the surveying organisation delivers the surveys and returns at an arranged time and date to collect them. Whilst obviously more costly, this method is considered (as we shall see shortly) to give worthwhile improvements in response rate. Most people will throw an unsolicited survey in the rubbish, but are more likely to complete it if they have agreed to do so personally, and know that someone is going to collect it.

Various compromise arrangements exist, including using follow-up letters and telephone calls. Again, you'll examine the impact of these on response rates in a later section.

Most surveys include a 'covering letter' to help potential respondents understand the purpose of the survey, and to persuade them to participate. Some survey veterans consider that an 'official-looking' covering letter (leading some respondents to believe that the survey is in some sense obligatory) will markedly improve the response rate. Others feel that incentives (small 'giveaways', or a chance to win larger prizes) are worthwhile. In each case, the aim is to increase the response rate and (less significantly) maximise the correctness of responses. The major advantage of mail surveys is their high 'penetration' (almost everyone has a mailing address, even if they don't have access to a telephone) and their low cost, which is really dependent on the exact method of administration.

The major disadvantages of mail surveys are their dependency on the reading and comprehension skills of the respondents, and the low initial response rates. It is considered absolutely critical to carry out some form of follow-up with those who don't respond. Of course, the more follow-ups one does, the more expensive it becomes and the more resistance occurs. There is a rule of thumb in a mail survey that 40% is a reasonable response for a first mail out. If you send the non-respondents a reminder (after a suitable period) you might get responses from 40% of the 60% who didn't respond the first time. So now you have a total 64% response rate. If you contact the remaining 36% again, and you get 40% of those to respond, you will then have about 75%, and so on. (After nine reminders you get - in principle - a 99.4% response rate, but you will have certainly worn out your welcome long before this point!) This is summarised in the following table:

Stage	Response Rate
First Mail	40%
First reminder	64% (40% + 40% of 60%)
Second Reminder	78% (40% + 40% of 60% + 40% of 36%)

In practice, if the survey is well administered, after three reminders you may get a response rate close to 80%. Some people - for various legitimate reasons - will always say no.

Telephone surveys vary enormously in quality. At one end are the statistically sound opinion polls carried out by organisations such as Roy Morgan, Gallup and ANOP. Using careful organisation and extensive background research they work with surprisingly small but very representative samples. At the other end are the useless and misleading 'phone ins' conducted by parts of the mass media that pretend to discover public opinion by asking people to ring in and record their 'vote'. It says something about the structure and quality of such surveys that almost anyone can guess the answer in advance.

All 'real' telephone surveys have a common structure. A list of potential respondents is drawn up, based on sources such as business directories, credit lists and membership lists of professional or social organisations. Each person on the list is contacted by telephone until enough have responded (hence the use of quota sampling). Once they have agreed to participate they are asked a pre-defined set of questions, usually quite simple in structure (such as multiple choice). The interview is unlikely to last for more than a few minutes, as many people will not participate if they think it will take more than that amount of time.

There are major advantages to this technique. It is a low cost method: the only costs are the telephone call (usually a local call) and the wages of the interviewers. Given the wide distribution of telephones in the western world, the method has high penetration.

But it is not perfect. There are still marked variations in access. Not everybody has a telephone, and the disparities are not random, but associated with variations in economic and social status. As a result, telephone surveys lead to under-representation of the economically disadvantaged members of society (the unemployed, single parents, pensioners, and so on). The consequent effects on the sample results are an example of bias due to non-coverage.

Even among those with telephone access, contact is not guaranteed; people may not be home when the interviewer calls. Interviewers can't expect a positive response if they ring at strange hours of the day or night. Anyway, some people simply don't like answering questions on the phone; today, many are suspicious of what they may consider the hidden motives of all unfamiliar callers.

Sometimes the telephone discussion leads to problems of mutual comprehension between the interviewer and the respondent, particularly where one is less fluent than the other in the language being used. Telephone surveys can only really employ the simpler type of questions, particularly multiple choice or yes/no questions; you'll examine the limitations on these in the next chapter.

There is usually a high degree of outright refusal to cooperate; people who might 'give in' and agree to participate if asked directly may be emboldened to say no and hang up when approached over the telephone.

Response rate

Our concern with response rates is linked to the effect of low rates on the representativeness of the sample, and the consequent accuracy of the population estimates. Anywhere you get less than 100% response rate is in some sense a 'threat' to the accuracy and usefulness of the survey; the lower the response, the greater the threat. As you've seen, maximising the response rate is a problem with all voluntary surveys. In general terms, the less the survey uses face-to-face questioning, the harder it is to get a high response rate.

One of the most overlooked components of a successful survey is the role played by the pilot survey. Once we have decided the overall aims and structure of the survey we should 'trial' the system with what is called a pilot survey.

Pilot surveys can serve several functions within the overall survey process. As you saw in the last chapter, a pilot survey is one method of obtaining an estimate of the population variance. The major function of the pilot survey, however, is to help 'tune' the proposed process for the main survey. We do this by using the pilot survey to find out whether the survey is going to be successful; that is, will it achieve an acceptable response rate, and provide reliable data on the relevant topics? To do this we need to find out several things from the pilot survey:

Is the survey too large or too small? If it is too large (there are 'too many questions') we will generally get a lower response rate, and there will be many unanswered questions in those surveys that are actually completed. If it is too small, we will not get the information we need to test our hypotheses, no matter how high the response rate.

Are the questions going to yield the information we require? As a simple example, if we want to know whether people are in favour of a particular proposal we will have to have a question that either directly asks about this, or a question that can yield this information during the processing of the survey results.

Is the survey layout clear and effective, or is it misleading? Do respondents get confused and, for instance, fill in sections that they are not supposed, or leave blank questions they should complete?

Are the instructions to the respondent (and to the interviewer) clear and unambiguous?

Are the people who are going to carry out the survey clear on the aims and methodology of the survey? Has their training been adequate?

Are there noticeable variations in the responses collected by different interviewers? Do some need more training? Will this need to be factored into the processing of the results from the main survey?

Administration

In administering the pilot survey we need to consider the following issues:

What is the optimum size (number of respondents) for the pilot sample? If the pilot sample size is too large it may waste resources that should be applied to the main survey, but if it is too small, potential flaws may not be missed.

Should the results of the pilot survey be used as if they had been collected during the main survey? In principle this is undesirable, as it implies that no changes will be made to the main survey as a result of conducting the pilot survey. In practice, however, the temptation to use the data collected during the pilot survey is often too strong.

Should the pilot survey be conducted (delivered) in an identical fashion to the main survey? If it is not (perhaps for logistical reasons) how sure can we be that the conclusions that are drawn from the pilot survey really apply to the main survey? It doesn't follow that a mail survey has to be piloted with another mail survey, although it usually will be. You can even learn something (but not much) by piloting a face-to-face interview system with a mail survey.

It is important to recognise just how significant the pilot survey can be in the overall success of the full survey. How we conduct the pilot survey, and whether we learn the right lessons from it, are major determinants of the quality and effectiveness of the final survey.

 

Questionnaire design

One of the foibles of human nature is that we expect - against most available evidence - that we will create something worthwhile every time we approach a problem. This expectation seems to increase with the (apparent) simplicity of the task. This is clearly true of questionnaire design. It seems so easy to write down a series of questions; how hard can it be?

The answer is: quite hard, really - if you want to do it well. The design and construction of questionnaires is a predominantly subjective process that is largely guided by the experience of the designer; the more questionnaires you have created, the more likely it is that the next survey will be closer to 'perfect'. Mostly this is because an experienced designer should (hopefully) learn from his or her mistakes, and will avoid repeating them.

Anyone who deals with questionnaire design develops, by personal experience and critical analysis of other people's designs, what one might call design guidelines (perhaps even rules) for good questionnaires. Everyone will have slightly different principles, so those presented here are not going to be universally accepted.

So what are the basic issues in questionnaire design?

Some surveys are as little as one page long, others may be twenty pages or more. For longer surveys we have to be convinced that the size is appropriate and will not have an unduly adverse effect on the response rate.

How much background information should be included in the survey, or in a covering letter? Of course, we must always include some instructions for the respondents, and possibly some for the investigator. If properly worded, such instructions should ensure that the survey is completed without major mishaps. But we obviously don't want the instructions to be so complex that reading them is an exercise it its own right.

What number and types of questions should be used, and how should they be mixed? As you'll see in the next section, there are two basic 'species' of question (open and closed), and numerous 'varieties' of closed question. Each can provide us a different type of data, so we will want an appropriate mix of them.

What constitutes an 'appropriate mix'? You want the respondents to give you the maximum amount of relevant information, if possible in their own words, but you have to place certain constraints on this process - mostly related to the physical space in the questionnaire. You can give them space for their own words but you will need to balance this with the effective use of closed questions, especially if you want the data that you collect to be comparable with that from other surveys.

In what order do you ask questions? If the survey requires basic demographic data (such as age, gender, marital status and occupation) do you place these questions at the beginning of the survey, or at the end? Generally the preference is for the end.

How do you arrange layout of the questionnaire so that respondents answer the questions they are supposed to answer, and not the ones they aren't supposed to? How do you actually phrase the questions? Will people respond more favourably (and more accurately) to questions posed in a 'chatty' style, or to those with a more formal phrasing?

The layout of the questionnaire can have a significant impact on how respondents react; too little attention is given to it. If you spread fifteen questions over fifteen pages some people will not fill it in simply because they think it will take too long to complete. If you cram all fifteen questions onto a single page, other people will find it too difficult to follow.

Should the questionnaire be broken into sections, where each section deals with a clearly defined topic, different from those in adjacent sections? At the start of each section you will need to give some general information about the completion of that section, but just how much information is appropriate?

Should questions be numbered, and should the numbering sequence indicate any section structure (Q1 .. Q20 or Q1.1 .. Q1.5, Q2.1 .. Q2.8 for example)? Do you want people to answer all questions, or do you want them to skip Q8 if it is not applicable to them?

All of this can be summarised as a simple problem: the questionnaire should be as clear, detailed and unambiguous as possible, but you don't want to insult the intelligence of most of the population. The basic principle for solving this problem is to balance compactness with an impression of legibility and spaciousness. The language used in the instructions and the actual questions needs to be concise without being obtuse, relaxed without being foolish. The respondents must not feel that they are being patronised, but they must also feel that the survey is serious and worth the effort of their completing it.

 

Out of the preceding discussion we can present some general guidelines for the layout of questionnaires.

There is a balance in the use of whitespace (between questions and sections) to give improved readability, without unduly increasing the apparent size of the questionnaire.

The questionnaire - particularly if administered by mail - needs to have a preamble that explains the overall aim of the survey; this can be part of the covering letter, or at the head of the actual questionnaire.

There should be general instructions to the respondents, placed usually at the beginning of each section.

It is usually a sound practice to number questions and, if the questionnaire is divided into sections, to have the section designation as part of the question numbering system.

There should be specific instructions associated with each question to aid in the correct completion of that questions. These might include phrases such as "Please tick one box only", "Tick as many boxes as necessary", or "Put a cross at the appropriate place on the line".

Wherever appropriate the respondents should be able to bypass questions (or whole sections) that are not relevant to them. This can be achieved by using filter questions, combined with instructions such as "If you answered YES to Question 8 please move directly to Question 12 (that is, do not answer Questions 9-11)".

If respondents are unsure about whether to answer a question, or which answer is the most appropriate, they should be provided with a "letout", such as "Don't Know" or "Not Applicable". When a large number of respondents choose such options, it is time to examine whether the question is badly worded, or int he wrong place in the questionnaire.

Question Wording

At the most detailed level, considerable attention should be given to the actual wording and terminology used in each question. Again, certain simple principles can be prescribed:

Most wording problems are associated with unintentional ambiguity; that is, the respondent infers that the question is asking about something other than what the designer intended it to.

It is possible to accidentally word questions in such a way that the response is largely controlled by whether the respondent has a similar cultural, educational or ethnic background to the designer of the survey, rather than by what he or she really thinks about the topic.

We commonly make linguistic assumptions about the respondents' vocabulary, grasp of grammar, and so on. Again, the answers may reflect this mis-comprehension rather the `true' opinions of the respondent.

Double negatives are widely used, partly in error,but occasionally they are used deliberately for emphasis. In survey questions they will always generate confusion.

The most difficult 'wording' problem is really a structural problem in the design of the questionnaire, caused by trying to combine questions in order to 'simplify' the process. This is the tendency to use 'double-barrelled' questions. For example, we might ask a question that says 'Are you a member of the Labor Party and did you support the party in the last election?' There are four possible answers to this question (Yes/Yes, Yes/No, No/Yes, No/No) because it is really two questions masquerading as one. Whatever answer we get, we will never know which of the four the respondent really meant.

The biggest mistake in constructing any questionnaires is to assume that the aim is to 'get people to tell you what they think'. Attractive as this approach may be, it has critical drawbacks:

It will have a dramatic impact on the time to administer the questionnaire.

The results will be difficult and time-consuming to code and analyse

Each respondent's amswer will be almost impossible to compare with those of other respondents. If one hundred people give us one hundred different answers, how can we generalise and draw conclusions from them?

This doesn't mean that we never use questions (which we call open questions) that elicit such responses, but that we must combine them with questions that provide us with the `skeleton' around which the answers to the open questions can be placed. Open questions are useful in conjunction with more constrained ones, of the type we call closed (or structured).

Closed questions give the respondents a finite (usually small) number of choices from which they can select one or more. Combining this with space for the respondents to add comments if they want provides the best of both worlds. If we know exactly what data we need it is better to use nothing but closed questions. It is also much easier (as you'll see in the next chapter) to extract data from closed questions during the subsequent coding and analysis phases of the survey.

The archetypal open question has a series of lines (or a blank space) in which the respondents is encouraged to write, in their own words, how they feel about the topic in question. Here is an example:

Closed question provide a set of answers that the designer of the survey (based on prior experience and responses in the pilot survey) considers will accommodate the majority of potential responses. The question asked in open form above can be presented in closed form is several different structures. The simplest offers a list of choices, with the option (perhaps) of adding others if they are not found in the given list.

 

 

There are different varieties of closed (or structured) questions, depending on the way in which the respondent is asked to place their answers. Usually they must tick a box or place a cross along a line. Note that we can regard deciding which of the various types of structured questions to use as one the of the things that we would expect to investigate with the pilot survey.

Here are examples of the most widely used structured question layouts.

`Ticking Boxes'

A sub-variety of structured question is called tabular because it is organised in such a way that the answers can be tallied directly into a cross-tabulation format.

	Education Level
Age group	Secondary	Teriary	Post-tertiary
20-30
20-30
30-40
40-50
50-60
Over 60