http://www.virtualsurveys.com/papers/paper_9.asp  

                    

                                                                                                                                                           

                                               

 

                                                                                                                                                           

                                                                                                                                                           

            (First ESOMAR paper on Internet data collection. Published at ESOMAR/EMAC Symposium -November 1996)

Abstract:

The paper examines current and future Internet usage in Europe. It discusses how the Internet is beginning to be used for data collection in market research and its benefits over other methods. More specifically it shows the results of Internet Opinion Polls conducted prior to UK General Election in 1997. Not only did these polls show exactly the same trends on conventional polls during the election period, but they also accurately predicted the final outcome. The results clearly support the likelihood that the Internet will become an established method of data collection of the future.

Introduction

Usage of the Internet is doubling every year. There are currently estimated to be around 20 million worldwide users and this figure is predicted to reach 150 million by the year 2000. However in the UK, penetration levels are estimated to be just 2% of households and about 5% of people in business.

The Internet is a very recent phenomenon and many people are still only beginning to consider how it can be used. Back in 1995, we at SGA were convinced it offered potential as a data collection method. Searching the literature (and the Web itself), resulted in little published data on how to use the Internet for market research. However in October 1995, some of the first articles were published comparing email with postal surveys over the Internet.

Mehta and Sivadas (1995) set up a study which showed that email could generate high response rates and similar ones to postal surveys. They also found it to be significantly quicker. They received a half of their email questionnaires within three days compared with three weeks to receive a comparable proportion of postal questionnaires. They found evidence of higher quality of responses and also pointed out the significant cost savings of the method (mainly for convenience of despatch).

In another study, Tse et al (1995) conducted a survey by email and internal post in the Chinese University of Hong Kong. This showed a markedly lower response rate for email (6%) compared with mail (27%). They explain the lower rate in terms of:

fear of the new email technology
difficulty of completely email surveys
traceability of the respondent for a potential sensitive subject (attitudes towards business ethics/corruption).

The purpose of this paper is to show how the Internet can, and has been, used commercially to conduct research. It also aims to provide guidelines for future studies.

Objectives

The study was conducted as part of a project for the Henley Centre on Media Futures (1996). The objective of that research was to understand how early adopters of new technology were using the Internet and how their views differed from the population as a whole.

The study was to be conducted via email and SGA therefore proposed to extend the study with additional fieldwork using both conventional postal methods and website survey methods to compare key response factors, i.e. response time, response rate and quality.

Method

As will become clear, conducting research over the Internet has many pitfalls and problems. The original database of email names was purchased from a publisher of a UK based Internet magazine and comprised both subscribers to that magazine and others who had interacted with it within the last year. The database purchased was supposed to be the subset of 5000 users who had specified their email address. However the database turned out to only consist of just 3,700 names, but that only 1,221 of these actually had email names!

Instead it was therefore decided to use all respondents that had specified an email name for the email leg of the test and complete the rest of the survey postally. (It was realised that those with an email name may have been demographically different to the remainder of the sample, but practicalities of the project necessitated this approach and its potential implications are discussed further below.)

The original design for the Website questionnaire leg of the test was to initially contact respondents by email. Due to the lack of email names, this part of the test was abandoned and instead it was decided to check out the effect of additionally offering postal respondents the opportunity to complete the survey on-line at SGA's website, if they wished to.

Table 1 therefore summarises the three methods examined for the research.

 

Care was taken to keep as many factors the same between the two types of survey. In both cases the same questionnaire was used (i.e. a printed email questionnaire was used postally), the same incentives and the same time schedule.

However there were some differences between the methods. In the case of the email group, respondents initially received an email asking them to participate in the survey and highlighting the incentives. To take part in the survey they had to simply "reply" returning the email back to SGA. SGA then emailed them the full survey. This method was adopted to reduce the potential irritation of receiving what could have been perceived as a large "junk" email.

One of the advantages of using email is that all responders (and therefore non responders) are identifiable. It was therefore decided to send a reminder email to all people who had requested the survey but not completed it. This was sent out after 7 days. In order to assess the effect of this the results have been analysed by those responding before and after it was received.

Sample

The sample comprised of the UK Internet users who were readers of an Internet magazine. As already discussed the database only comprised a limited number with email addresses. The sample size for each of the cells was therefore as follows:

Email 1221

Postal with website option 1769

Postal with no website option 1000

In spite of the database comprising only contacts within the last year, a relatively high proportion (35%) of the email names proved to be invalid, with a result that just 795 emails were successfully sent. The reason for the high failure rate was due to a combination of factors:

mispellings/incomplete email names
respondents changing service providers/addresses
respondents giving URL's not emails
fictitious emails (e.g. pope@vatican)

Questionnaire

The questionnaire covered questions on Internet usage, attitudes towards the Internet/technology and detailed questions on the time spent on the Internet and other activities. It was a very long survey to complete taking between 20-40 minutes on average. In its printed form it covered 12 pages.

Because of the constraints of email, the survey had to be typed in a basic ASCII text editor. It could not include any non standard characters (including £), bold, italics, or font sizes/changes. Furthermore care had to be taken not to exceed a 70 columns width to avoid word wrapping. In addition respondents had to be instructed to ensure that their email viewer was set to a fixed spacing font to align all the boxes.

An example of the style of the final survey can be found in Table 2 (Image not available).

Note, a few problems were also encountered with email file size (34k). Some email systems have been set up to not permit large emails to be sent/received. In these cases, the survey was split into two parts.

Incentives

Incentives are necessary on any self completion project to enhance response rate. In view of the length of the questionnaire and the sample, two incentives were offered. All respondents were offered £2 for completing the survey. In addition a prize draw for a £400 Widescreen TV was also included for the first 250 people to complete the survey. Note, instead of receiving the £2 themselves, respondents were also given the opportunity to donate this to charity. This proved a very popular option and presumably fitted very well with the altruistic nature of the Internet culture.

Results

1. Response Time 

$ Including 7% received after a reminder sent out after 1 week.

The findings clearly show a major time advantage for email over postal methods. Nearly two-thirds of all email questionnaires were received at SGA within three days of the original contact. This includes the time taken to reply to the premail and for SGA to mail out the survey. In the same time, only 1 out of the 437 postal questionnaires had been returned.

The average postal response time was 11 days. Part of this length was caused by interruptions to the postal supply due to industrial action during the fieldwork. (Indeed, it took some letters 6 days to arrive having been posted first class!) However even without this effect (estimated to be about 2-3 working days delay), email would still have exhibited a significant time advantage.

Furthermore, virtually all replies to the premail were received within 2-3 days and none after 7 days. This suggested that reading of electronic mail boxes may not be as infrequent as sometimes suggested, e.g. by Tse et al (1995).

2. Responses Rate

The overall response rate obtained from email was quite reasonable (13.5%). Furthermore, it is likely that such higher rates could have been obtained with a shorter simpler survey.

Table 4. Total response rate

$ Amongst those receiving emails (i.e. excluding failed mail). The response rate from the email survey was directionally (pvalue = 0.258) lower than that obtained from the postal survey (15.4%). This slightly lower level of response by email was in spite of the fact that this group had a reminder. However, as can be seen from Table 5, its inclusion had relatively little effect on response rate. Without it, response rate would have been at least 12.6%.

Table 5. Results of email mailing

Therefore the results do show, unlike previous studies, that very similar response rates can be obtained for email versus postal methods.

Including the on-line website option for completion of the survey enhanced the overall response rate slightly, 15.4 to 17.0 (pvalue = 0.276). It was interesting to note that relatively few took up the option to complete the survey electronically and just 18 questionnaires were received by this method. However it is possible that its inclusion encouraged a few more to complete the survey that would have not otherwise done so. Lack of access to the Internet at the location of receiving the postal questionnaire may also have had an influence. (Note, some were posted to work addresses and some to residential addresses.)

3. Response Quality

Response quality is very difficult to quantify because it is largely a function of the amount of effort and thought devoted by respondents in completing the survey. However McDaniel & Rao (1981) have proposed it can be quantified along three dimensions: item omission, response error, and completeness of answer. Table 6 shows the results of analysis of these dimensions.

Table 6. Quality of response

 

Two full open ended questions were included in the questionnaire on the benefits of the Internet and improvements sought. A larger number of comments were coded from the surveys completed by email. (Furthermore, we even had one email respondent who wrote a mini-essay in response to Q.11). These findings are in line with the results of the Mehta and Sivadas (1995) survey. However, its effect is not conclusive from the current survey as the samples responding in email and postal surveys differed in their involvement with the Internet. Those responding via email were more heavier users and biased more towards those in academic establishments. It could therefore be hypothesised that their knowledge of the Internet may have been greater and so they may have had more comments to make about the Internet.

Significantly higher levels of response completeness (i.e. any answer to the question) were obtained for the email survey. This again provides evidence of higher quality for internet surveys. The average level of item omission (or question left blank) has been analysed. (This was done for the first three questions only as it could be argued that most other questions, no answer was a valid response for some people.) This analysis showed fractionally higher levels (pvalue = 0.465) for those responding by email. This result may have been linked to the inherent difficulty of completing a survey in email format.

 

 

 

 

4. Research Cost

Table 7. Estimated administrative costs of both methods (per 500 responses)

                                         Postal  Email

(Total mailout required)         (3247)  (5705) $

Questionnaire set up cost     £200    £200

Printing cost                        £1000  -

Despatch cost                     £1150  -

Telephone calls                   -          £25

Despatch administrative time £400    £150

Total cost                           £2750  £375

$ Including 35% failed mail. The cost for administering the survey by email was substantially less than for postal methods. The main saving was within printing and despatch costs. Table 7 gives the hypothetical administrative mailing costs of conducting the survey postally and by email. It is estimated that it costs seven times as much to carry out the mailing by post.

Furthermore, in comparison with the alternative of carrying out the survey by telephone, the total research costs were less than half.

Future Recommendations for Internet Surveys

1. The survey has provided clear evidence that the Internet can be used as an effective data collection method. Furthermore, it can offer significant timing and cost benefits.

However, using email may not be the optimum method for conducting Internet surveys. SGA has conducted other surveys where respondents are contacted by email and asked to visit our website to complete the survey. This not only speeds up the response time but also provides a better questionnaire interface. See Table 8. In terms of response time, SGA recently received 78% of its website surveys within 24 hours of requesting participation. Furthermore this method was clearly popular with respondents as most chose it over replying by email.

 However if this method is adopted, the option to complete the survey by email must also be given for the following reasons:

Not all browsers work with HTML questionnaires (Internet Explorer 3.0 doesn't!)
Not all people contactable by email have access to the WWW. Indeed, in many businesses, companies have (currently) restricted usage.

2. The survey has also highlighted the pitfalls (e.g. accuracy) of using email lists as sampling frames. Furthermore, it is almost impossible to obtain such lists commercially. There is no global directory of email names. Companies such as Bigfoot are trying to create one but will not supply them commercially. Currently no UK Internet Service Provider rents out lists. As a cautionary tale, note that Compuserve do rent out a postal list of UK subscribers and indeed SGA did try to use it on this survey. However at the final hour, Compuserve USA banned SGA from renting it!

The clear way forward is to develop panels of Internet users willing to take part in future studies. A number of companies, including SGA, have panels of Internet users. From these it is possible to speedily contact samples of eligible respondents.

3. The main criticism of the Internet for data collection concerns the unrepresentativeness of the current user base (at least in most European countries). This is exacerbated by the self completion nature of the medium. Clearly this is a major problem for surveys requiring nationally representative samples. However that is not the area in which Internet should be used for data collection. Instead it should be used for specific target groups which the sample is appropriate, e.g.

research on computer products
research on the Internet itself (e.g. Websites)
research in academic establishments
research in business environments
for early adopters of new technology
employee research
certain youth research.

Furthermore criticisms of low response rate because of its self completion nature should be critically reviewed. Response rates to all surveys are declining the UK. Many factors are cited including concerns about interacting with strangers. The impersonal nature of the Internet bypasses these. In addition, we believe that for certain targets (e.g. senior business people) response rates over the Internet may well be comparable if not better than face to face levels (and significantly cheaper).

The Internet and the New Marketing

Using electronic mail can fit very well with the New Marketing for certain products. Already virtually all companies with websites offer a facility to email comments and suggestions directly to them. If handled correctly with prompt efficient feedback to queries this can strengthen customer loyalty. Companies like Ford in the USA, receive 1,000's of emails every week with comments on their products. Indeed, many people who may never have bothered to write a formal letter of complaint (or praise) seem to be using email to communicate directly with companies.

In the same way, I predict that as Internet penetration grows, it will be increasingly used as a cheap, speedy and effective way of customer communication. Furthermore, it is likely that panels will be developed specifically amongst databases of product/service users who are regularly used for surveys. All of these will not only provide an efficient method of customer feedback but also act to strengthen the link between end user and original manufacturer/supplier that is so key to the New Marketing.

References:

Henley Centre (1996). Media Futures Report (to be published Nov. 1996).

MCDANIEL, S. W. & RAO, C.P. (1981). An investigation on respondent anonymity's effect on mailed questionnaire response rate and quality. Journal of the Market Research Society, 23, pp 150-160.

MEHTA, R & SIVADAS, E. (1995). Comparing response rates and response content in mail versus electronic mail surveys. Journal of the Market Research Society, 37, pp 429-439.

TSE, A. C. B. et al (1995). Comparing two methods of sending out questionnaires: Email versus mail. Journal of the Market Research Society, 37, pp 441-446.
                                                                                                                                                       

 

                                       

                                                                         

     © 2002 Virtual Surveys Limited Privacy Policy
Virtual House, 3 Seton Drive, Hampshire RG27 9QS United Kingdom
Email: