Survey Research

em Metodologia da Pesquisa
 
Overview
Survey research is the method of gathering data from respondents thought to be representative of some population, using an instrument composed of closed structure or open-ended items (questions). This is perhaps the dominant form of data collection in the social sciences, providing for efficient collection of data over broad populations, amenable to administration in person, by telephone, and over the Internet. Some forms of survey research by telephone or the Internet may be completely automated. Critics of survey research methodology hold it to be a method which artificially forces respondents to formulate opinions, masking the complexity of conflicting views and unconscious biases within each respondent, and critics note that in many arenas (e.g., race relations) survey items poorly predict actual behavior. See also focus group research and also the separate sections on sampling and on standard measures and scales. Also, analysis of repeated surveys is discussed in the separate section on time series analysis.
Key Terms and Concepts
  • Survey Instrumentation. The survey "instrument" is the schedule of questions or response items to be posed to respondents.
  • Items are the individual survey questions or statements for which a response is solicited from the respondent.
  • Interviews and questionnaires. Ordinarily, "interviews" refers to the face-to-face administration of a survey instrument, whereas "questionnaires" refers to mail or other indirect methods of administration.
  • Response structure refers to the format of items, whcih may be open-ended or structured. Structured items may be multiple choice, multiple response, dichotomies, rank lists, or a variety of other formats.
  • Survey model. A social science model is can be depicted graphically as a set of variables (represented by circles) connected by a set of causal effects (represented by single-headed arrows) and correlations (represented by double-headed arrows). The survey should, at a minimum, include items pertaining to each circle and arrow in the model. Multiple items to measure each variable and relationship are preferred.
  • Survey order. Typically, after an introduction which discloses the sponsorship of a survey, a survey begins with non-threatening items which arouse interest. The first question should be clearly related to the announced purposes of the survey (not a background item, for instance). Some recommend the second question be open-ended, to allow the respondent to "get into" the subject. Non-threatening background information questions (ex., demographic information) should be posed early so these controls will be available if the respondent fatigues and does not answer all the later items. The survey then proceeds to attitude questions, often sequencing from general and less threatening items toward more specific and more sensitive items. Sensitive background items, particularly the income item, are usually put at the end. However, the more toward the end of the survey an item is, the lower its response rate is apt to be. For this reason, balancing the foregoing considerations, it is desirable to administer the survey with different forms with different orders so as to lessen the order/response bias.
  • Filter items are ones which allow the elimination of unqualified respondents during post-processing (ex., respondents who lack minimum information to respond intelligently).
  • Cross-check items are internal check items which test consistency with other responses. Asking age at one point and date of birth at another would be a pair of cross-checked items. An extension of this is the split-form interview, which might be administered to a both partners in a marriage with a view toward cross-checking for consistency.
  • Random probe items of an open-ended nature are randomly interspersed by item and by survey form such that any one respondent is only probed on a few items. Probe items are "Could you tell me a little more about what you mean?" or "Could you give me an example of what you mean?" Probe responses are used to better understand the meaning of frequency distributions of responses to the corresponding structured items.
  • Item bias. Bias may creep into survey items in a variety of ways.
    • Ambiguity. Questions should be specific, avoiding generalities. Ex.: "On a scale from 1 to 10, how popular is President Clinton at this time?" This example begs the question, "popular with whom?" Some respondents will answer the item in terms of popularity with themselves, others will answer in terms of popularity with their peers, and yet others will respond in terms of their recollection of past opinion polls. Items often need to be anchored in one or more of the familiar journalistic dimensions: who, what, when, where, why, and how? If it is possible for respondents to interpret questions in dissimilar terms, they will. "Have you participated in a political rally?" is also ambiguous, for instance, because participation is undefined and some will answer affirmatively due to passing within earshot of a political speaker, while others will answer negatively because, while they attended rallies, they were not the organizers. As a third example, "Do you favor government policy toward crime control?" leaves ambiguous the level of government (local, federal, state) as well as just what dimension of law enforcement is at issue. As a fourth example, when value preferences are requested by ranking "above average," "average," "below average," etc., ambiguity is introduced unless the question is answered, average in comparison with what? In general all pronouns, nouns, and noun-phrases should have a clear referent.
    • Mutual exclusivity. When multiple response items are allowed, bias is introduced if the items are not mutually exclusive yet only a single item may be selected.
    • Non-exhaustive response set. Bias is introduced when the response alternatives available to the respondent leave out valid choices they would otherwise make. The most common example is leaving out such responses as "neutral" or "don't know" when, in fact, respondents may well be neutral or may actually not know, rather than be hiding their "true" responses which the researcher is trying to force out by omitting these categories. Ambiguous terms, such as "parents," can also lead to non-exhaustive response sets, as by omitting the distinction between birth parents and adoptive parents. Likewise, "single" may omit the distinction between "never married," "widowed," and "divorced."
    • Rank lists. Ranking can be a challenging task. Many survey researchers recommend that respondents not be asked to rank more than four or five items. Beyond that, respondents may give arbitrary rankings just to get past the item.
    • Loaded terms . Ex. "Do you lean more toward the pro-life or toward the pro-abortion position on issue of termination of late-term pregnancies where the health of the mother is threatened?" This example is biased because one position is labeled with its most favorable label (pro-life, rather than anti-abortion), while the other position is labeled with its less favorable label (pro-abortion, rather than pro-choice). As another example, more people will "not allow" something than will "forbid" the same thing.
    • Leading questions. Ex., "Do you favor an increase in the federal minimum wage to $8.00?" This example is not as leading as an item of the type, "You favor X, don't you?" but it is still leading because it does not legitimize both affirmative and negative responses. A negative response may appear stingy or mean-spirited toward the poor, and this may bias the survey toward too many affirmative responses. A better method is to phrase such policy options in the form, "Some people favor X, while other people are opposed to X. What is your opinion?"
    • Recall items. People's ability to recall the past is limited. The more current and specific the question reference, the better. If recall is necessary, the time frame should be as recent as possible and not over six months unless the reference is to major events (ex., marriage, changing jobs, buying a car).
    • Unfamiliar terms and jargon. Ex., "Do you consider environmental regulation of wetlands to be an invasion of the sovereignty of the states?" Terms such as "sovereignty" are apt to not be well understood by typical survey populations. When a term not in popular usage must be used in an item, such as an item asking opinions about the Civil Rights Act of 1964, the interviewer must precede the item with a brief explanation. Wherever possible, familiar terms should be substituted for unfamiliar terms.
    • Requiring inaccessible information. An item may use familiar terms but require information most respondents would not know. For instance, a question about Stalin's treatment of the Cossacks might have been acceptable long ago, but today's population of respondents is apt to know little about this subject and perhaps not even recognize "Stalin." As a second example, the item "Is your family income above, about equal to, or below the official poverty rate for a family of four?" is flawed, among other reasons, because people are not apt to know what dollar amount is the referent today, making the item unacceptable for a determination of fact (although possibly acceptable for a determination of self-perception).
    • Multidimensionality. A form of ambiguity arises when items are multidimensional. Ex.: "On a scale of 1 to 10, please rank the performance of the president?" The respondent may be torn between multiple dimensions: personal vs. official performance, or domestic vs. foreign policy performance, for instance. Example two: "Do you believe President Clinton lied under oath in the Monica Lewinsky scandal, impairing his effectiveness as president?" This item calls for two opinions, not one: one on lying under oath and one on subsequent effectiveness. It should be broken into two separate items.
    • Compound items. Items with compound clauses may not be multidimensional but may involve undue complexity (see below). For instance, the item, "Do you have or have you every had a physical, mental, or other health condition which has lasted over six months and which has limited the kind of work you could do on your job?" is better broken into two items: "Do you have or have you every had a physical, mental, or other health condition which has lasted over six months?" and the follow-up item, "If you answered yes to the previous question, did this condition limit the kind of work you could do on your job?"
    • Complexity and memory overload. It is possible to overtax the respondent by requiring an excessive memory burden. The more complex the item, the easier it is to overload memory. The explanation should not demand the respondent learn complex new material. Simply using double negatives in an item may introduce undue complexity. If there are over five alternatives, a show card should be used to allow the respondent to view the alternatives, not simply hear them orally in an interview situation.
    • Poor grammatical format. Weak grammatical format can introduce bias. For instance, the item, "Would you say that you approve very strongly, strongly, ..." presents "dangling alternatives" which the respondent must memorize before even knowing what the question is. This format is frustrating and may bias responses toward the first-presented response, or toward negativity. Putting the subject first is the preferred order.
    • Hypothetical items. Hypothetical items (ex., "What would you do if ...") creates a difficult challenge for respondents. Seriously considering such items requires time for imagination and consideration. People tend to base responses to such items on their most-related actual experiences, and it may be better to ask about such experiences directly.
    • Inappropriate assumptions. Items should not contain false or arguable premises. The respondent should not be required to make a false or arguable assumption in order to respond to an item on its face. For instance, "How much improvement in your life would passage of the Equal Rights Amendment make? A great deal, some, very little, or none?" is an item which assumes that the effect of ERA passage could not be negative on a respondent, forcing respondents who believe the effect would be negative to either skip the item or give a response which does not represent their views.
    • Mismatched item and response set. The response categories should be appropriate to the dimension probed by the item. For instance, "How unusual do you think it is for a person to donate $100 or more to a presidential candidate? A great deal, some, very little, or none?" is a mismatch because "a great deal unusual," "some unusual," etc., are not grammatically acceptable responses and will confuse respondents.
  • Survey error, over and above item bias, survey error includes such factors as faults in sampling, coding, tabulating, data processing, interviewer bias, researcher bias, and data misinterpretation.
  • Pretesting is considered an all-but-essential step in survey research. No matter how experienced the survey researcher, pretests almost invariably bring to light item ambiguities and other sources of bias and error. In fact, Converse and Presser (1986: 65) argue cogently that a minimum of two pretests are necessary, with pretest sizes of 25 - 75 administered to respondents similar to those who will be in the final sample. The first pretest may have up to twice the number of items as the final, as one purpose is to identify weaker items and drop them from the survey. Items may also be dropped if the first pretest shows they exhibit too little variance to be modeled. The first pretest will also have many more probe items that the final, and respondents may even be told they are being given a pretest and their help solicited in refining the instrument. Other purposes of the first pretest are to see if respondent interest is aroused, if respondent attention can be maintained, if interviewers and respondents feel the survey has a natural flow, and if interviewers experience problems such as need to repeat questions, need to correct misinterpretations of questions, need to handle volunteered information, sections where the respondents wanted to say more, or questions which seemed sensitive. The first pretest results, hopefully, in a much-changed, finished instrument. The second pretest is used simply for polishing, trimming, rearranging, and other refinements, but not adding new dimensions or making major substantive changes in the survey. If such changes prove necessary, a third pretest will also be needed.
  • Split sample comparisons. After pretesting, if there is any remaining doubt about the reliability of one or more items, the researcher should consider split sample comparisons, where two versions of the same item appear on two different survey forms administered randomly. If the mean responses of the comparison subsamples differ significantly, the inference is made that the two versions represent two different items, not two versions of the same item.
 
Assumptions
  • Survey researchers often assume that attitudes lead to corresponding actions, but this is often not true, as discussed below.
 
 
Frequently Asked Questions
In some areas, such as voting, there appears to be a close correspondence between how respondents say they will vote and how they actually do vote subsequently. In other areas, such as race-related attitudes, the discrepancy between what people say and what they do appears to be substantial. Tartar (1969) found 65% of respondents changed their behavior away from surveyed attitudes when confronted with actual situations requiring behavioral response. An early survey by Ehrlich (1969: 34) concluded, "Studies on the relation of attitudes and behavior have almost consistently resulted in the conclusion that attitudes are a poor predictor of behavior." Ehrlich gave seven reasons for this gap:
      1. There may be no clear way for an attitude to become expressed in a behavior.
      2. Attitudes may reflect fantasy or sublimation more than realistic responses.
      3. Individuals may not be willing to disclose honest expression of certain opinions.
      4. Individuals may translate opinions into action in unpredictable ways, such as suppression and inaction, displacement into other arenas, and rationalization.
      5. Individuals may lack the informational resources to act.
      6. Individuals may lack the opportunity to act.
      7. Individuals may lack the sense of efficacy to act.
  • What are the pros and cons of using open-ended items?
Open-ended items are frequently used as complements rather than alternatives to structured items. As a follow-up to a structured item, an open-ended item can bring to light unanticipated interpretations and variations in the meaning of responses. For instance, in a ranking item about job satisfaction, a structured response may show "income" to be the most important satisfaction factor, but a follow-up open-ended item may show some respondents mean "high income," whereas other respondents mean "steady income." Open-ended items are not necessarily more accurate, however. An open-ended question may elicit one range of preferences, yet when other choices are presented in a structured question, respondents may well alter their open-ended preferences to re-rank in the light of previously-unconsidered alternatives in the structured item. Ideally, one would use both, with an open-ended probe question after every structured item. This, however, would be prohibitive in time and fatigue costs of the interview, which is why random probe questions are used, as discussed above.
  • Is it okay to force a choice, or is a 'no opinion' option necessary?
It is very tempting to leave out 'no opinion,' 'not sure,' and 'don't care' options in structured questions. Forcing a choice not only yields the direction of preference, but also requires the respondent to give a more considered response. However, forcing a choice does distort the responses of individuals who genuinely have no opinion, are genuinely unsure, or who genuinely do not care. When the researcher can assume everyone really has an opinion, then forced choice is justified. If not, the researcher should retain the no preference options, or should consider filter and intensity questions. Filter questions precede an item and are of the type, "Here is a statement about X. Not everyone has an opinion on this. Here is the statement: ... Do you have an opinion on that?" If the respondent answers 'Yes' to the filter question, then a forced-choice item is presented. Intensity items follow forced choice items and ask "How strongly do you feel about that -- extremely strongly, very strongly, ...?" A separate intensity item is needed because extremity on the forced choice item is not equivalent to intensity: one may hold an extreme position non-intensely and may intensely hold a middle position. In using forced choice rather than offering a 'no opinion' option, the researcher is assuming that those who would have given 'no opinion' responses will be low intensity on the intensity item, but not necessarily in the middle on the forced choice item itself..
  • How can I get a more valid response to highly sensitive questions?
This issue is treated in depth by Fox and Tracy (1986). The randomized response strategy takes subjects "off the spot" by mixing randomized responses with real ones. The basic randomized response method: Consider the question "Have you ever used an marijuana?" Respondents can be told to flip a coin or use some other randomizing device, then if heads appear (not seen by the interviewer) respondents are told to answer the question honestly but to answer "Yes" regardless if tails appear. If the result for a survey is 60% "Yes," we can reason that the true response rate is 20% (half of true "Yes" responses will also be part of the 50% expected tails-determined "Yes" responses and half not, so the true percentage is twice the excess of the observed response [60%] less the random response [50%]). Fox and Tracy extend this to polytomous and quantitative responses and discuss various refinements as well as alternative approaches which have proved problematic.
    •  
  • How do I handle the fact that not all respondents answer all items in my questionnaire?
Andrew Gelman, Gary King, and Chuanhai Liu (1998) present a method of analyzing a series of independent cross-sectional surveys in which some questions are not answered in some surveys and some respondents do not answer some of the questions posed. The method is also applicable to a single survey in which different questions are asked, or different sampling methods used, in different strata or clusters. The method involves multiply-imputing the missing items and questions by adding to existing methods of imputation designed for single surveys a hierarchical regression model that allows covariates at the individual and survey levels. Information from survey weights is exploited by including in the analysis the variables on which the weights were based, and then reweighting individual responses (observed and imputed) to estimate population quantities. They also develop diagnostics for checking the fit of the imputation model based on comparing imputed to non-imputed data. They illustrate with the example that motivated this project --- a study of pre-election public opinion polls, in which not all the questions of interest are asked in all the surveys, so that it is infeasible to impute each survey separately. See Gelman, King, and Liu (1998).
  • How do I handle the need to translate items across languages, yet have the same meaning?
This is the focus of Behling and Law (2000). Simple direct translation of validated U. S. scale items into another language often will not create a valid scale in the other language. One must consider three dimensions of the problem: semantic equivalence, conceptual equivalence, and normative equivalence of items. An item may be acceptable in one dimension but not in another. Semantic equivalence is sought through the translation/backtranslation method, having independent translators translate from one language to another, and then back again, to see if the original and re-translated item remain the same. Conceptual equivalence may be sought by such methods as factor analysis, demonstrating that items in one language version have the same factor structure (load similarly on underlying factors) as in another language version. Behling and Law recommend caution in this and other statistical tests, however, urging that theory be used to validate each language version independently. Normative equivalence requires such strategies as developing close relations with respondents, using trusted agents for interviewing, providing assurance of anonymity or confidentiality, and pilot testing extensively. Behling and Law give numerous additiona practical suggestions and insights into the translation problem and this text is strongly recommended for those who will face translation issues.
.
Bibliography
Arksey, Hilary and Peter T. Knight (1999). Interviewing for social scientists: An introductory resource with examples. Thousand Oaks, CA: Sage Publications. Covers design, triangulation, human subjects protection, report writing, and other basic topics.
  • Behling, Orlando and Kenneth S. Law (2000). Translating questionnaires and other research instruments: Problems and solutions. Thousand Oaks, CA: Sage Publications. Series: Quantitative Applications in the Social Sciences No. 133.
  • Bourque, Linda B. and Virginia A. Clark (1992). Processing data: The survey example. Thousand Oaks, CA: Sage Publications. Quantitative Applications in the Social Sciences Series No. 85. A non-technical work which covers designing forms, pretesting, training data collectors, data entry, and data preparation, including handling nonresponse, missing data, and outliers.
  • Bourque, Linda B. and Eve P. Fielder (2002a). How to conduct self-administered and mail surveys, Vol. 3. Thousand Oaks, CA: Sage Publications. Covers pretesting, pilot testing, cover letters, tracking non-respondents, and other mechanics.
  • Bourque, Linda B. and Eve P. Fielder (2002b). How to conduct telephone surveys, Vol. 4. Thousand Oaks, CA: Sage Publications. Covers types of phone surveys, instrument design, and other mechanics of telephone surveys.
  • Converse, Jean M. and Stanley Presser (1986). Survey questions: Handcrafting the standardized questionnaire. Thousand Oaks, CA: Sage Publications. Quantitative Applications in the Social Sciences Series No. 63.
  • Dillman, Don A. (1999). Mail and Internet surveys: The tailored method, Second edition. NY: John Wiley and Sons. Probably the most-used graduate guide to survey research, covers use of survey waves in analysis of nonresponse.
  • Ehrlich, Howard (1969). Attitudes, behavior, and the intervening variables. American Sociologist 4(1): 29-34. Cited in relation to the gap between attitudes and behavior.
  • Fink, Arlene (2002a). The Survey Handbook, Vol. 1. Thousand Oaks, CA: Sage Publications. Covers qualitative surveys and survey ethics.
  • Fink, Arlene (2002b). How to Ask Survey Questions. Thousand Oaks, CA: Sage Publications. Covers focus group questions, factorial questions, and conjoint analysis questions. Discusses Internet surveys.
  • Fink, Arlene (2002c). How to design survey studies, Vol. 6. Thousand Oaks, CA: Sage Publications. Covers validity issues in survey research as well as cross-sectional, cohort, and case-control common designs.
  • Fink, Arlene (2002d). How to report on surveys, Vol. 10. Thousand Oaks, CA: Sage Publications. Covers survey-based report-writing, including datagraphics, tabulation, and psychometric aspects.
  • Fowler, Floyd J., Jr. (2001). Survey research methods, Third edition. Thousand Oaks, CA: Sage Publications. Best-selling introductory text.
  • Fox, James Alan and Paul E. Tracy (1986). Randomized response: A method for sensitive surveys. Thousand Oaks, CA: Sage Publications. Quantitative Applications in the Social Sciences Series No. 58. Covers randomized response methods to encourage response to sensitive items.
  • Gelman, Andrew, Gary King, and Chuanhai Liu (1998). Not asked and not answered: Multiple imputation for multiple surveys. Journal of the American Statistical Association, March, 1998. You can retrieve the paper from File Transfer Protocol (FTP) to ftp wizard.ucr.edu, then login as user: anonymous, password: , and cd to pub/polmeth/working_papers97, and retrieve gelma97c.zip.
·  LaPiere, Richard T. (1934). Attitudes vs. actions. Social Forces, Vol. 13 (December): 230-37. This is the classic article challenging survey research and empirically showing its limitations.
·  Oishi, Sabine Mertens (2002). How to conduct in-person interviews for surveys, Vol. 5. Thousand Oaks, CA: Sage Publications. Covers interview scripting, transition statements, interviewer job descriptions, interviewer training, handling of response errors, data cleaning.
·  Peterson, Robert A. (2000). Constructing effective questionnaires. Thousand Oaks, CA: Sage Publications.
·  Rea, Louis A., Richard A. Parker (Contributor), and Alan Shrader, ed. (1997). . Designing and conducting survey research: A comprehensive guide (Jossey-Bass Public Administration Series). San Francisco: Jessey-Bass.
·  Salant, Priscilla, with Don A Dillman (1994). How to conduct your own survey. NY: John Wiley and Sons.
·  Tartar, Donald (1969). Toward prediction of attitude-action discrepancy. Social Forces 47 (4): 398-404. Cited in relation to the gap between attitudes and behavior.
·  Tourangeau, Roger, Lance J. Rips, and Kenneth A. Rasinski (2000). The psychology of survey response.. NY: Cambridge University Press.
·  U. S. General Accounting Office (1991). Using structured interviewing techniques. PEMD-10.1.5, 7/91. Available online in .pdf format.
·  Winstein, Raymond M. (1982). The mental hospital from the patient's point of view. In Walter R. Gove, ed., Deviance and mental illness, Thousand Oaks, CA: Sage Publications. This famous article shows the pitfalls of participant observation research (ex., Goffman's and Rosenhan's classic studies) and demonstrates how systematic survey research captures the true experience of mental hospitilization. Nesbary, Dale (1999). Survey research and the World Wide Web. Boston: Allyn and Bacon.
 
 
Parte superior do formulário
Parte inferior do formulário