What surveys can and can’t do

Surveys are just a tool. Like any tool they can be constructed well or badly, and used well or badly, for good or bad purposes. Surveys deal in abstractions of both attitudes and behavior. Any survey represents an implicit model of reality, framed at an abstract level in terms of relations among idea constructs. These constructs are operationalized as variables, each of which is in turn defined by one or more measurements. It’s important to remember that this is all made up by the researchers, for particular purposes. Attitudes are defined primarily by what is measured in their name. Behaviors may involve concrete actions, but the meanings assigned to those behaviors are also attributed by the researchers rather than being inherent in the actions themselves.

Even the best-constructed and best-administered survey can’t transcend the model on which it is based. There are well-established procedures for putting surveys together and for assessing the reliability and various kinds of validity of the constructs being measured. These are not always invoked, often because such tests might reveal serious weaknesses in the constructs and/or the model, but that doesn’t mean that they aren’t useful.

Surveys are frequently used to generate numbers that can then be manipulated in various statistical procedures. Statistical tests have only one real function; that is, to caution against inferential error by comparing detected relationships among data elements against the probability that the same results could have been obtained by chance alone. Again, the statistics are no more useful than the model underlying the study, and have no interpretation outside the frame of the model.

I’ve always held and taught that social data are in fact living things, since they express properties of living things, and therefore ought to be accorded the same degree of respect for their integrity that human subjects rules accord to all research participants. Among other things, this entails listening carefully for the authentic voice of the data. Given the enormous number of possible statistical procedures that might be invoked, it’s all to easy for the less-than-scrupulous researcher to in effect “waterboard” his or her data, forcing the data to tell a story of the researcher’s own choosing. This twisting of the meaning of data can be either conscious or unconscious; the latter is more dangerous.

My core point here is that a survey can’t be either discussed or evaluated outside of the context of the research within which it is embedded. The research context defines the model, which in turn defines the constructs, variables, and measures that make up the survey. Context also defines the population under study and the nature of the samples, if any, that are drawn from that population for practical and logistical reasons. The properties of the data themselves condition the kind of analytical procedures that can be applied and what to expect from them. The generalizability and utility of the findings, if any, beyond the specific context of the research project is never more than rhetorical — that is, the degree to which the researcher can make a convincing argument why they might apply in other similar settings across time or space.

Survey research, then, is always limited by its conceptual model. None of the phenomena in the model are “real” — even measures of the physical properties of things acquire meaning only by virtue of their inclusion in some mental model. Attitudes and behaviors exist only to the degree that they are defined and named through measurement. Models are never “true” as such, only more or less useful for particular purposes. Survey research that is grounded in these basic principles can be informative and helpful; survey research that forgets them is more likely to be misleading than useful.

Surveys aren’t just a technology; they are actually an entire epistemology.