What happens when a survey estimate doesn’t match a known benchmark
Survey researchers sometimes measure the quality and validity of survey data by comparing it to known benchmarks. But from time to time, researchers face challenges when data collected through surveys doesn’t match sources that are widely acknowledged as accurate.
For example, a survey may overrepresent men or highly educated people when compared with census data. And it is well documented that certain behaviors, such as rates of church attendance, can be overreported in surveys in comparison with rates reported in time-use diaries. In these situations, researchers often try to understand why the estimate varies from the benchmark beyond the margin of error. More specifically, they might try to answer two questions:
1. Did the survey suffer from measurement errors resulting, for example, from the composition of the sample or question wording?
2. Did the benchmark and the survey question measure two different things?
Pew Research Center recently faced this challenge when evaluating data from a survey in Western Europe that asked (among other things) about church taxes or fees collected by some European governments to fund officially recognized religious groups. Across several countries included in the study, we encountered some variation between the share of adults who say they pay a church tax and what is reported by governments and religious authorities.
In two countries, Germany and Austria, the magnitude of variation was especially large: We found that significantly more people say they pay the church tax than is officially reported by the German government and the Austrian Roman Catholic Church. This prompted us to explore both questions outlined above: Were there biases that inflated the number of church tax payers in our results? Or did our survey question measure something different from the benchmark?
After ruling out several common sources of measurement bias in surveys, we felt that our question, which on the surface asks about a behavior, is better interpreted as an attitudinal measure. It appears to tap into whether Europeans see themselves as broadly eligible or obligated to pay church taxes, rather than the more narrow question of whether they personally pay church taxes at the current time. Looked at another way, saying they do not pay church taxes is tantamount to saying they have deregistered from their church or religious group. In this post we detail the steps we took to assess the church tax question and how to best interpret the findings.
Background
As part of a larger survey on the role of religion in public life in Western Europe, the Center asked respondents in eight European countries, “Do you currently pay the church tax, or not?” This question seems straightforward and, based on our preparatory research, relevant in these countries. In six of the countries polled, church taxes or fees are mandatory for registered members of officially recognized religious groups whose income is above certain thresholds. People can opt out only by officially deregistering from their church or religious institution.
In analyzing the survey data, we compared the share of people who say they pay the tax with official statistics maintained by governments or religious authorities. Gaps between our survey findings and official data varied in size, but all were outside the margin of error: In Denmark, for example, 80% of adults told survey interviewers that they pay the church tax, while Danish government data show that 76% of adults pay it. (The margin of error for Denmark is plus or minus 3 percentage points.) In Finland, 71% of respondents say they pay the tax, compared with 63% of adults in official statistics. And in Sweden, the difference went in the opposite direction, with 68% of survey respondents saying they pay a church tax, while official sources indicate that 80% of Swedish adults pay it.
Several factors could result in such discrepancies — in some countries respondents may try to meet social and legal expectations when answering questions about civic behaviors like paying taxes (what survey researchers refer to as “social desirability bias”), or respondents may be unable to accurately recall a specific behavior or action. In addition, people participating in a survey can be systematically different from people who refuse to participate.
In Germany and Austria, however, we encountered even larger gaps. While 71% of survey respondents in Germany say they pay church taxes, official data suggests that only about 27% of German adults do so. Similarly, in Austria, 76% of survey respondents say they pay church taxes, while only about half of Austrian adults actually pay the tax, according to the Roman Catholic Church in Austria. (Austria has small Protestant and Old Catholic populations who also pay church taxes, but data for these groups is unavailable.)
Did the survey suffer from sampling or other errors?
The gaps in Austria and Germany prompted us to repeat the steps we typically take to evaluate the validity of survey data. These are the steps we followed:
Evaluating sample composition: To be nationally representative, surveys must closely match the demographic makeup of the country. In other words, the regional distribution, age distribution, education levels and gender balance in the survey must accurately reflect the national parameters usually available through census data.
Survey researchers typically correct for under- or overrepresentation of particular groups through weighting. Underrepresented groups are weighted up, while overrepresented groups are weighted down to match the census or other parameters.
Keeping in mind that a respondent’s likelihood of paying the tax is correlated with demographic factors such as age and place of residence, we once again evaluated the demographics of the samples in Germany and Austria and found that the survey did not over- or underrepresent key demographic groups and that the data was weighted correctly.
Comparing the data with other surveys: Behavioral or attitudinal data from other surveys can help detect potential biases beyond standard demographic variables. In this case, we were unable to find a previous survey that had asked Germans or Austrians whether they pay church taxes. But past surveys have asked about religious behavior in both countries. If our survey drastically overrepresented religious people, we might explain the discrepancies between our survey results and church tax benchmarks in Germany and Austria as a result of “overly religious” samples.
We looked at previous studies for comparably worded questions. For example, the highly respected European Social Survey (ESS) has asked a question about the frequency of church attendance that is very similar to a question included in our survey of Western Europe. If our survey inadvertently overrepresented highly religious people, the bias presumably would affect not only the percentage of our respondents who say they pay the church tax, but also the percentage who say they attend religious services. The 2016 ESS found slightly lower shares of people who attend religious services monthly than the 2017 Pew Research Center survey. But the differences in the estimates between the two surveys (3 to 4 percentage points) were not large enough to suggest the kind of systematic bias that would inflate the share of church tax payers by 30 to 40 percentage points.
Assessing question wording: The conceptual framing and wording and translation of survey questions can also produce error.
The church tax question was vetted by academic experts and tested prior to field work, and a post-hoc review confirmed that the wording was appropriate. Our translation, meanwhile, went through several rounds of review. After fieldwork, we confirmed again that the German translation used in both countries was true to the original question wording:
Do you currently pay the church tax, or not?
German (Austria and Germany): Zahlen Sie derzeit Kirchensteuer, oder nicht?
If many people refuse to answer a survey question, it may be a sign that the wording is confusing. However, no more than 4% of respondents in any country declined to answer the question.
However, appropriate wording and accurate translations are not sufficient to rule out other challenges associated with question performance. We know from previous research, for instance, that recall can be a source of error in survey questions that ask about past behavior. Recall issues can be compounded by contextual factors. For instance, in some European countries, taxes are automatically withheld on behalf of employees. Unless workers pay close attention, they may not know whether they are paying specific fees, such as the church tax.
Respondents may have understood the question as asking about their obligation or eligibility to pay church taxes. That is, some people who are registered as members of a church or other official religious group may have answered “yes” because they think of themselves as eligible to pay the church tax, and they may have said “no” only if they have deregistered from a religious group or were never registered at all. In many European countries, people are officially registered with a church in infancy, at the time of baptism, and stop being registered only if they take formal steps to deregister.
European experts consulted by Pew Research Center said that in Germany and Austria, it is widely believed that all registered Catholics and Protestants must pay church taxes. But, in reality, only those with income high enough to trigger a general income tax bill are required to contribute money to their churches. Retirees, nonworking spouses, students and others may think of themselves as church tax payers because they used to pay the tax or because someone else in their household pays it.
Indeed, there are many Germans and Austrians who may think of themselves as participants in the church tax system, but who do not actually pay any taxes at all. According to Germany’s 2011 census, there were 43 million German adults registered as Christians — 65% of the overall adult population. But only 30.8 million (or 72% of Christians) were registered as income tax payers, and, after deductions and exemptions, only 23.5 million Christians (roughly 55% of Christians and just 35% of all adults) actually paid any income taxes and were therefore even eligible to pay church taxes in Germany. In Austria, official data are available only for Catholics, the country’s largest religious group. Out of an estimated 4 million officially registered Austrian Catholic adults, just 3.4 million pay the church tax.
Evaluating correlations with attitudinal measures: At this stage in our assessment of the church tax question we judged that there was sufficient reason to doubt its efficacy as a behavioral measure. But did the question nonetheless measure something meaningful?
When we broadened our examination of the survey data, we found that the attitudes of people who say they pay the church tax differ in a logical, consistent way from the opinions offered by those who say they’ve opted out of paying. For instance, church tax payers are much more likely to answer in the affirmative to separate questions about whether churches and other religious institutions help the poor and needy, strengthen morality and bring people together.
Conclusion
Not all survey questions perform as desired, and some can be interpreted in unintended ways. In this case, discrepancies between external benchmarks and our survey findings about the share of people who pay church taxes led to a thorough reexamination of our samples and question design. We could detect no obvious bias in our samples. We did discover contextual factors, however, that suggest the question may have been interpreted as asking about one’s obligation or eligibility to pay the church tax, as opposed to recalling the conscious act of payment.
When we took the added step of exploring correlations between paying church taxes and attitudes toward religious institutions, we discovered that our intended behavioral measure aligned well with several key attitudinal measures about the social and moral impact of religious institutions. Given this, we felt comfortable interpreting our question less as a record of tax compliance and more as a willingness to support churches and other institutions with public tax revenue.
Neha Sahgal is associate director of religion research at Pew Research Center.
