Defining and measuring internet freedom

This article evaluates concept formation and measurement in Freedom on the Net 2017, the latest edition of a yearly report that measures the level of internet freedom in 65 countries. The report is compiled by the US-based NGO Freedom House, working with a team of contracted researchers around the world. Researchers answer a standardized template of questions and give numerical scores, which are then aggregated by Freedom House into a single number representing the level of internet freedom in each country. We conclude that Freedom House’s is the best comparative index of internet freedom available, but its approach raises concerns about concept clarity and internal validity.

Conceptualization

Before Freedom House can measure internet freedom, they must decide what it is. It may seem intuitive to write a prose definition, such as “the ability to impart and receive information through digital media without restriction.” But Freedom House’s methodology section does not attempt to explicitly define the concept. Instead, they state that the project methodology is “grounded in basic standards of free expression, derived in large measure from Article 19 of the Universal Declaration of Human Rights”, which reads

“Everyone has the right to freedom of opinion and expression; this right includes freedom to hold opinions without interference and to seek, receive, and impart information and ideas through any media regardless of frontiers.”

Freedom House then moves directly to operationalizing their definition it into 21 questions, such as “Does the government impose restrictions on ICT connectivity and access to particular social media and communication apps permanently or during specific events?,” to be answered by researchers in drafting of the report. The questions are divided into three categories: Obstacles to Access, Limits on Content, and Violations of User Rights.

To a certain extent, the last phase of the operationalization is outsourced to the individual researchers, who must decide what the (broadly-worded) questions means in their national context, measure reality against them, and represent their conclusion through numbers and text.

Because Freedom House eschews a formal definition, the set of questions posed to researchers is effectively both a definition and an operationalization of the concept of internet freedom. This approach is similar to a “family resemblance”-style definition: there are various configurations that qualify as free, partly free and not free, but no one condition is either necessary or sufficient. However, it is not clear whether this interpretation accurately represents Freedom House’s theoretical conceptualizing of internet freedom, or if it instead a sees internet freedom as a unitary concept which can be measured indirectly through various different questions. The latter would argue for Freedom House to publish measures of interitem association, which it does not do.

In addition to Article 19, two clarifying statements do aid the reader in guessing at Freedom House’s underlying conceptualization of the concept. First Freedom House maintains that it “does not maintain a culture-bound view of freedom,” which implies it believes freedom is a universal human experience rather than a construction of Western society. Second, the statement “The project particularly focuses on the transmission and exchange of news and other politically relevant communications, as well as the protection of users’ rights to privacy and freedom from both legal and extralegal repercussions arising from their online activities,” which implies that an inability to use the internet to order fried food would not harm a country’s freedom score as badly as the inability to use a social media platform to criticize a restaurant for a late delivery.

Measurement

Freedom House’s measure is a comparative, observational, weighted summation index compiled through an expert survey methodology. It includes 65 countries int, which were selected “to provide a representative sample with regards to geographical diversity and economic development, as well as varying levels of political and media freedom.”

Freedom House distinguishes its measure from those that only take law or government behavior into account, arguing instead that it is an empirical reflection of reality: “The index does not rate governments or government performance per se, but rather the real-world rights and freedoms enjoyed by individuals within each country.”

Measurement is a five-step process. The first step is the setting of the methodology by Freedom House, which happened once at the beginning of the project. The other four steps occur yearly:

  1. Freedom House selects about 70 researchers to complete the report for all countries to be studied, most of them in-country.
  2. Researchers individually track political developments in their assigned country and write their own reports, including scores, over a one-year period (for the 2017 report, this is June 2016 to May 2017).
  3. Regional meetings of researchers (often with Freedom House staff in attendance) give feedback on each other’s reports and revise scores.
  4. Freedom House fact checks and reviews the information, then compiles it into the final report, including scores and written country reports.

The 21 questions answered by researchers each have subquestions which serve as a guide to interpretation to specific national contexts. For example, the question “Does the government place restrictions on anonymous communication or require user registration?” is accompanied by four subquestions such as “Are website owners, bloggers, or users in general required to register with the government?” and “Are users able to post comments online or purchase mobile phones anonymously or does the government require that they use their real names or register with the government?”

Importantly, each question is assigned a score range, for example 0–3 or 0–6. Based on the question and subquestion text, researchers are asked to assign a score within that range for each question, with 0 being the most free and the maximum number being the most restrictive. The varying maximum numbers serve as a weighting function.

Evaluating the approach

We believe that Freedom Houses’s is the best-realized index of internet freedom available. The effort towards empiricism makes this more valuable than law-only approaches in which researchers review the statutes of different countries. It gives a sweeping comparative view that provides global context for observing individual countries. One could argue this is particularly powerful when applied to the internet, since it is inherently global and interconnected. From a practical perspective for advocacy (one of the stated goals of Freedom House) it is useful for pressuring countries that are lagging in freedom to raise their standards, because it allows simpler comparisons to more free countries.

However, the project is not without downsides. Compared to some other comparative indices that measure an abstract concept, such as the as the Polity IV dataset (measuring democracy), its coding system is less sophisticated and more open-ended. Though this is not necessarily bad, it does raise questions about whether Freedom House’s system could better operationalize the concept, since they make no attempt in their methodology to justify the relative simplicity of their measure. More specifically, we identify two potential problems: lack of a clear definition of internet freedom and internal validity.

One has to wonder whether the operationalization and methodology of measurement is harmed by the lack of a clear definition of internet freedom. Indeed, methodologists Johnson, Reynolds and Mycoff (in Political Science Research Methods, 8th ed. 2016) warn against not defining concepts before starting a research project, writing that it may “harm the quality of knowledge produced.”

We consider internal validity in this project to be the extent to which each country report and score accurately represents the state of internet freedom in each country. Reducing any social science concept to a single number for a whole country is inherently simplistic and unable to consider the subjective experience of each country with the care that other approaches might give. Particularly, it cannot numerically capture (though the county reports can shed some light on this) differences in internet freedom within a country. This would be especially problematic in a country with a stark divide in internet freedom across different demographic groups or regions. Even if the country is relatively homogeneous, the standardized design could prevent a researcher from fully taking into account an unusual condition that affected internet freedom in their country. It is also concerning that Freedom House does not attempt to justify the weights of its various questions.

Overall, the ambitious project makes a contribution to our understanding of digital rights, but cries out for a clearer definition of internet freedom, and may benefit from a more sophisticated measurement methodology.