Top-10 most cited software engineering papers authored by Brazilian professors

Considering only papers indexed by CSIndexbr, published between 2013–2018, and using citation data from Crossref.

Crossref is an official DOI registration agency. Whenever a publisher asks Crossref for a DOI, it must provide several meta-data about the paper being registered, such as title, authors, date of publication, etc. Publishers can also provide the DOIs of each reference contained in the paper.

In this way, Crossref is creating a database of citations; given a DOI, this database stores other DOIs that cite it. More interestingly, this dataset is public (or at least part of it) and can be accessed using a REST API.

For more details about Crossref and open citations, see this interesting article, recently published at Nature News.

We used Crossref API to retrieve the number of citations of Software Engineering papers listed by CSIndexbr — a Brazilian Computer Science Index — ,which collects papers published by Brazilian professors in selected venues (journals and conferences), in the last five years (2013–2018).

Before presenting the results, we answer four important questions:

Why are you not using Google Scholar citations? Because they are "closed"; Google’s terms of service forbids data collection by bots or similar programs.

Why the number of Crossref and Google Scholar citations do not match? First, because Crossref only considers citations coming from papers with a DOI. The secod reason is because some publishers do not provide citations meta-data, as answered by them on Twitter:

As a result, Crossref citations are normally lower than the ones provided by Google Scholar. However, we also suppose there is a high correlation between both measures (although this needs to be verified).

Why are you collecting this data? Because we want to assess pros and cons of adding Crossref’s citations to CSIndexbr.

How to get the number of citations at Crossref (and therefore check the citations mentioned in this post)? You can use a url like this one (where the last part is a DOI): Then search for is-referenced-by field.

After this long introduction, here are the top-10 most cited software engineering papers authored by Brazilian professors (only SE papers published in 2013–2018, and indexed by CSIndexbr, ~300 papers):

  1. Mining questions about software energy consumption
    Gustavo Pinto (UFPA); Fernando Castor (UFPE); Yu David Liu 
    MSR 2014 
    Citations: 51
  2. Creating a shared understanding of testing culture on a social coding site 
    Raphael Pham; Leif Singer; Olga Liskin; Fernando Figueira Filho (UFRN); Kurt Schneider 
    ICSE 2013 
    Citations: 45
  3. Establishing Theoretical Minimal Sets of Mutants 
    Paul Ammann; Márcio Delamaro (ICMC/USP); Jeff Offutt 
    ICST 2014 
    Citations: 41
  4. Automated Behavioral Testing of Refactoring Engines 
    Gustavo Soares; Rohit Gheyi (UFCG); Tiago Massoni (UFCG)
    IEEE TSE 2013 
    Citations: 38
  5. On the reliability of mapping studies in software engineering
    Claes Wohlin; Per Runeson; Paulo Anselmo Silveira Neto; Emelie Engström; Ivan Machado (UFBA); Eduardo Almeida (UFBA)
    JSS 2013 
    Citations: 37
  6. An empirically based terminology and taxonomy for global software engineering 
    Darja Smite; Claes Wohlin; Zane Galvina; Rafael Prikladnicki (PUC-RS)
    EMSE 2014 
    Citations: 31
  7. Software engineering at the speed of light: how developers stay current using twitter 
    Leif Singer; Fernando Figueira Filho (UFRN); Margaret-Anne D. Storey 
    ICSE 2014 
    Citations: 30
  8. Extracting relative thresholds for source code metrics
    Paloma Oliveira; Marco Tulio Valente (UFMG); Fernando Lima 
    CSMR/WCRE 2014 
    Citations: 29
  9. Twenty-eight years of component-based software engineering 
    Tassio Vale; Ivica Crnkovic; Eduardo Almeida (UFBA); Paulo Anselmo Silveira Neto; Yguaratã Cavalcanti; Silvio Meira (UFPE)
    JSS 2016
    Citations: 24
  10. On strategies for testing software product lines: A systematic literature review 
    Ivan Machado (UFBA); John D. McGregor; Yguaratã Cavalcanti; Eduardo Almeida (UFBA) 
    IST 2014
    Citations: 24

Just a few comments about these papers:

  • 5 papers are from conferences; 5 papers are from journals.
  • The top-3 papers are from conferences.
  • 5 papers have non-Brazilians as first authors.
  • 8 papers have non-Brazilian as co-authors; 2 papers only have Brazilian authors.
  • 3 papers are from 2013; 6 papers from 2014; and 1 paper is from 2016.

The following chart shows the full distribution, for ~300 papers. 182 papers have at most 5 Crossref citations; other 62 papers have between 6 and 10 citations.