2021 Year in Review
We started the Journal of Systems Research (JSys), a new diamond open-access journal, in 2021. We outlined the reasons why we do so, along with its goals and aspirations here. In this post, we look back on one year of operating JSys. We first recap the need for JSys, then describe how JSys operates, and discuss its first year of operation.
We saw three main problems with the current publishing ecosystem.
First, it is expensive to publish research. Most conferences require at least one author to register (several hundreds of dollars) and to present in person (in non-COVID times). Travel, especially international travel, can be upwards of one thousand dollars. Though grants are available to partially cover students, this does not help researchers in countries like India, where both income and support for things like travel is an order of magnitude lower than in countries like the USA. Journals are not much better, requiring upwards of one thousand dollars for article processing charges. These expenses simply block out a large portion of the global research community from publishing research.
Second, many conferences do not have open access. Indeed, the entire business model of organizations like ACM and IEEE is dependent on generating revenue through paywalls. Open access is essential for rapid dissemination of results: communities such as Machine Learning have strongly embraced open access. Some funding sources also require any published research be open access, which is in opposition to the paywall business model.
Finally, our current publishing model creates artificial scarcity. By coupling the publication of papers with presentation slots at conferences, deserving papers get rejected simply because there are not enough presentation slots. This happens even when the conference is entirely online! Consider for example the ICML conference deciding arbitrarily that the acceptance rate was too high and decided to be reduced by 10%.
How does JSys solve these problems?
Publishing in JSys is completely free. It costs nothing to publish papers in JSys, or to read JSys papers. JSys is a diamond open-access venue. You might be surprised to hear this: how is this viable for the long term? The truth is that publishing papers costs very little: for example, the Journal of Machine Learning Research has been diamond open-access for 20 years!
The trade-off is that JSys is focussed entirely on publishing papers. JSys does not conduct conferences, sponsor students, or any other activities that might be good for the community. ACM and IEEE for instance undertake a wide variety of activities that are sponsored by the paywall revenue.
JSys decouples publication from presentation; authors of papers published at JSys are free to present their work at various conferences. As a result, JSys is able to accept every paper that meets our bar for publication, without concern for external metrics such as acceptance rates.
How JSys operates
The JSys journal is hosted by eScholarship, an open-access publishing platform from the University of California system. eScholarship takes care of hosting published papers, submitting them to indexing services, and obtaining DOIs for each published paper. This leaves the JSys team free to focus on reviewing papers.
The reviewing process takes place on OpenReview. Paper submissions are public, with decisions and reviews released publicly after the review process is over. Artifacts are also reviewed on OpenReview, but privately, since artifact evaluation may require providing ssh keys or other details.
The editorial team at JSys is comprised of the Editor-in-Chiefs who oversee the entire journal. The editorial board is divided up into areas. An area has one or more Area Chairs who are ultimately responsible for reviewing decisions in that area. The Area Chairs invite expert reviewers for their area to the editorial board.
There has been an excellent response from the community to JSys: we currently have eleven areas open, with 80 participating reviewers. We are thankful to all reviewers who participated in JSys in the middle of a pandemic.
Long-term viability of JSys. The recurring costs incurred by JSys are small, in the order of $30 per year to maintain the jsys.org website and obtain email accounts in the jsys.org domain. Our papers are hosted on eScholarship, and our reviews are hosted on OpenReview — both these platforms are free to use. Thus, JSys does not have any significant costs for running a diamond open-access journal, and will continue as long as it has support from the community for reviewing papers.
The review process
Each submission gets initial results in 1.5 months from the date of submission. The decision can be Accept with Shepherding, Revise, or Reject. Accepted papers have one month to produce the final camera-ready version. Papers with a Revise decision have three months to produce the revised version. Rejected papers cannot submit to JSys for one year.
JSys tries to have objective criteria for decisions: for example, see the criteria for Systemization of Knowledge papers. The aim is to avoid making decisions based on subjective criteria like whether the paper is interesting to the reviewers (who represent a small fraction of the community).
JSys has mandatory artifact evaluation for papers that introduce a new tool or a software solution. There is a separate review board headed by the Artifact Evaluations Chair that evaluates submitted artifacts and tries to determine if they can be used by a third party to reproduce the main results in the paper.
Results from 2021
JSys has quarterly deadlines: in February, May, August, and November. For the inaugural deadline, we decided to go with a March deadline. Thus, for 2021, the deadlines were March, May, August, and November.
We received four submissions for the first deadline, and 10 submissions overall.
Total number of submissions: 10
Under review: 2
Revise decision: 2
Accepted after revisions: 6
The most common decision is Revise. This indicates that for a majority of the papers, the concerns identified by the reviewers can be fixed within three months. Reviewing a revision is significantly easier than reviewing a new submission, and cuts down on the reviewing load for the community. Authors also experience significantly decreased randomness in reviews. The papers also increase significantly in quality by undergoing revision.
Among the six papers accepted for publication, four have been published at JSYs. Check out the papers here.
The papers have also been indexed by Google Scholar.
Each paper is accompanied by a foreword by the Area Chair, introducing the paper to the systems community. The reviewers are acknowledged. The paper links to the artifacts (compulsory if the paper is a Solution or Tool paper) and the reviews. The paper is distributed under the Creative Commons license, with the copyright retained by the authors.
The papers have received significant attention from the community, as reflected in their viewership stats. In just over three months, JSys has received over 1200 paper views and over 400 downloads. This compares favorably with many academic venues.
Observations from running JSys
Having operated JSys for one year, we wanted to reflect on the experience. At a high level, I think we proved that it is possible to run a diamond open-access journal independent of organizations like ACM or IEEE. Our recurring costs are minimal, and so it will be possible to keep running JSys for a long time.
The coordination costs of running something like JSys is non-trivial, even with the decentralized nature of JSys. At our current scale, two editors-in-chief are enough to run the whole process, but as we scale to 100 or 1000 annual submissions, this will start to break down. Thus, we need to put robust processes in place to handle the large scale of submissions.
Part of what makes coordination tricky is that we have many parts running concurrently for each submission: we have the main paper itself and its artifact (evaluated independently). If the paper is revised and resubmitted, we have to evaluate it alongside new submissions. We need more eyes on the whole process to make sure deadlines are met appropriately.
Things would be easier with software tailored for our use-case. We are currently using OpenReview, which allows for a lot of flexibility. The support from OpenReview has been excellent. We look forward to customizing OpenReview more to track revisions, couple papers and their artifacts, and more.
Overall, we are happy with the first year of JSys. We were able to publish four interesting papers at essentially zero cost, with the support of the systems community. We believe these papers will have significant impact in the years to come.
We look forward to another year of JSys! The submission portal for the Feb 2022 deadline is open: go ahead and submit your papers!