A new journal for systems research
TLDR: The review process and publication model for systems research is not optimal in a number of ways. We propose a new diamond-access journal, the Journal of Systems Research (jsys.org) to help improve the current state of affairs. A concrete proposal for the journal is available here.
The current state of affairs in systems research
Authors prepare and submit papers to conferences such as SOSP, OSDI, FAST, NSDI, ATC, SOCC, and EuroSys. These conferences have between 13% to 25% acceptance rate. We see a number of problems with the current state of affairs:
- The community reviews the same papers again and again. Papers rejected from one conference are re-submitted to another conference. At least three reviewers read the paper at each conference; often, the reviewers are not the same. There is no shared state between the conferences (although some conferences ask for changes since the last submission). Papers narrowly rejected at one conference need to be reviewed from scratch at another conference. This is sub-optimal, and wastes a lot of time for the community overall.
- Decisions are arbitrary. PCs are good at identifying the bottom 50% of papers. Different PCs identify a different set of papers to accept, as it mostly comes down to subjective judgements of value. The real problem is that the chances of acceptance do not monotonically increase with time as authors respond to reviewer comments in good faith; on resubmission to a different PC, the new reviewers may not agree with what the old reviewers asked for, and might ask for a new, and perhaps conflicting, set of changes.
- Lack of revisions leads to rejects for small issues. Current conferences operate under the model where accepted papers can be “shepherded”. While shepherds can ask for writing changes, they are not allowed to request significant extra experiments (to be precise, they can ask, but the authors can say no. The paper is still accepted). As a result, papers that are narrowly below the bar are rejected; if revisions were allowed, these papers would be accepted, saving the community significant time spent in re-reviewing the paper.
- Our current process is demoralizing to students. Good work may be rejected multiple times (even if each time a revision would have resulted in acceptance). Submitting even strong work becomes a gamble; it may or may not be accepted. It becomes impossible to say “I’ve worked on this for a couple of years; I’ve gotten feedback from senior folks who vouch for the work. I’m confident it will be accepted at SOSP/OSDI”. Seeing work bounce between venues due to the arbitrary nature of conference decisions is not a great motivator for science and research. Talented students may choose to leave.
- Our current publication process is extremely expensive. Publishing a paper in a systems conference requires traveling and attending the conference. This costs upwards of $1000 for each paper. This is prohibitively expensive for students and professors in countries like India, and the cost itself becomes the blocker for publication.
- Our research is not available freely for the public to read. We pay significant costs to publish a paper (often by using tax-payer funds given by organizations like NSF), yet these papers are not available for the public to read. Consider SOSP 2019: all papers need an ACM subscription to read.
Our proposal: the Journal of Systems Research
We propose a new diamond-access journal, the Journal of Systems Research (JSys). JSys is inspired by the Journal of Machine Learning Research, a diamond-access journal that has been in successful operation for 20 years. Diamond access indicates that the papers are free to read, submit, review, and publish. The full concrete proposal is available at jsysr.org.
The design of JSys borrows heavily from the databases, security, and crypto communities. JSys has several interesting features:
One-shot Revisions. Similar to the PVLDB model, authors get a single chance to revise their papers based on reviewer comments. Authors should submit the revised manuscript within three months of getting the “revise and resubmit” decision.
Quick Turn-Around. Initial decisions will be returned to authors in 1.5 months. Final decisions after revisions will be returned to authors in 5.5 months (1.5 initial review + 3 months for revision + 1 month for reviewing revised manuscript). The paper will be published one month after the final accept decision.
A light review load. A common problem many conferences face is being unable to find enough reviewers for all their submissions. JSys will return manuscripts to authors if the appropriate reviewers are already fully occupied; authors can choose to submit at the next quarterly deadline or submit to a different conference/journal. Thus, JSys will ensure high-quality of reviews by making sure reviewers are not overloaded.
No cap on number of accepted papers. JSys is not tied to a conference, and hence does not have to consider constraints such as number of presentation slots. JSys will accept every paper deemed by reviewers as contributing to the field.
No page limits. JSys recognizes that current conferences force authors into describing their systems and techniques in 12 pages, regardless of the complexity of these systems. While this provides focus and works well for some papers, it is detrimental to other papers which need more space for fully describing the system and experiments. JSys does not have a page limit, either for the submitted version, or the final camera-ready version.
Multiple paper types. Authors can submit one of five types of papers: regular, vision, systemization of knowledge, analysis, or experience reports. Reviewers will be provided guidance on how to evaluate each type of paper.
Papers will have names of reviewers and an optional review summary. Each paper will have a “Reviewed by” line containing the names of the JSys editorial board members who reviewed the paper. Prior to publication, the reviewers and authors work together to make the paper as strong as possible.
The full concrete proposal is available at jsysr.org. We invite feedback from the community regarding the proposal! You can either comment on the google doc or participate in the discussion at the #jsys channel at the CS Slack.
Why a new journal? Why not use ACM Transactions on Computing Systems?
JSys differs in several ways from ACM TOCS, ranging from the quick turn-around to accepting multiple paper types. However, the most importance difference is that JSys is diamond open-access. TOCS requires a ACM subscription to read — this goes against one of the core goals of JSys. While ACM is seeking to transition to gold open access (authors pay open-access fees) by 2027, we believe diamond open-access is the right goal, and that it can obtained today.
At a higher level, ACM’s model is to collect taxes from authors (open-access fees, registration fees, etc) and use it to perform various outreach and education activities (like the K-12 outreach program). Authors have no choice about this tax, or where the money is spent. While the outreach activities ACM engages in are important, we do not believe they should be funded via taxes on authors. We believe publishing should be free, and that outreach activities should be funded via separate sponsorship drives.
Why would anyone submit to JSys instead of to the existing conferences or journals?
We believe JSys offers compelling benefits that the current conferences/journals do not.
The first, and most important benefit, is that the one-shot revisions ensures papers fixable in three months are not rejected. This covers a large percentage of the papers narrowly rejected at systems conferences (missing experiments, details, related work, etc). Revisions at JSys will be reviewed by the same set of reviewers who reviewed the original submission. Think about the papers narrowly rejected at conferences: rather than roll the dice at systems conferences, submitting to JSys ensures acceptance using the revision mechanism. This contributes significantly to the mental health of students.
Second, JSys is diamond open-access. Publishing at JSys is free, and reading JSys papers is free. This has several benefits. It allows authors from low-income countries to freely publish systems research without worrying about registration and traveling costs. This drastically expands the pool of authors who can submit to JSys. Since the research is open-access, it can be freely shared, and more people will be able to read and build on the work.
Third, JSys supports a collaborative model of review, where the reviewers and the researchers work together to make the paper stronger. Reviewers will be able to ask questions to researchers throughout the review process, to clear up mis-understandings. The final manuscript will have a “Reviewed By” line, with the names of the reviewers. Thus, they have an incentive to make the paper strong.
Fourth, JSys will ensure high-quality reviews by doing admission-control of submissions. Reviewers will never have to review more than 12 manuscripts a year. This provides reviewers significantly more time to provide high-quality reviews. This is in stark contrast to the current model where reviewers are heavily overloaded with review requests, leading to rushed reviews.
Finally, JSys does not have page limits. For research that requires more than 12 pages to adequately describe the contributions, JSys will be an attractive option.
Why would the community adopt JSys? What happens to the existing conferences?
We believe JSys offers an opportunity to centralize reviewing and make it significantly more efficient than it is today. We believe reviewers are tired of endlessly reviewing the same papers over and over again; JSys will drastically reduce the total amount of hours spent reviewing across the whole community.
We believe current conferences and JSys are complementary to each other (similar to the PVLDB model). Research is submitted to JSys and eventually accepted. Papers accepted before a certain date can be presented at the next conference. We believe conferences should go back to their original purpose of networking among participants. Publication and presentation of research is thus decoupled; conferences may choose from among papers accepted to JSys to invite to present.
What is next for JSys?
JSys is an ambitious idea, and cannot happen without buy-in from the community. Please provide feedback at JSysr.org (and add your name to the Supporters list!). We are looking for institutional support to make JSys a reality: if you can help with this, please reach out to vijayc@utexas.edu or the #jsys channel at the CS Slack.
