Working Group on Peer Review of Full Papers at CHI: Call for Volunteers

Review Working Group Co-Chairs: Regan Mandryk and Sven Mayer

TL;DR: The CHI Steering Committee is putting together a working group on peer review of full papers. There are three ways to contribute (member of the working group, expert panelist, community stakeholder) described below, with expected time commitment and responsibilities. Volunteer by July 15, 2024 at https://forms.gle/mhfCRGVWaCrZ5SZD7.

The CHI Steering Committee is forming a working group to reimagine peer review at CHI. According to our policies (https://chi.acm.org/policies-processes/working-groups-policy/), working groups are short-term committees expected to deliver on a specific mandate within a particular remit. This working group will focus on articulating challenges with the peer review process of full papers for the CHI conference and articulate proposals for changes to address these challenges. Members of the working group will engage with the community through interviews, focus groups, and surveys, and will analyze existing data held by the CHI Steering Committee. As part of our remit, this working group will establish several expert panels that will provide input from different methodological perspectives reflected in research published at CHI and groups of community stakeholders that bring specific perspectives.

Mandate: To engage with the broad CHI community on challenges and solutions for our current process of peer review of full papers at CHI. The working group is tasked with considering the following:

• Articulate what is a ‘CHI paper’.
• Formulate a policy and process for desk rejections and assisted desk rejections at CHI.
• Provide a recommendation for whether CHI should implement the CRediT system, and if so, provide a process for inclusion into the review pipeline.
• Draft policies for CHI on open review, anonymous submissions, revise and resubmit at CHI, minimum qualifications for reviewers/ACs/SCs, submission limits, paper lengths, reviewer incentives/disincentives, the use of artificial intelligence in review, inter-conference review sharing, and citational justice after discussions with the community.
• Create author and reviewer resources focused on the logistical aspects of publication at CHI, including: accessibility, templates, and the proceedings pipeline.
• Create reviewer resources for CHI on reviewing interdisciplinary work, and best practices for review (may be subdomain specific and ever changing, e.g., including positionality statements, asking for quantitative or intercoder reliability for qualitative work).
• Manage the transition of CHI to ACM Open.
• Consider the current subcommittee structure at CHI and propose changes as necessary, including the keywords used in PCS.
• Consider other issues of peer review as raised by the larger community in discussions.

Remit: The working group will be concerned with full paper publications, as reviewed for and published by the CHI conference. Although there will be findings that may apply to other tracks (e.g., extended abstracts) or venues of publication (e.g., other SIGCHI conferences), and the working group will share knowledge with various stakeholders, the final report will be specific to full papers at CHI.

Timeline: The working group will conduct its activities over one year and deliver on a final report by August 2025.

Call for Volunteers

Members of the Working Group

Description: The working group (WG) will be responsible for leading the community discussions around peer review at CHI, will analyze data related to full papers at CHI, will generate proposals for changes, and will gather feedback on these proposals. The WG will aim to reflect the diversity of the CHI community, in terms of geography, epistemology, identity, career stage, and career type.

Expertise: We expect members of the working group to have significant expertise in terms of peer review of full papers at the CHI conference, either as paper chair, technical program chair, or paper subcommittee chair and to have published full papers at multiple CHI conferences.

Expectations of Contribution:

• Analyze data from PCS and surveys gathered by CHI Steering Committee
• Gather knowledge from other research communities in peer review practices and challenges
• Design and deploy surveys to larger CHI Community
• Process survey data and distill findings
• Conduct interviews and focus groups with expert panels
• Lead community sessions with various stakeholder groups
• Propose new plans and articulate them in blog posts

Time commitment: Twice monthly meetings for one year with additional workload when focus groups and interviews are being conducted and interpreted

Members of the Expert Panels

Description: We will have multiple expert panels that reflect the methodological diversity of research approaches in the CHI conference. The members of the expert panels will provide specific knowledge related to gold standards of evaluation for a method, problems of evaluating research contributions using this method, and potential solutions for addressing problems within peer review. Across the expert panels, we aim to reflect the diversity of the CHI community, in terms of geography, epistemology, domain, methodological approach, identity, career stage, and career type.

Expertise: We expect that members of the expert panels will have experience reviewing for and publishing full papers at CHI. Further, we expect that expert panelists will have served as an Associate Chair for one of the CHI subcommittees.

Expectations of Contribution:

• Participate in interviews and focus groups with WG
• Edit survey questions and provide input on interpretation of findings
• Provide feedback on proposed changes to peer review at CHI
• Recruit input from community members on proposed changes

Time commitment: 6–8 meetings over a 12-month time frame

In addition, each expert panel will have a chair, who will be the lead point of contact for that group.

Community Stakeholders

Description: There are various community stakeholders who may wish to represent their perspectives to the WG and expert panels in a deeper way than participating in community sessions or providing survey data. These stakeholders could be related to career type, stage, identity, geography, or other perspectives that we haven’t yet considered.

Expertise: We expect that community stakeholders are part of the SIGCHI community, and have published and reviewed full papers at CHI.

Expectations of Contribution:

• Participate in interviews and focus groups with WG
• Provide feedback on proposed changes

Time commitment: 2–3 meetings over a 12-month time frame

To Volunteer

Please fill out the volunteer form at https://forms.gle/mhfCRGVWaCrZ5SZD7 by July 15, 2024. We will get back to all volunteers by August 15, 2024. Work begins in September, 2024.