CHI ’23: Overview of the Post-conference Survey
The CHI post-conference survey ran for two weeks after the conference. We closed it on May 14. Overall, 832 persons took part to the survey, accounting for approximately 17% of the total number of CHI attendees (4,722 in total). This article gives a concise overview of the survey findings. Interpretations are based on a thematic analysis conducted to analyze the responses received for the open-ended questions.
About attendees’ experiences
The following chart depicts the distribution of respondents based on their attendance at the conference. The green bar represents respondents who attended the conference in person, while the brown bar represents those who attended virtually. Among the respondents, approximately 10% attended the conference virtually. This percentage is slightly lower than the actual percentage of virtual attendance, which was approximately 17% for this year’s conference.
The distribution of responses to the question “How valuable was your CHI experience?” is presented below, according to a breakdown by attendance type: on-site attendance on the left and virtual attendance on the right. The chart shows that on-site attendance was highly valued, primarily due to the opportunities for networking and the quality of scientific content presented at the conference. In contrast, feedback from virtual attendees was less positive. Analysis of respondents’ comments revealed two main reasons for their dissatisfaction: 1) remote presenters did not receive sufficient visibility during the on-site conference, and 2) virtual attendees expressed a sense of isolation as they were unable to engage and interact with other conference participants. These factors contributed to the less positive evaluation of the CHI experience among virtual attendees, as compared to those who attended in person.
The survey included a series of questions where respondents individually rated various aspects of the conference, including sessions and activities (such as papers, demos, posters, panels, SIGs, student competitions, workshops, and keynotes) as well as conference events (such as receptions, exhibits, job fair, and town hall).
Overall, attendees provided mostly positive feedback about the sessions and activities, with all venues receiving equal appreciation from on-site attendees. However, comments from some on-site attendees highlighted the need for improvements in the organization of the poster venue. Specifically, respondents mentioned that the spatial layout of the hall can be confusing, and the program schedule too packed, leaving insufficient time for interaction with poster presenters. This issue was further reflected in a subsequent question, where 32% of respondents stated that they were unable to see or interact with some posters or demos they had intended to.
In contrast, feedback from virtual attendees was consistently negative across all venues, although some comments mentioned that certain SIGs managed the hybrid experience well. Concerns were also expressed regarding the Job Fair event, as it lacked significant engagement. Additionally, comments revealed that some attendees expected the town hall sessions to allocate more time for discussing important topics, such as the environmental costs associated with the conference location.
Respondents had the opportunity to evaluate how well different tools and aspects of the conference met their needs. This included the conference website, the Progressive Web App (PWA) used to navigate the program, as well as the different types of content offered by the conference, such as streamed content, recorded content, and on-site presentations. The distribution of their responses is presented below.
Overall, people highly appreciated the on-site sessions. However, ratings for streamed and recorded content were lower, which is likely due to the limited availability of such content. Regarding the PWA, feedback was mixed, with some attendees expressing their satisfaction with its features, while others expressed certain concerns. Usability issues that were brought up by respondents have been compiled and will be taken into account by SIGCHI to the best of their abilities.
To accommodate the hybrid format, presenters were requested to create pre-recorded videos, which has become a standard practice. For papers, case studies, and alt.chi presentations, the duration of these videos was set to 7-to-10 minutes. For posters (including LBW, student competitions, and Doctoral Consortium) and demos (Interactivity), the duration was set to 3 minutes. In the survey, we asked respondents to provide their perspective on whether these durations were suitable, considering both the attendee and presenter viewpoints.
In the charts below, the large grey areas in the bars indicate that a significant proportion of attendees did not watch these videos. This observation is particularly pronounced for poster presentations. This is not very surprising given that a majority of survey respondents attended the conference on-site. However, many comments expressed satisfaction towards these videos, emphasizing their role in facilitating asynchronous consumption of content. It is worth noting that there were some comments that highlighted the amount of work involved in producing these videos for presenters.
Regarding the duration itself, which was the primary focus of these questions, a majority of respondents assessed the current durations as suitable (indicated by the blue bar in the charts). Very few respondents expressed a preference for shorter videos (represented by the light purple bar). We can conclude that the current video durations should not be reduced.
About the evolution of the selection process
The survey included a series of questions about the selection process at CHI. In particular, we were interested in the comparison between the previous system based on rebuttals and the current system based on revise and resubmit (R&R).
The first question aimed to determine the percentage of respondents who had previously submitted a research paper and their experience with each system. Notably, over 40% of the respondents had submitted papers using both systems, enabling them to provide insightful comparisons as users of both approaches.
Have you already submitted a research paper to CHI?
---------------------------------------------------
| 43.06% | Yes - in both the previous system and the current system |
| 20.68% | Yes - in the current system (R&R) only |
| 10.73% | Yes - in the previous system (rebuttal) only |
| 26.44% | No |The charts below provide a comparison between the two systems, based on different criteria. Green bars are when rebuttals are considered better, brown bars when R&R is considered better, and grey is “no difference”.
On the first line, we consider 1) the quality of papers, 2) the quality of the dialogue between authors and reviewers and 3) the quality of reviews. Overall, the quality of the review process is perceived as better with R&R than with rebuttals.
On the second line, we can observe the drawbacks associated with R&R. Specifically, the charts illustrate how respondents assessed the workload for authors, reviewers, and Program Committee (PC) members. The large green bars indicate that, in comparison to the rebuttal system, R&R increases the workload for all participants involved in the process.
The survey concluded with a final question that aimed to assess respondents’ overall preference between the two systems. The results indicate an inclination towards the revise and resubmit (R&R) system:
Overall, what system do you prefer?
-----------------------------------
| 40.72% | R&R |
| 19.94% | Rebuttals |
| 39.34% | No preference |About accessibility
Among the respondents, approximately 2.79% requested an accessibility-related feature at the conference. The survey also explicitly inquired if attendees had encountered negative experiences in terms of accessibility. The chart below gives an overview of the responses, revealing that some respondents did report negative experiences.
Upon analyzing the comments provided in response to this question, the negative feedback is primarily attributed to the challenges faced by virtual attendees in interacting with the conference. Regarding experience on site, a few comments mentioned how overstimulating it can be for some individuals.
The survey included a couple of questions regarding positive experiences and suggestions for improving accessibility. Respondents provided valuable insights and mentioned several positive experiences that contributed to their overall satisfaction. Specifically, they highlighted the availability of sign language, captions for content, the presence of a quiet (desensitization) room in the conference center, and support for families (childcare).
Furthermore, respondents also provided suggestions for enhancing accessibility. In terms of virtual attendance, they want more support to interact with attendees and presenters on site and remote. As for the on-site experience, attendees expressed the desire for a wider range of food options with clear indications regarding their composition. Additionally, they would like more seating areas where they can rest and relax.
About transportation
The series of questions related to transportation revealed that approximately half of the attendees arrived by train (note though that a large proportion of respondents come from Europe). During the conference, participants either traveled on foot or took the train. This was greatly facilitated by the provision of complimentary passes for public transportation in Hamburg, which were included with the conference registration. The respondents expressed a high level of satisfaction regarding this initiative, with half of them stating that it had influenced their travel plans in some way.
How did you travel to Hamburg? (select all that apply)
------------------------------------------------------
| 58.08% | Plane |
| 53.48% | Train |
| 5.43% | Car |
| 4.04% | Other |How did you travel around Hamburg while attending the conference?
-----------------------------------------------------------------
| 82.61% | On foot |
| 73.71% | By train |
| 6.40% | By car |
| 3.62% | By bike |About willingness to attend CHI ’24
In response to the final question about the location of the CHI ’24 conference, we received a diverse range of opinions. When asked about the likelihood of physically attending the conference in Hawaii, respondents were divided. Approximately half of them expressed a strong interest and stated that they would like to go or were likely to attend. On the other hand, the remaining half indicated that they would not attend or would prefer to participate virtually.
In the free comments section, respondents raised several concerns regarding the conference being held in Hawaii. These concerns primarily revolved around the carbon footprint associated with long-distance travel, the time and cost involved in reaching the destination, and the potential negative impact on the local communities in Hawaii.
As a final note, we would like to thank all the respondents who took the time to provide valuable feedback. We have carefully read and analyzed all the detailed comments, categorizing them into relevant themes and summarizing the key points. The input and insights shared by the attendees are very valuable to everyone involved in SIGCHI, the CHI Steering Committee, and the overall organization of CHI.
