A GeCKo Symposium retrospective
The GeCKo Symposium, on Integrating Generic and Contextual Knowledge, was initially conceived as a regular, offline scientific event in Barcelona, on May 18th 2020. The pandemic forced a choice between cancelling, postponing and holding the event online, where we decided the latter. A partial transition to online events seems desirable independently of the pandemic (e.g., for reasons of sustainability), and online events offer advantages (as well as disadvantages), so we hoped to be able to use the current urgency to gain some experience — and share it with others. That’s why we wrote this.
“We” are the organizers of the GeCKo Symposium: Laura Aina, Gemma Boleda, Thomas Brochhagen, Desmond Elliott, Carina Silberer, Ionut-Teodor Sorodoc, Matthijs Westera and Sina Zarrieß. Let us here also thank all keynote speakers, poster presenters, commentators, reviewers, and participants who made this event possible.
tl;dr
Each section below has a “too long/lazy; didn’t read”. Here’s all of them together:
- We relied on Slack as a lo-fi virtual conference center for a lobby, announcements, poster sessions, and it worked very well for our purposes.
- Pre-recorded talks to view prior to the symposium with live Q&A’s on the day itself has many advantages and some disadvantages. Some of the disadvantages could have been avoided by having fewer keynotes or spreading the event over multiple days.
- For the tiny poster sessions of GeCKo, relying on Slack for virtual poster rooms worked reasonably well. There is, however, much room for improvement in how the presenters (are instructed to) shape their own presentations.
- When organizing GeCKo we overlooked some obvious important things.
The topic of the GeCKo symposium
GeCKo concerned the challenge of how to combine generic knowledge with situation-specific information. This challenge arises in many applications of Computational Linguistics and connected areas and applications, such as Machine Translation, Natural Language Inference, and Language and Vision. We organized GeCKo in order to address this challenge in an integrative fashion, drawing inspiration from across the field. The GeCKo symposium was aimed at 1) understanding the issues involved in the integration of generic and situation-specific information in Computational Linguistics, across applications and research areas; 2) identifying ways forward; and 3) cross-fertilizing Computational Linguistics with Machine Learning, Linguistics, and Cognitive Science researchers working at this junction.
Some basic numbers
GeCKo had 7 keynote speakers (5 pre-recorded with live Q&A, 2 live talks), along with 18 contributed posters. The symposium attracted 191 registrations (registration was free and non-committal), 142 of whom also registered for a designated GeCKo Slack workspace (see below).
On the day itself, the maximum number of simultaneous attendants was around 70 for the three keynote Q&A sessions in the morning, 90 for the live keynotes session in the afternoon, and 40 for the final keynote Q&A session and closing remarks. There were no large differences in the number of attendants between the start and end of each session.
We will occasionally report findings from an evaluation questionnaire we sent out. This form was completed by 34 respondents, about half of whom were regular participants (others were keynote speakers, commentators, poster presenters and organizers). The 34 people who responded to our evaluation questionnaire attended on average 4 of our 7 live keynote talks/Q&As.
Using Slack as a virtual conference center
tl;dr. We relied on Slack as a lo-fi virtual conference center for a lobby, announcements, poster sessions, and it worked well for our purposes.
In some previous online conferences we had been missing the feeling of a shared virtual space, akin to the lobby or coffee room in a conference center, where you can say hi and chat prior to the conference and in between sessions, see who else is there, and be notified that the next session is starting. We created a Slack workspace for this reason.
See the list of channels and some example announcements in Figure 1, and Figure 2 for a screenshot of the lobby. Besides a channel for announcements (“next keynote session is starting!”) and a lobby for random chat, we had a separate channel for each keynote talk, and a separate channel for each poster session. See further below for an explanation of the keynote and poster channels.
The role of this Slack workspace as a virtual conference center was rated “very positive” by 21 of the 34 respondents to our evaluation questionnaire (with 2 “no opinion”, and 1 “very negative”, and the rest “positive”). The channels and threads structure of Slack can be a bit confusing with no prior familiarity. About two thirds of respondents said they read most of other people’s posts, and one third rated themselves to be an active participant on Slack. Almost half used the “People” tab on Slack to see who else was present, and 40% used Slack’s private messaging to get in touch with others.
One possibility, especially for larger conferences, would be to create separate channels for interest groups to facilitate more topical discussions and networking with people with similar interests. However, for the GeCKo symposium we left this up to the participants themselves.
Slack is a rather lo-fi solution, and no doubt more professional, purpose-made virtual conference center software exists. For us it worked well enough, and we are likely to use it again in the future for a similar-scale conference. For much larger conferences we expect that the moderation possibilities, user management and channel organization tools are a bit limited, and special-purpose tools will offer an advantage worth the investment.
Keynote talks: pre-recorded and live
tl;dr: Pre-recorded talks to view prior to the symposium with live Q&A’s on the day itself has many advantages and some disadvantages. Some of the disadvantages could have been avoided by having fewer keynotes or spreading the event over multiple days.
Five of our seven keynote talks were pre-recorded and made available in the week prior to the symposium, with only a live Q&A session on the symposium day itself. The remaining two were given live on the day of the symposium. Because our keynote speakers had initially accepted to give live talks (pre-covid), we didn’t want to enforce a new format on them, especially not with so little time to prepare. Therefore, we gave our invited speakers the choice between pre-recorded or live. Had we planned an online conference from the start, we would have preferred a uniform format.
20 of the 34 respondents to our questionnaire viewed at least 3 of the 5 pre-recorded keynotes, with 2 respondents having viewed none and 3 having viewed all. 21 respondents attended at least 4 of the 7 live keynotes/keynote Q&As on the day itself.
Why pre-recorded talks + live Q&A?
Both types of talks were appreciated, with most respondents seeing mostly advantages but also disadvantages to the use of pre-recorded talks. We will focus on assessing our choice of pre-recorded keynotes + live Q&A. There are three main advantages for this format.
First, passively watching more than a few live talks in a row could, online, be a bit too monotonous. Although our two live talks were appreciated too, we think a full day of live talks would not have been suitable for an online symposium. After one or two live talks, the passive viewing mode, sat in your chair behind your desk, can be quite exhausting. By contrast, the Q&A sessions are in general a bit more exciting to watch, with different people speaking and interacting.
Second, it affords greater depth of the Q&A, both because people who ask the questions (in most cases) have thought about them in advance, and because more time was available for questions and follow-ups. In addition, there are didactic reasons why a ‘flipped classroom’ is supposed to work well for teaching classes, and many of these reasons presumably carry over to the conference setting. Some advantages are rather obvious: with a pre-recorded talk, you are able to speed up or slow down the speaker, replay portions you didn’t quite understand the first time, and think longer about the question you may ask.
Third, and finally, it gives participants the flexibility to watch the talks at their leisure.
The option of pre-recorded talks + live Q&A also has two main disadvantages, however. The first is that it forces participants to make room to watch the talks outside the time of the symposium. For some, reserving a full day for a symposium is the only way to free up time and attention to watch talks. In the case of GeCKo, the day itself was filled with sessions, leaving us no option but to require that participants watch the recorded talks in the days prior. However, something which didn’t go as planned is that some speakers didn’t manage to pre-record until the final weekend before the conference, leaving too little time for people to watch the talks ahead of time. Moreover, we had some confusion, with some respondents mentioning that it wasn’t clear to them at first that they were supposed to watch the pre-recorded talks ahead of the symposium. These two aspects should improve if the format becomes more standard.
The second disadvantage is that there will be some lag between when a participant has watched the talk and the Q&A period, such that their memory of the content of the talks may fade. GeCKo was on a Monday, so if one leaves the weekend in between (assuming we had had all the talks ready earlier) the time lag is of at least 2 days.
For future events, it could be good to either have fewer keynotes, or spread out the event over multiple days, in which case one could reserve time directly prior to each live Q&A for watching the corresponding recorded talk. That way, people who prefer to cram everything in a single day, or who wish to refresh their memory on the day itself by scrolling through the talk again, have the opportunity to do so.
Starting the live Q&A with a summary
We had a designated ‘commentator’ during the first 10–15 minutes of each live Q&A, who would give a brief summary of the keynote talk and pose the first questions, optionally borrowing from the questions posted on Slack (see below). Overall this was a nice way to get the Q&A started, but it also meant that there was less time for the rest of the audience to pose questions. Instead of relying on a commentator to bootstrap the Q&A, it may also be a good idea to ask the keynote speakers themselves to start the live session with a 5 minute summary of their talk, instead of a commentator. That would also mean less organizational overhead: it was quite difficult to find volunteers (on short notice).
Collecting questions on the pre-recorded talks prior to the symposium
For the pre-recorded keynotes, participants were encouraged to submit questions on Slack prior to the symposium — this was the main intended purpose of having a keynotes channel on Slack. About 40% of our respondents posted comments or questions on keynote talks, and in many cases the keynote speaker already replied on Slack.
Initially we had only a single channel to gather questions on all the talks (with one comments thread per talk) but this quickly became a mess because people (of course) wouldn’t just post single questions but whole lists of them, as well as longer stretches of text clarifying and motivating their questions. Moreover, keynote speakers already posted their replies there, too, whereas our original plan was merely to collect those questions for the live Q&A on the day itself. For this reason we split the keynotes channel up into a single channel per keynote.
One reason for our original plan, to use Slack only for collecting questions, was that we did not want to burden the keynote speakers with having to monitor Slack; instead, only commentators and session chairs would need to take a look there prior to the live session, so that they could pose any relevant questions to the speaker. This plan should have been communicated better by us, but in the end the more interactive use of Slack in this regard seems to have been appreciated anyway.
How to have audience participation online without risking chaos?
Having experienced some online conferences where the only way to speak is to enter a question that gets accepted by an invisible moderator, for GeCKo we really wanted to give our audience the ability to unmute themselves (and share video) without such top-down intervention (except for a session chair, as in an offline conference). But we were a bit weary: we did not know how many people would attend, and we were not very experienced with online conferencing ourselves, so we were afraid that having 100+ people with the ability to unmute could be chaotic. (Note that having more than 100 participants with this ability would have required Zoom’s ‘large meeting’ extension.)
In retrospect these fears may have been unwarranted. Moreover, our audience turned out to consist of at most 90 people at any given moment. This made the audience division unnecessary and merely added overhead as well as some confusion and feelings of unease among participants. Therefore, for a similar scale conference in the future, we probably would not bother with the audience split and simply make all attendees regular participants (or ‘panelists’ in Webinar mode), with the ability to unmute themselves if they want. People will generally behave, and if not, then there’s a moderator (or multiple) who can mute and ban.
How we implemented the audience split
For the sake of completeness, here is how we implemented the audience split (to be clear: we would not do this again for a similar-scale event). We split the audience in two parts, which we called ‘active’ audience (with the ability to unmute and share video) and ‘passive’ audience (spectator only). In our registration form we asked whether people preferred to be in the active audience or in the passive audience, with some explanation of what this entailed (but which not everyone may have registered). About one third of registrants indicated a preference for passive audience, one third for active audience, and one third no preference (we simply included them in the active audience). This initial audience division notwithstanding, we were pretty flexible during the day itself, and passive attendees who liked to participate more actively could simply ask our helpdesk (on Slack) and they would be ‘promoted’ to active audience.
On the implementation side we used Zoom’s Webinar mode for this audience division, which is a Zoom functionality you may need to request separately. It allows you to have an audience split in two parts, with up to a hundred panelists who can unmute themselves, and the rest spectator-only attendees. The Zoom terminology of ‘panelist’ and ‘attendee’ didn’t quite reflect what we had in mind (‘panelist’ sounds a lot more demanding than simply having the ability to unmute yourself), and this may have resulted in some confusion initially. But other than that, Webinar mode served our purposes just fine.
For larger events, other, more adequate tools may exist for allowing an unlimited passive audience to spectate, such as live-streaming to another platform (e.g., youtube). We did not look into this possibility for GeCKo.
Poster sessions
tl;dr. For the tiny poster sessions of GeCKo, relying on Slack for virtual poster rooms worked reasonably well. There is, however, much room for improvement in how the presenters themselves (are instructed to) shape their presentation.
Besides our keynotes, we had contributed abstracts or papers to be presented as posters. We had three short poster sessions with 5, 5 and 8 posters, respectively. Each poster session directly followed a keynote session, starting with 1-minute-per-poster booster talks in the same video conference room as the keynotes. Here, the poster presenters would unmute themselves and give a 1 minute talk one by one, while one of the organizers shared their screen and clicked through a single PDF containing all the slides concatenated (title slide + content slide per poster). We didn’t want poster presenters to have to share their own screen, because that often takes a while to set up, they may share the wrong window, etc., which wasn’t worth the hassle for just 1 minute each. This booster format and implementation worked really well.
After the boosters, the session chair would direct our audience to Slack, which had a channel set up for each of our three poster sessions. There, each presenter had announced their poster in advance, with some basic info and a URL to a zoom/hangouts/etc. room hosted by themselves. We offered them the choice of which video conferencing system they used, with a preference for either Zoom (in line with the main sessions), and otherwise something that runs in the browser and requires no prior installation. We think mediating the poster sessions with a Slack channel worked well in our case, with our relatively small number of posters, but for larger poster sessions a special-purpose website might be better; a Slack channel offers very little in the way of sorting, organizing or commenting on the posters in a structured way.
The poster format itself needs some thought to work well on-line, but perhaps in respects that are more the responsibility of the poster presenter than of the organizer. For instance, many poster presenters essentially gave an ordinary conference presentation, showing slides and giving long talks. We didn’t explicitly instruct them not to do this — in fact we offered very little guidance — but next time we definitely should. Perhaps one should even instruct the audience, at the start of the poster session, to be assertive and interrupt poster presenters who are doing this, i.e., “feel free to interrupt and ask poster presenters for a 1 minute summary, and feel free to leave at any point”. Some of our participants indicated they felt they were stuck with a single poster for half an hour. It meant that there was often only time to view one or two poster ‘presentations’ from start to end, and that’s not really the point of a poster session. Nevertheless, 20 of our 34 questionnaire respondents still rated the poster session as “positive” or “very positive” (with most remaining people saying “no opinion”).
Some things we did not think of but should have
tl;dr. When organizing GeCKo we overlooked some obvious important things.
What if the connection is too bad?
Our sound/video experience was excellent all day, except in the final session, where one keynote speaker and one commentator seemed to have a slow connection. We later learned that we could have asked them to dial in by phone instead, which is a functionality Zoom offers. In any case, in our case these problems were a bit annoying but not too severe, which is good because, for some reason, we had not really prepared for this eventuality. (We did consider ‘What if the meeting host dies?’ 💀, but not this.)
What about harassment in an online setting?
We did not have a code of conduct, for instance with a protocol for harassment cases. Although to our awareness there were no unpleasant interactions in Slack or Zoom, the online environment in general and Slack and Zoom specifically may bring new challenges in this regard. For instance, it is relatively easy to join our Slack workspace anonymously and send private messages to attendees. Anticipating these challenges and having an explicit code of conduct would help one deal with such eventualities and encourage victims to report cases.
What about diversity and accessibility in the interactive sessions?
Beyond striving for diversity among our presenters and commentators, we did not have an explicit approach towards diversity among our participants, leaving some aspects implicitly to the chairs (e.g., ensuring diversity among those asking questions, for instance by trying to take the first question from a woman: see https://oxfamblogs.org/fp2p/how-to-stop-men-asking-all-the-questions-in-seminars-its-really-easy/).
Other aspects of diversity and accessibility we simply did not consider. For instance, we did not consider the possibility of having automatic transcriptions for the talks (an attendee suggested www.otter.ai).
Lastly, since GeCKo was originally planned as an event in Barcelona, most keynotes and contributed talks as well as the organizers were in Europe, and with our schedule we stuck essentially to the regular working day according to the Western European timezones. This made our event less accessible to people in different timezones. Now, physical conferences may have a similar location bias, since it is cumbersome and costly to travel around the globe, especially for a one-day symposium. But one would hope that online events could in the future do a better job than we did at avoiding the analogous restriction.
