Move everything online.
How to run big (neuro)science conferences online — neuromatch.io
Moving fast, breaking things, enabling community
Dan Goodman, Brad Wyble, Titipat Achakulvisut, Isil Bilgin, Sofie Van Den Bossche, Konrad Kording
Conferences are being forced online by the coronavirus, they should move online because of Carbon footprints, and they have an obligation to move online for reasons of inclusion. So the question is how we can rescue the various aspects of good offline conferences into the online world. We just ran neuromatch.io, an online only neuroscience conference that ended up being popular (~3k registrants, ~1k maximally viewing simultaneously, 2k online, ~400 chat messages/hour, ~500*6 matches, overall cost <$200). We feel that we succeeded at moving a lot of the good things of legacy conferences into this new medium. We failed fast. For example, the first open mike discussion was not great, but we managed the next day, within 5 hours, to completely change the format. In this blog post we will go through the components that worked for us.
PART I: MANAGEMENT
First, conferences have a broad base of scientists who present short talks or posters. For that we need a forum for such presentations. Posters are not great online and we thus instead utilized talks of 12 minutes duration with 3 minutes afterwards reserved for discussion. We had considerable tech issues but for the future we have a near perfect solution(see below). Another problem relative to, say, posters, is the limited time there was for interactions with the audience. However, there is a massive advantage of using short talks: they are perfectly scalable. By running six sessions in parallel we were able to accept every one of our abstract submissions. Why should we as organizers or reviewers decide which talks are given? People can vote with their feet. And if someone’s talk is only for a specialist audience, who are we to prevent them from talking? Broadly accessible short talks felt far more inclusive and the talks were of an impressive quality.
Second, conferences tend to have contributed talks. The abstract submissions with the potential for widest appeal are often chosen at conferences to be presented in front of a large audience. We thus upgraded 10 of the talks to longer 18 minute talks followed by 4 minutes of discussion. Some of the best discussions with the community were sparked by these contributed talks. A key to the success of all of the talks is how they are moderated and while there is no science about that Konrad wrote down some observations what worked for him. We believe that contributed talks are important as it gives early career researchers the chance to highlight upcoming research.
Third, conferences have established scientists give invited talks. The difficulty in recruiting these scientists is that they tend to be busy. This was remarkably easy given that there are somewhat fewer distractions in these times. It is also far easier because giving a talk no longer involves spending days of traveling to be at the right place at the right time. Second we need a format. Online discussions are far better than real-world discussions because people can vote on the questions. Not only that, but the questions are even linked to the video so we can search for question text on google and get linked right into the crowdcast in the recording. As such we reserved 30 minutes for the talks and 15 minutes for the discussion. We found the discussion and engagement driven by these invited talks to be of incredible quality.
Fourth, conferences have biobreaks. We found that having a break of about 10–15 minutes every hour worked very well. People could have a short time off, the organizers had time to stage the next speaker. Short breaks are very attractive in online meetings. During these short breaks some participants would chat with one another (we never blocked the chat) about various scientific topics. It also gave us organizers the chance to continuously optimize the parameters of the conference.
Fifth, conferences have discussion sections and socials. On day 1 we ran (on crowdcast) an open mike discussion section where anyone could ask questions and come on stage. We found this to be somewhat difficult; the discussion simply drifted relatively fast and it was hard to give it meaningful structure. It was nice to see many people contribute but it felt like we could never get closure on questions. We thus decided during day 2 to completely change the format. On day 2 we ran one session where we invited two speakers to have a moderated discussion (Richards and Cisek) and it worked very well. Each made their initial point for 10 minutes. And then we started a discussion between the speakers and later included questions from the audience. It worked so well, in fact, that many people felt that we simply did not have enough time. We also ran another session where we had 5 scientists, each making a point for 5 minutes. They then discussed with one another and the audience about these issues. We found that sticking to strict timetable was very helpful. The discussion, again, was great. A challenge for the moderator was keeping the contribution length to be of a similar order of magnitude. As a social, we asked participants to show us where they are and show us their pets/ artwork/ kids/ lives. It was great fun. But altogether these discussions on day 2 and socials both worked considerably better than their legacy equivalents but it still felt like there could be massive improvements by playing with the formats.
Sixth, conferences have one on one discussions, usually between people working on similar issues. To mirror this real world aspect we asked participants for an abstract that characterizes what they are interested in. We then used an algorithm to assign people to the six other participants that are most similar to them in interest (see technology section below). We have run this mechanism a good number of times in legacy conferences (Computational Cognitive Neuroscience) and it worked great. We are confident that these meetings between the 500 people that signed up for them will lead to a good number of collaborative projects and, maybe more importantly, will help the community build bridges.
Seven, chat between participants. All our main sessions were in crowdcast. So people could chat in realtime about the presentations. This chat channel was often even more useful than the talk itself. People were giving links to papers for background. They were answering beginner questions. They were clarifying points of potential disagreement. We had designed a code of conduct that never got broken during the conference (https://neuroonline.github.io/). Altogether the climate was incredibly positive there. And no one got shushed for explaining something while the speaker was talking.
Eighth, conferences have a time and a place. Online conferences have more time and more place. We had participants on just about every timezone. Running just a few hours a day would have excluded many scientists. We thus ran very long days (day one ended considerably past 9PM EDT, day two started at 4AM EDT). The cost for adding extra sessions to legacy conferences is that participants get tired. But in an online world viewers do not need to see all talks. In fact, for just about any participant there will be talks that they can not see. Any hour is good for some participants and bad for others. So the only fair thing seems to be to run for much of the 24h period. Also as everything is recorded the real-time need is far reduced. Our participants did not complain much about the timing. An online conference effectively has more time.
Ninth, conferences do marketing. Ours was largely limited to twitter and to a collaboration with the journal eLife. But regular tweeting about the conference along with buy in from a good number of people, above all the invited speakers produced a good amount of buzz. We had 1500 sign ups before we started neuromatch. That number doubled through a rather massive twitter presence during the period of neuromatch.
Tenth, conference organizations have an organizational backbone. In legacy conferences nearly all of the organization is done ahead of time because solving problems in real time is difficult given that most of the organizers are not online at any given time. With online conferences, almost everyone is accessible during the conference. This makes it much easier to deal with problems as they occur because resources can be redistributed to address problems with flexibility that is not feasible in a physical conference. We found that, in a time of rapid change, formalized committees are far too slow. Instead we operated with an ad-hoc, idealism based, system that was never truly organized and still, or maybe because of that, would move incredibly fast. For example, under zoombombing we had groups of people working on the problem within minutes and developed new guidance extremely quickly. But in the end we had a small team of two tech specialists who worked for a few weeks, a handful of core organizers who mostly worked actively for about a week, along with a dozen some reviewers who worked for a day, pull off a conference that would traditionally have taken a year and a large team to organize.
There are many other emerging online conferences to learn from. We have been following them, tried to learn from them, and received help from many people involved with these conferences. For example, Twitter conferences such as OHBMX have emerged. To help with the coordination of online conferences we have started the initiative Neuro->Online, Now which communicates actively on mattermost. We want to invite all our readers to help us improve online conferences. We expect a burst of incredible innovation that will make online conference strictly superior to their legacy equivalent.
Maybe on important factor to consider is the impact of moderation and emotional factors. We found that striving for positivity and inclusion is crucial. Here are two comments we want to share “The easy going style of the organizers — helps with the socializing and bringing in the human aspects of the talks.” and “It seems to be less formal and less status-aware to me than other conferences, and I love that.” Positive community is one of the central goals of any conference. A supportive environment should be the goal of online conferences and we believe they can fill that role.
PART II: TECHNOLOGIES
Online conferences require great technolgies to make them work. We will discuss here what worked for us (crowdcast + youtube+ homegrown matching+slack or mattermost) and what did not. We will focus on the aspects under our control. The obviously important factor that people without fast internet connections can not meaningfully participate is crucial. As is the fact that some communication protocols are blocked by some countries. Here we review the factors that are under our control.
Short talks. We used zoom as platform for these which may have been a mistake. Zoom proved to be overly prone to zoombombing, trolls would constantly come into rooms trying to share annoying material. There are ways of locking down meetings that avoid that (see our script). We only learned too late that it is possible in crowdcast to run parallel sessions. Slightly suboptimal is that to do so you need to share the crowdcast login with a number of co-hosts. But for typical meetings with less than ten parallel sessions it is reasonably easy to find the trusted co-hosts.
All other conference talks. We needed software infrastructure and we used crowdcast with live streaming to youtube. Crowdcast makes it feel as if it is just a very small group of people you are dealing with. Which really put all speakers at ease. It also makes it easy to ask the right questions — the audience will propose questions and vote up the good ones. The questions at neuromatch were far better than those at legacy conferences. It also allows asking the question askers on stage. This was really useful as it highlighted the community nature. Crowdcast also records the events and then shares them after the transmission is over. Crowdcast did required some juggling of participants as it only allows at most 4 people (including the moderator) on stage which required some scheduling but we found it easy enough to deal with it. Live streaming on youtube further improved the experience. People can rewind on youtube and they can speed up or slow down the presentation. They can use this to catch up. Moreover, there are some countries that block crowdcast but allow youtube. Lastly, people with very bad connections get better videos from youtube. The crowdcast/ youtube combination seemed to fulfill all the needs of our conference attendees and felt incredibly personal.
One-on-one matching: We used a standard natuaral language preprocessing algorithm to embed all abstracts as topic vectors. We then used a linear programming approach to assign scientists to similar other scientists. The full code can be found here (https://github.com/titipata/paper-reviewer-matcher) and is easy to use.
Communication: We used slack, mattermost, and other virtual channels. Our conference suffered a bit from having several different communication formats (email, slack and mattermost). It would have been better had we spent some time in advance picking a platform and setting up the necessary channels. We also had a web page which was beautiful. However, it was suffering heavily from the load of a total of 16,000 users. Towards the end of the conference we had hardened it to stand the travel. But big providers (dropbox) tend to be more stable than the homepage we built so we ended up also serving our agenda on dropbox once the server got hot. If you run your own page and your conference goes viral you will have some trouble.
What we’ve learned and how we think we can improve
- In many ways online conferences can be better than legacy conferences
- With crowdcast talks can be made feel very personal
- Just like a legacy conference, an online conference should have all the components
- Matching is really useful for producing interactions
But we could have been better:
- Parallel sessions in crowdcast would have been far better than our zoom solution
- The matching and the rest are still not seamlessly integrated
- Open mike discussion sections are problematic
- Communicating