Why Chat at YHack?
Analyzing the application of chatbot technology for hackathons
Recently, we’ve been creating Facebook Messenger bots in hopes of leveraging the existing Messenger platform to alleviate common communication issues. One shared experience that our team reflected on was the friction in interaction between staff and participants at hackathons — on new campuses and under time pressure, participants need an efficient way to get the information and help they need to have a great hackathon. Similarly, hackathon organizers need a reliable way to send updates to the participants and promptly address any questions that may come up. Several months ago, we had the opportunity to work with the staff at YHack, Yale’s annual hackathon, to design software that would address these concerns for both staff and participants. We spent the weeks leading up to the hackathon implementing our platform and ultimately delivered two complementary products: a Messenger bot for hackathon participants and web-based admin panel for the staff.
Overall, we were pleased with how this first “test run” went and are excited to continue improving and polishing our platform. In this post, we will introduce the features of our product and share some of the insights we gained through observing participant and staff behavior during the hackathon and from analyzing the message transcripts afterwards.
The Big Picture
Throughout the 36-hour hackathon, our messenger bot handled 2,073 total messages from 226 different users. Breaking down the messages received by hour, we can get a read for the engagement of hackathon participants over time.
We can see three main usage spikes during the hackathon, as labelled on the graph:
- Soon after the opening ceremony, when the bot was announced to the participants.
- Halfway in, during the afternoon of the second day. Spike aside, usage around this time could be read as a general stabilization of platform use as teams grew more accustomed to the bot.
- Right before the hackathon ended, comprising of last minute questions and concerns as hackers figured out how to submit their projects. Though this information had been announced in other forms, it was exciting to see that the hackers turned to our Messenger bot to find this information and ask more nuanced questions about the submission process.
One unintended result from our analysis is using the bot activity as a proxy to analyze the general sleep trends of the hackers at YHack — on the first night, activity bottomed out around 6 AM; by the next night, hackers seemed to be more fatigued and only made it to about 2 AM.
Quick Responses
The easiest way for hackers to access the bot’s functionality is through Messenger’s persistent menu, the hamburger icon in the lower-left corner of the chat. For the hackathon, we categorized our functionality into three choices:
- General info: With this choice, hackers could see some basic information about the hackathon set and updated by the admins (i.e. location of bathrooms, showers, snacks).
- Help session: With this choice, hackers could start a direct line of communication with hackathon admins to ask more personal questions.
- Upcoming events: With this choice, hackers could see the nearest food, fun, talk, or ‘other’ event occurring.
After the hackathon, we charted the distribution of how these choices were selected:
As the graph shows, the distribution was relatively evenly distributed, with the upcoming events comprising the largest proportion of responses. Of these, the most popular type of awaited event was (what else) food, totaling 98 of the 274 queries. Overall, we were glad to see that all categories were substantially used.
Help Sessions
Of the three menu options, we were most looking forward to seeing the live help sessions in action. While we hoped that the quick responses would allow for many questions to be instantly answered, we knew that there would be more complex questions that still required responses from live members of the hackathon staff. To make use of the convenience of the bot platform, we built a way for users to directly communicate with staff within the same Messenger chat window.
During the hackathon, 158 help sessions were initiated and resolved, comprising of 764 help messages. We were happy to see so many questions being answered through our platform, allowing for hackers to more effectively seek help and address issues. One potential area for improvement we noticed was having to explicitly start a help session — people often began to type their questions without doing so, thus causing them to have to resend their messages once they did start their session. Because having to explicitly start chats is different from the flow users are accustomed to in a chat setting, we hope to improve this in future iterations and enable users to send messages at any time.
On the other end, we were also able to observe how staff interacted with the help sessions. While they generally responded promptly, there were stretches where no staff was manning the sessions, and thus users’ questions were not answered. In the future, we may suggest to hackathon staff to choose one person whose designated role is to keep an eye on the bot, or to use the bot to notify staff whenever new help sessions are created.
Feedback
After the hackathon, we sent out a poll to participants asking them to describe their experiences using the bot. Although we had plenty of contact and feedback from hackathon staff, we wanted to see how the actual users liked the bot. We sent the following question and waited for the responses to roll in:
“Thanks for coming to YHack! We hope you had a great time. Please let us know what you thought about using the Messenger bot by rating us from 1–5 below (1 — not useful at all, 5 — extremely useful) and sending any comments our way — any feedback would be greatly appreciated!”
In the end, we had 96 of our users respond:
Our final average rating was 3.646 — it seemed that most users enjoyed using the bot and felt that it added to the hackathon experience. Reading through more detailed user feedback, we were also happy to see that people had a great time interacting with our bot, with many citing the bot as a quick and easy way to get help.
In the future, we hope to improve how we gather feedback by building a more formalized process, potentially linking to a custom survey after users rate their experience. Adding a way for participants to leave feedback at all times could also lead to more users leaving feedback whenever they run into an issue or bug. Additionally, many hackathons send out a post-hackathon survey to their attendees. Adding a section in this survey about our Messenger bot might improve the quality of the feedback we receive as well. Since the hackers will already be reflecting on their hackathon experiences while answering questions, it’s likely that they’d be able to provide more concrete and detailed examples of how they used the bot and what they thought about it.
Acknowledgments
Thank you for reading about our experience — we hope you found our insights interesting! In future blog posts, we’ll take deeper dives into the different areas of our bots and walk through their development and iteration processes. We’d also like to extend special thanks to the YHack staff for allowing us to pilot our hackathon platform at their hackathon. They were extremely supportive of our platform and accommodating of our (many) questions and concerns, and we are grateful to them for the time and effort they spent publicizing our bot to hackathon participants. Lastly, we’d like to thank all of the YHack participants for serving as our pilot users — we’ve received a ton of data and feedback that we’re excited to integrate into the next iteration of our platform.
If you run a hackathon (or other event) and are interested in a chatbot, let us know! Email us at bytesizedevelopment@gmail.com to learn more about the custom services we offer.
