The UN PET Lab’s first Hackathon: Results, Analysis & Key Takeaways

The United Nations PET Lab’s Hackathon last November demonstrated the ingenuity of privacy-enhancing technologies. The 72-hour long competition saw 196 teams, joining in from across the world, put theory into practice as they were one of the first to experiment and deal hands-on with PETs.

We organised the Hackathon with collaborators from the Federal Statistics Office in Switzerland, and with core developers from Harvard, IBM Research, and Microsoft; to put together a variety of leading privacy data science tools and frameworks that participants utilised during the challenge. The dataset itself came from our partners at the UNHCR, and although safe to use, emulated the types of challenges refugees face in day-to-day context.

Reflecting on the experience of running our first international Data Science Hackathon, we were impressed with the overall accuracy of the results. This blog post is an insight into how participants engaged with the resources provided in order to overcome the age-old privacy-utility trade off: Preserving the informational content and utility of sensitive data while providing better security and privacy protections than ever.

Read more about the competition in detail here.

The Top 5 🙌

Here are the top 5 teams!

1. 🥇 ‘Oxford Team’ with a score of 0.8042
2. 🥈 ‘integrate.ai’ with a score of 0.7686
3. 🥉 ‘ONS Ventura PETs Detectives’ with a score of 0.7652
4. ‘DPForever’ with a score of 0.7319
5. ‘LostInTheNoise’ with a score of 0.7316

Libraries of Core Developers

Participants could choose from the following libraries to analyse the data during the challenge:

1. differential-privacy-library — (IBM Research)
2. OpenDP — (Harvard)
3. Smartnoise-synth — (Microsoft)
4. Smartnoise-sql — (Microsoft)

Queries

Our records of the queries made throughout this competition have revealed the best-performing strategies used by participants, ensuring everyone had a fair and open playing field.

📌 1469 total queries were made during the Hackathon.

📌 1227 queries were made by the teams who submitted results at least once.

📌 13 teams, however, made some queries but did not submit anything. There were 242 of such queries.

📌 4 teams did not make any queries, but submitted results.

For further analysis into the queries made by the participants, we’ve grouped teams into:

1. Group 1 — Teams who made queries and submitted their result and,
2. Group 2 — Teams who made queries but did not submit any results

Figure 1 — Number of queries from group 1 teams

Figure 2 — Number of queries from group 2 teams

Teams in group 1 made the most number of queries with smartnoise_sql and the overall feedback was that the library was, for the most part, user friendly.

Figure 3 — Count of query type for group 1

Teams in group 2 made the most number of queries with smartnoise_synth, but did not submit any results. We received mixed feedback, with some participants finding the library useful, and others faced with a few difficulties down the road to generate any meaningful synthetic data. Despite the challenges faced, the overall results show that the top 10 winning teams used smartnoise_synth the most.

Figure 4 — Count of query type for group 2

Here are the precise numbers of libraries used by the top ten teams:

1. smartnoise_synth: 82
2. Diffprivlib: 44
3. smartnoise_sql: 27
4. opendp: 33

Figures 5 and 6 below show the exact breakdown of the libraries used by all the teams in groups 1 and 2.

Figure 5:

1. 183 smartnoise_synth queries in total
2. 198 smartnoise_sql queries in total
3. 187 Diffprivlib queries in total
4. 42 opendp queries in total

Figure 5 — Library breakdown for teams in group 1

Figure 6:

1. 149 smartnoise_synth queries in total
2. 62 smartnoise_sql queries in total
3. 139 Diffprivlib queries in total
4. 17 opendp queries in total

Figure 6 — Library breakdown for teams in group 2

Visualise the data easily with figures 7, 8 and 9 providing graphical representation of the cumulative sum of query types for groups 1 and 2.

Figure 7 — Cumulative sum of each query type for group 1

Figure 8 — Cumulative sum of each relative proportion of query type for group 1

Figure 9 — Cumulative sum of query type for group 2

Some overall stats…💡

The most popular libraries used by participants is a tie between Smartnoise Synth and Diffprivlib. Here are the full numbers:

1. smartnoise_synth: 18 teams
2. Diffprivlib: 18 teams
3. smartnoise_sql: 17 teams
4. opendp: 6 teams

Diffprivlib had a median score of 10. Interestingly, whenever participants used Diffprivlib, they experimented and tried out more options.

Epsilon and Delta spent 💸

Participants started off with 0 epsilon and 0 delta used, and their access to the sensitive datasets were through REST API Calls. On every call, participants spent their privacy budget but in return learnt something about the underlying dataset. Their ranking was based on the combination of their accuracy minus the privacy budget spent on them.

Figures 10 and 11 show the mean and the maximum total epsilon used by participants.

Figure 10 — Histogram of mean of total epsilon spent

Figure 11 — Histogram of maximum total epsilon spent

Figures 12 and 13 show the mean and the maximum total delta used by participants.

Figure 12 — Histogram of mean of total Delta spent

Figure 13 — Histogram of maximum total delta spent

Finally, figure 14 below shows the summary of how each team spent their budget, how they made queries to work out an overall winning strategy and how much each query truly can truly cost you in the competition.

Figure 14 — Query summary of all teams

Through trial-and-error, participants learnt the importance of careful consideration in their application of epsilon and delta spending, as each decision carried great weight.

A few words from Oblivious ✍️

Once again, a huge thank you to all the participants, mentors and team from the Federal Statistical Office, UNHCR, OpenDP, Diffprivlib, and Smartnoise whom this event would not be possible without. We are proud to have formed an invaluable, inspiring community during these past few months.

A special mention to Pauline Maury-Laribiere at BFS Switzerland for the original analysis, and Girish Joshi from Oblivious for writing up this blog post based on Pauline’s jupyter notebook 🙌

If you’re keen to join us, our Discord server is the perfect space to be for meaningful connections, discussions and upcoming opportunities!

If you’d like to take part in organising or even just participate in our hackathons, drop us a line at hello@oblivious.ai. You can also try out the tooling yourself by signing-up for the free version of our product today.