Software evolution to serve a global community
Before I joined Meedan as a software engineer in late 2011, this amazing team had already noticed that new media was ending the scarcity in which mass media journalism functioned. This was having repercussions for both newsgathering and publishing—no longer were audiences solely reliant on foreign reporters to tell the story. This was particularly true of the reporting of the Middle East, where citizen media been used so powerfully in the midst of such colossal social and political change during the Arab Spring. And citizen media should be verified.
That’s when Meedan started working on this problem — clearly a very difficult nut to crack. It all started with a participatory liveblog with leading Egyptian independent daily Al-Masry Al-Youm.
Less than one year after I joined Meedan, around October 2012, the software engineering team (myself included) started working on a second iteration of this platform for collaborative fact-checking, which at that point was called Checkdesk, which we launched in July 2013.
Checkdesk was a verification desk for the digital newsroom, designed to help curate user-generated content during breaking news and to connect journalists to citizen sources on the ground. We developed it in cooperation with six leading media outlets in the MENA region: Al Masry Al Youm and Welad El Balad Media Services in Egypt, 7iber in Jordan, Nahar Ashabab and Social Media Exchange (SMEX) in Lebanon, Ma’an Network in Palestine, and Al-Ayyam in Syria.
From a technological perspective, Checkdesk was built as a monolithic web application using Drupal, an open source platform for web content management. As a CMS, Drupal was among the most successful and seemed like a great fit for our needs, because it strikes a great balance between getting up and running in no time, and the ability to extend its functionality almost infinitely. It served us quite well between 2013 and early 2016 and during that time we had the opportunity to make numerous contributions to its community and attend some Drupal conferences to share all the expertise we gained with that work.
As our applications grew, we started facing scalability challenges with Drupal. These challenges manifested in many ways, from code maintainability to lower performance, proliferation of defects and automated testing. It was time for us to rethink our software architecture. We had to implement more features, go mobile, support more languages, handle more content, serve more users and face new social events—mainly elections.
We accepted a big challenge at that time, which was to completely rewrite Checkdesk, in six months, from scratch, with a completely new code base.
Personally, I had some good pre-Meedan experience with the open source web application framework Ruby On Rails since 2009. During my early days at Meedan I had the opportunity to implement a prototype in Ruby On Rails for social media translation, which was called Translatedesk. This prototype evolved into a micro services architecture made of several Ruby On Rails APIs which were at the core of our translation product, Bridge.
So we decided to pursue a similar software stack approach as Bridge and re-implemented Checkdesk as a multi-service application: some GraphQL APIs written in Ruby On Rails, a web application using React.JS and, as usual, everything was released as an open source project. We also started a new culture for our software development process, which includes writing automated tests for everything seeking 100% code coverage, continuous testing, code quality with maximum GPA at Code Climate and containerization of our softwares.
Not only the code was rewritten, but also the branding: Checkdesk was brought out from behind the ‘desk.’ The 3.0 version of our open source software was rebranded as, simply, Check.
Check provides a central workbench facilitating online verification newsroom collaboration, allowing participating journalists and editors a simple and powerful framework to work together to investigate and annotate links, media and claims. This new code base has been serving us well since then, and during that time we implemented many improvements, new features and performance enhancements in order to accommodate its increasing usage. Since we first released it in October 2016, it was used by OJA-winning election monitoring projects like Electionland (United States, 2016), CrossCheck (France, 2017) and Verificado (Mexico, 2018), in addition to similar projects in Asia, namely Indonesia, Philippines and India.
In India we had a new big challenge to face: misinformation on WhatsApp. WhatsApp plays a significant role in elections in many developing countries, and in recent years WhatsApp-based rumors in India have caused misinformation issues that in some cases led to violence and death. In December 2018 we started to integrate Check into the WhatsApp business API and released this feature in early April 2019, just in time for the elections in India. Check Message was born: our tool for fact-checking against disinformation at scale, seeking scalability, reliability and great performance.
Our collaboration in India, called Checkpoint, was officially launched in April as the first real use case of Check Message. In total, Checkpoint gathered over 80,000 claims for verification and our software was useful to group duplicate requests, allow similar items to be related, prioritize most requested items and, in general, optimize the fact-checker’s work as much as possible.
After this first release, we generalized our software module from WhatsApp-specific to more platform-agnostic, and we now support other closed message platforms, like Twitter DMs and Facebook Messenger. This module was implemented as a Check bot. Check allows for extensible scripts and functions that help you augment your verification experience with tools like private messenger integration, archiving and language detection. We call these scripts bots, and they live in the Bot Garden. We hope that this bot infrastructure can be used by other developers around the world to integrate Check with other tools and with their own workflows.
We keep working on Check in order to improve user experience, content ingestion and augmented verification. Stay tuned! What I can say is that I’m really enjoying the things we are working on right now and really excited for the next challenges we’ll face and the new software support we’ll implement in the upcoming years in order to make Check even more global. I invite everyone to join us on that. If you are a software developer interested in open source and in these challenges we’re tackling, get in touch on GitHub. If you are journalist and want to use Check for your project, get in touch with us on our website.
Ultimately, it’s a collaborative effort.
