How We Rebuilt Live Coverage for Election Night (and Beyond)
The live-coverage page that provided rolling updates on election night was the most-viewed story on WSJ.com in 2020. Behind the scenes, we spent months overhauling the live-coverage experience to support best-in-class coverage for this major event and beyond.
In late 2019, armed with a list of features, newsroom requests and new designs, the engineering team began discussing technical implementation. We decided to completely rebuild the live-coverage system from the ground up. Our challenge was to have it up and running in less than three months, by Super Tuesday (March 3, 2020). We would then continue adding features and improvements leading up to election night.
Here are some of the key features and the technical challenges involved in our journey:
- Engineering for Election-Night Traffic: At its core, we had to build a system that could quickly reach a massive audience and provide real-time updates without breaching any technical system limits. The team used Google’s Firestore technology to power the backend. We restructured our data schema to allow us to push updates in an incremental fashion. We overhauled how content is cached, so that the system could handle sharp spikes in user traffic often seen after push notifications went out to our readers. Finally we worked to make our systems and infrastructure secure, performant, reliable and fail-safe to the highest extent possible.
- Improved Experience for Readers and Editors: The new live-coverage page designs were based on user research and analytics data. The main goal was to optimize for mobile as that is where the team was expecting most traffic to come from on election night. We introduced collapsible highlights and new placements for “featured” content including posts that are pinned to the top, such as the live election-results map. We added support for multiday events by linking multiple live-coverage pages together in an ongoing story. Under the hood, we upgraded the frontend to a more robust web-rendering platform. The team also built a new tool for editors to post live-coverage content. The result was an engaging and snappy page for our readers and an improved authoring tool for the newsroom.
- Multiplatform Reach: We needed to reach our audiences in many different places beyond WSJ.com: Dow Jones Newswires, Apple News, Google AMP, Google Search, Facebook News and more. We built a content pipeline that allowed “plug and play” of new platforms through a single component rather than requiring bespoke integrations for each. This design saved us a lot of time. By election night, our live-coverage content was flowing to readers on several platforms beyond our own apps and website.
- SEO Improvements: Unlike standard article pages, live-coverage content is updated very frequently — often every few minutes. Therefore traditional tools that provide information to search engines — keywords, sitemap, story promotion — aren’t enough. We added structured metadata to the live-coverage event pages, specifically for aggregators and search engines to better understand the content. This included providing a preview of most recent posts on search results. Our content pipeline also notifies Google in real time whenever page content is updated, resulting in Google crawling the page. All these changes, done in collaboration with our SEO colleagues Edward Hyatt and Richard Nazarewicz, gave us a significant boost on search.
- Automated Election Results: On election night, we needed a way to instantaneously deliver race calls to our readers, wherever they are. We built a script that fetches results data from the Associated Press and automatically publishes updates to our live-coverage page and simultaneously to other platforms. Additionally we generated card-like images for social sharing, which were also used by our social-media editors in real-time. Social-media posts with these cards generally outperformed those without them, leading to increased engagement on social media and back to WSJ.com.
This project is a testament to how news experiences are at their best when we focus on both the product technology and content. Our engineers traveled to Washington, D.C., and shadowed coverage editors as they used the live-coverage tool. We collaborated closely with editors working on our election content and strategy, including Josée Rose, Becky Bowers and Tim Hanrahan.
In February 2020, we launched the new authoring tool for editors, followed by a new reader experience while ensuring backward compatibility for old events. We managed to deliver ahead of schedule, before Super Tuesday.
From there, given events in the news, we quickly used the revamped live-coverage experience more than we’d anticipated. The newsroom started using live coverage to cover coronavirus and the racial-justice protests. Working from home in the midst of the global pandemic, the team added support for multiday events, Apple News and Dow Jones Newswires integrations, while also making continued SEO adjustments.
Editors continued to use live coverage throughout 2020 to cover the Republican and Democratic conventions, presidential debates, Amy Coney Barrett’s Supreme Court confirmation hearing, former President Donald Trump’s Covid-19 diagnosis and more. Our final stretch was building out the automated race calls that also stood out visually on social media and infrastructure preparations for election night. All the hard work paid off and election week went very smoothly.
Our work is far from over. The team has an exciting product roadmap in front of us. Live coverage already has been front and center in the new year for the Georgia runoffs, the Capitol riot, Mr. Trump’s second impeachment and President Biden’s inauguration, and will surely continue to be a critical tool for The Wall Street Journal newsroom in 2021.
Alka Sulakhe is the Engineering Director of the Newsroom Tools Team, which creates and manages many of the tools used in The Wall Street Journal and Dow Jones newsrooms. Palani Kumanan, Peran Osborn, both Distinguished Software Engineers on the same team, and Alyssa Zeisler, R&D Chief and Newsroom Tools Product Owner, contributed to this post.
