Tech Fails: BBC’s £100M Digital Media Blunder
Two out of three software development projects end with partial or total failure, and the most substantial projects fail most often. This effectively wastes billions of dollars and thousands of person-hours in value every single year (1). In this series, we look at preeminent failures in big tech projects, try to uncover the factors that led to the disaster, and figure out the lessons to be learned.
Today, we’re investigating the demise of British Broadcasting Corporation’s Digital Media Initiative (DMI) project of 2008. Envisioned as an ultramodern digital media catalog, the DMI would allow staff to quickly produce high-quality audio and video content directly from their computers. Five years later, the BBC finally terminates this taxpayer-funded project with almost nothing to show for it, after spending nearly £100m.
The Project
The DMI was a complex project, aimed at transforming how BBC leveraged digital media to produce content for its audiences. The vision was to pave the way for “a new creative environment in which production would take place in a digital file format from the moment of filming to the moment of consumption by the audience. (2)”
In February 2008, the BBC Trust approved the DMI scheme, as well as £81m in funding. Technology company Siemens was contracted to develop the project, and completion was expected within the following year. Multiple components of the new media pipeline were under development:
- An Online Archive of BBC’s programs, in both digital and physical media format, allowing search and metadata.
- A Core Database scoped and architected like an enterprise-wide system for data management across the BBC and a technical platform not just for DMI but for other programs.
- A new set of Production Tools, allowing BBC staff and partners to develop, edit, share, and manage content from their desktops.
- A Media Infrastructure component, designed and built with a focus on Production Tools, enabling users to ingest, organize, edit and send content to craft or archive using through a single interface.
Fabric will benefit everyone involved in the development, creation, sharing, and management of BBC’s content by bringing together the production and enterprise processes and tools through a web-based user interface and universal media storage for archive and production needs. (2)
The Demise
By February 2009, Siemens still had nothing to deliver. In September 2009, after months of criticizing and finger-pointing between Siemens and the BBC, the DMI contract to Siemens was canceled by mutual agreement, in a £27.5m settlement — with BBC losses already amounting to £10.7m and 21 months of project delay.
By September 2010, one year after bringing the project in-house, the National Audit Office estimated that the expected project costs now exceeded benefits by over £38 million. The new completion date was now delayed to around Q4 2017. Still, their official 2011 report was not all negative towards BBC’s stewardship, with the head of the NAO stating that “since taking the program back in-house, delivery of the system has progressed well, and users have responded positively. (5)”
In 2011, delivering the infrastructure for DMI is outsourced to a consortium of three third-party companies. Interviews with the BBC team revealed that before this transition, “the original implementation using open source Drools rules engine faced memory leak instability and was eventually deemed too difficult to resolve. (2)” A BBC spokesperson said: “The consortium is primarily responsible for delivering the infrastructure for DMI as this was the most cost-effective way to deliver this part of the project. They are on schedule for delivery by the end of the summer. (8)” They were not.
In May 2013, the BBC decided to keep the archive database component but shut down the rest of the DMI program. The new BBC Director General at that time, Lord Tony Hall, admitted that “the DMI project has wasted a huge amount of license fee payers’ money and I saw no reason to allow that to continue which is why I have closed it.” BBC’s technology chief, John Linwood, was fired in July 2013 over the project’s demise. Linwood had been appointed in 2009 from Yahoo and had formerly worked at Microsoft.
A Public Accounts Committee report underlines the scale of the failure: only a single show was ever actually produced using the DMI platform:
The BBC also told us that it was using the DMI to make many programs and was on track to complete the system in 2011 with no further delays. This turned out not to be the case. In reality, the BBC only ever used the DMI to make one program, called Bang Goes the Theory. (6)
Being unable to benefit from the platform developed by their own parent company, some units took things into their own hands. BBC Sport decided, at the time, to contract outside suppliers to design its own digital video archive system at the cost of just £500,000 — a fraction of the millions spent on the DMI (7).
Where did all those millions go? Here’s the official breakdown: (4)
- Contractors — £46.7m
- IT — £37.2m
- Siemens costs — £24.9m
- Consultancy — £8.4m
- BBC staff — £6.4m
- Other — £2.3m
What Happened
Multiple expert reports have been published to cast some light on the motives behind the DMI project downfall, including one from the National Audit Office in 2014 (3), and two by PricewaterhouseCoopers and Accenture in 2013 (9)(2). Here are the main determinants that were identified:
Bad Requirements
- Requirements for the DMI project did not live in a single, central repository, but were instead fragmented across a multitude of services, including Confluence and JIRA projects.
- Even worse, many of the specs lived within transient Excel or Word documents.
- Less than a third of the identified defects were directly linked to business requirements. This problem made it hard to know which issues had been fixed, within which area, and how the overall solution would be affected.
- Also, 80% of end-to-end tests for new functionality did not link directly to a business requirement.
Missing Design Oversight
- In the final DMI architecture, multiple vendor products exist that perform similar purposes (e.g., Tomcat and WebSphere).
- The inconsistency is a result of misalignment of technology guidelines among teams, resulting in some groups using open source software while others using proprietary software for similar purposes.
- The affected products were already live in production at the time, going through a full development cycle without any member identifying and addressing design issues.
- The Technical Quality Group could have implemented stronger design governance principles and been responsible for the mitigation of technical risks.
Bad Testing
- Some components had not been included in a software release process to be formally tested. Therefore, they were not officially progressed through the project lifecycle to production systems.
- Most components had no set of comprehensive, repeatable tests, including data and media file validation, stress, or stability.
- Many components were not integration tested.
Bad Outsourcing Management
- The BBC did not have an up-to-date assessment of its contractor’s capacity and capability to deliver the program. The project was initially outsourced to Siemens based on a strategic partnership agreement signed in 2004.
- When taking technology development in-house, the BBC did not test whether that was the best option.
- No independent technical assurance reports on system design were commissioned.
- No minimum technical and management requirements for effective oversight of contracts were established.
- No procedure was set in place to intervene and secure the delivery of outsourced contracts rather than waiting for either contract non-delivery or termination.
Faulty Governance
- By having sponsors and business stakeholders on the DMI Steering Group, there was insufficient separation from the day-to-day management to enable them to provide adequate oversight and challenge of the project.
- The financial benefits of the program were initially overstated and unchallenged.
- DMI did not provide clear and transparent reporting on progress against plan, the cost to complete, or delivery of benefits to enable effective decision-making within the governance structure.
- There was a considerable change in the personnel fulfilling several critical roles in the governance structure over the life of the program:
The DMI is an excellent reminder of how, in most cases, failure has nothing to do with the software, but everything to do with management and people. In written testimony to the House of Commons, Bill Garrett (former Head of Technology, BBC Vision Productions) notes:
Too many staff members and contractors jobs depended on DMI continuing; many of them recognized the project had little chance of success; however, speaking up would impact their careers and livelihood. Many senior figures had reputations invested in DMI. (…) In some cases, I believe certain individuals acted willfully to subvert governance processes and falsify value propositions to deceive the various governance panels. (10)
The Digital Media Initiative was not the only BBC project to be scrapped after years of expensive funding, tough. The Socrates program saw £8.3million spent on a computer system to monitor foreign news, which has never worked. But that’s a story for another time!
If you enjoyed this article, press the “Follow” button below for more content that will help you become a better technology professional. You can also subscribe to our YouTube channel, where you’ll find our best articles made into nice videos you can enjoy.
Telepat North is an invitation-only club for outstanding coders, offering members free books, courses, learning resources and swag! If you’re interested, you can learn more by visiting north.telepat.io.
(1) The Chaos Report 2015 (The Standish Group International, Inc, 2015 ) — https://www.standishgroup.com/sample_research_files/CHAOSReport2015-Final.pdf
(2) BBC DMI Technical Review (Accenture, 2013) — http://downloads.bbc.co.uk/aboutthebbc/insidethebbc/howwework/reports/pdf/bbcreport_dmi_technicalreview_hypotheses_and_findings.pdf
(3) Memorandum on Digital Media Initiative (National Audit Office, 2014) — http://downloads.bbc.co.uk/bbctrust/assets/files/pdf/review_report_research/vfm/dmi/dmi_nao.pdf
(4) BBC was ‘complacent’ over failed £100m IT project (BBC, 2014) — https://www.bbc.com/news/entertainment-arts-26963723
(5) The BBC’s management of its Digital Media Initiative (National Audit Office, 2011) — https://www.nao.org.uk/report/the-bbcs-management-of-its-digital-media-initiative/
(6) Fifty-Second Report, BBC Digital Media Initiative (Public Accounts Committee, 2014) — https://publications.parliament.uk/pa/cm201314/cmselect/cmpubacc/985/98502.htm
(7) BBC’s troubled £133m digital video archive delays ‘tapeless’ future (The Guardian, 2013) — https://www.theguardian.com/media/2013/may/03/bbc-digital-video-archive
(8) BBC outsources design of infrastructure for DMI (Boardcast, 2011) — https://www.broadcastnow.co.uk/bbc-outsources-design-of-infrastructure-for-dmi/5028576.article
(9) BBC Digital Media Initiative Review of the BBC’s management of DMI (PwC, 2013) — http://downloads.bbc.co.uk/bbctrust/assets/files/pdf/review_report_research/vfm/dmi/pwc_dmi.pdf
(10) House of Commons Public Accounts Committee, BBC Digital Media Initiative, Written Evidence (Bill Garrett, John Linwood, 2014) — https://publications.parliament.uk/pa/cm201314/cmselect/cmpubacc/writev/bbc.pdf
