How many houses are we actually building?

An insight into how we are currently monitoring housing delivery and how modern approaches to better collection, management and use of data can be crucial in the wider context of planning reform.

Introduction

Being a planning geek, data geek, digital geek and therefore a complex systems geek, it never fails to astonish me the number of moving parts within the ecosystem of planning. Bearing in mind just how important and sensitive the metaverse around the question ‘How many houses are we actually building?’ is, I am consistently astonished (why Paul, you may say!) why we have been so bad at understanding this for quite so long.

I have worked in, and with a number of councils in and around the ecosystem that surrounds this question over the years and it feels an apt time (with a new renewed focus on accelerating house building) to share some thoughts.

A quick recap: Monitoring housing development in the UK. What we currently do…

In the UK, monitoring the number of houses built each year is a critical task for local councils. This process ensures compliance with national housing targets, informs local planning policies, and aids in forecasting and allocating resources for infrastructure and public services. Housing monitoring in councils is not only a key indicator for local plan policy success, it is also the basis for which other policies (such as school allocation, transport or health) are determined.

This data is paramount to understanding accurate council revenue forecast, from Council Tax to benefit allocation and New Homes Bonus. It is also a key factor in relation to five year housing land supply challenges. a comprehensive look at how councils monitor housing development annually.

Pretty crucial hey?

Current approach to data collection:

Obviously this is not comprehensive but generically this is the gist….

Planning Permissions: The initial step in housing monitoring involves tracking planning permissions granted. Councils (mostly) maintain detailed records of all approved planning applications, which include information on the number and type of dwellings to be built. This data provides an early indication of potential housing supply.

Commencements and Completions: Councils monitor the progress of construction through site visits (yes, in person!) and notifications from developers. Building Control departments play a pivotal role by conducting inspections at various stages of construction to ensure compliance with building regulations. They record when construction commences and when it is completed.

On-Site Surveys: Regular on-site surveys are conducted to verify the progress and completion of housing projects. Surveyors visit construction sites to collect data on the number of units started, under construction, and completed. These surveys help validate the information provided by developers and ensure accuracy.

Council Tax Records: This is a bit chicken and egg as we need to monitor progress and this doesn’t happen until houses are built, but new dwellings must be registered for Council Tax once they are habitable when Revs and Bens (Revenue and Benefits) team record when a new property becomes liable for tax, indicating that it is complete and occupied.

Integration and management

Spoiler alert…. This is not done usually done well due to capacity and capabilities in councils!

Data Repositories: Councils maintain various databases to store and manage housing data. These include planning databases like Uniform, GIS systems for spatial data, and specific databases for tracking housing commitments and completions. The integration of these data sources is crucial for comprehensive housing monitoring but currently VERY MANUAL at best….

Annual/Authority Monitoring Report (AMR): The AMR is a key document (but one that almost no one has even heard of) produced by councils to report on housing delivery. Among many other important things like success of policies etc, it compiles data from multiple sources to provide an overview of housing completions and progress against local plan targets. The AMR includes detailed tables and analyses of housing trends, completions by type and tenure, and future housing land supply (usually locked up in pdfs or non queryable document formats!).

AMR Blueprint through the planning and delivery cycle

Challenges and inefficiencies

I’ll get into some specifics of this in the deep dive further down

Data Fragmentation: One of the main challenges councils face is the fragmentation of data across different departments and systems. Housing monitoring data is often held in silos, making it difficult to compile a complete and accurate picture of housing delivery.

Manual Data Collection: Many councils rely on manual methods for data collection, such as site surveys and paper-based records (Yes, paper, do you remember that?). This process is time-consuming and prone to errors, leading to inefficiencies and potential inaccuracies in the data.

Duplication of Efforts: Multiple departments may independently collect similar data for different purposes, leading to duplicated efforts. For example, both Planning and Building Control departments may conduct site visits to monitor the same developments, wasting resources.

A deep dive

There is quite a lot of interest from councils but only a small number have done any real work on this area

Where there has been real opportunity to ‘get under the bonnet’ with councils on specifics it has been a super interesting to see how the journey can play out, identifying the ‘as is’ inefficiencies in the current housing data monitoring process, focusing on integrating disparate datasets, improving data accuracy, and reducing the resource-intensive nature of data collection.

We aren’t at a standing start with how we apply an overarching hypothesis to these type of challenges and as all councils have to undertake this process (in broadly similar ways). The combination of multiple, siloed data sets could provide a more comprehensive and efficient understanding of housing monitoring is a pretty solid assumption. I would argue that this is also almost identical to the challenges endured by most if not all planning authorities, and even small margin improvements will bring enormous financial and operational efficiencies to many actors operating in this ecosystem.

Working from the (low) risk assumption that there are several data sets around housing monitoring, held by separate teams and systems and collected in parallel, that combined might have the potential to provide a much more accurate, less resource-intensive, and faster understanding of housing monitoring.

Mapping the housing monitoring data journey through the lens of the Annual Housing Monitoring Report (AMR) provides focus because of its importance and pain points.

• The AMR is the key tool to enable the council to understand their progress against their local plan policies and the impact of their interventions in the market.
• It’s characterised by a reliance on manual data collection and entry, leading to the need for high levels of quality assurance.

Triangulating this by conducting in depth interviews with people across departments holding workshops with stakeholders to distill insights and prioritise opportunities helps validate and confirm some key issues.

Thoughts

Fragmentation and Trust Issues: Current data journeys are fragmented, leading to significant trust issues. The manual nature of data collection introduces inaccuracies and necessitates extensive quality assurance, further eroding confidence in the data. This fragmentation results in a lack of a cohesive understanding of the data across departments.

Inconsistent Data Strategies: Inconsistent approaches to collecting, storing, and accessing data across departments leads to inefficiencies and incompatibilities, making it difficult to integrate and utilise the data effectively.

Duplication of Efforts: Multiple departments independently collect similar data, often through on-site visits, leading to unnecessary duplication of efforts. Only one department, Building Control, is mandated by regulation to conduct such monitoring, highlighting the inefficiency and redundancy in the system.

Pain Points

• Low Confidence in Data: Users do not trust data unless they have seen the development in person, reflecting skepticism about the accuracy of the collected data.
• Lack of Process Understanding: An lack of visibility or understanding among wider teams and officers about the broader data journey and being unaware of what happens before or after their part of the process, leads to inefficiencies and missed opportunities for collaboration.
• No Single Data Source: The absence of a centralised data source (this is a wider issue for local government in general and many are moving towards investing in building centralised data and insight capacity and capability) means that departments spend significant time locating and compiling the data they need, resulting in substantial inefficiencies.
• Duplication of Data Collection: Several departments often conduct independent on-site monitoring of the same developments, collecting similar data for different purposes. This duplication is both time-consuming and unnecessary.

Potential Interventions and Solutions

1. Data Journey Directory: Establishing a directory to signpost where housing monitoring data sits within a council. This directory would increase awareness and reduce duplication by providing visibility into existing data and identifying data owners.
2. Digitising Site Survey Data Collection: Implementing tools for on-site survey data collection and creating shared data standards to reduce manual data entry, increase data confidence, and facilitate data sharing across departments.
3. Developing a Shared Data Strategy: Creating a unified data strategy to harmonise data collection, storage, and access practices across departments. (There are more and more councils now commissioning holistic data strategies and this will be incredibly important for housing monitoring). Coherent and co-designed strategies will foster a data-positive culture and enable more effective data-driven decision-making.
4. Centralised Housing Monitoring Data Store and Dashboard: Developing a central repository and dashboard to integrate data from multiple departments. This solution would allow for better cross-referencing and reduce the duplication of data collection efforts. There is a case that this should be undertaken at scale over a larger footprint as the data is so critical especially geo-spatially overlaid with other critical economic and social data sets.
5. Focus on People and Processes: Obviously the big one but short iterative proofs of concepts can really help show value to catalyse change alongside technological improvements. A holistic approach that includes training and process optimisation is essential. Foster a data-positive culture within the organisation. Encourage the use of shared data standards and practices, and promote the benefits of data-driven decision-making to gain buy-in from all stakeholders.
6. Data Strategy and Governance: Develop a comprehensive data strategy to unify data collection, storage, and access practices across departments. This strategy should include clear data governance policies to ensure consistency and compliance.
7. Technology Integration: Implement technologies to digitise data collection and centralise data storage. This integration will ensure compatibility, improve accessibility, and facilitate more efficient data management.

Opportunities

Heaps of low hanging fruit here IMHO to do some relatively speaking simple work and reap serious multifaceted rewards.

Some of these challenges are actually pretty straightforward to address without major financial outlay. As with many of the process issues in councils the common theme is around legacy process and culture. There are also ways that standardised reporting requirements could be used to catalyse some of the change required in councils to improve our collective view of house building and monitoring and reap the benefits of open and accurate data for forward planning.

This will be especially crucial with the new housing targets and 5 yr housing land supply requirements. There is also a role for combined authorities in their new strategic planning capacity as potential regional data hubs to give insight in how and where wider economic geographies should focus there housing delivery and forward plan for supply chains more effectively. This could also provide longer term security for inward private investment built on accurate insight, and given government ambitions around growth this will be of utmost importance against the backdrop of the current financial landscape.