How a Common Data Model can improve productivity and ensure scalability
By Ping Liu, Principal Engineer in Digital Loan Originations at Macquarie Bank
Overview
In today’s data-driven landscape, large organisations face challenges in managing and utilising data efficiently and effectively. Fragmented or inconsistent data formats can lead to data quality issues. A Common Data Model (CDM) is essential to centralise and standardise data, ensuring data consistency, accuracy and quality which would not only improve operational efficiency by reducing redundant data efforts, but also empower teams with reliable data for better decision-making and analytics.
The benefits of CDM
At Macquarie Bank, we want to create the best loan originations experience for our customers. A CDM standardising data across our home and car loan domains allows for a consistent representation of customer information, credit history, loan details and financial data. This uniformity facilitates seamless data exchange and integration between various systems and applications in both domains. It can play a crucial role in the end-to-end origination transformation experience and the business processes for home and car loans.
Advantages of embracing a CDM in software engineering
Data consistency and quality
Using CDM, data is stored and represented consistently across different parts of one or more software systems. This helps to enhance data quality and remove duplication, inaccuracies or inconsistencies. With CDM, errors and inconsistencies are detected and resolved early in the development process, reducing costly data-related issues in production.
CDM can incorporate data governance and security principles, helping to enforce data access controls. It can also help to simplify the process of data analysis and reporting because the data is organised and labelled consistently. This makes it easier to extract insights, generate reports and perform analytics on the data collected by the software.
Faster development and improved collaboration
Developers can leverage pre-defined data structures and schemas from the CDM, reducing the need to create data models from scratch. This accelerates software development by allowing teams to focus on building application logic rather than spending excessive time on data modelling.
This common understanding of data structures fosters better collaboration between software engineers, and alignment on data definitions enables smoother data sharing across different parts of the organisation. It also reduces the need for complex data transformations when integrating different systems, ultimately saving time and effort in software development.
Scalability
A well-defined CDM simplifies the process when a change of data structure is required. Engineers can make updates to the model and propagate these changes throughout the system more efficiently, reducing the risk of introducing bugs or data inconsistencies.
A well-designed CDM can future-proof a system to some extent by accommodating new data types, attributes or relationships as business requirements change or evolve without disrupting existing systems.
The necessary considerations when adopting CDM
While embracing CDM offers substantial advantages, many organisations may struggle or face challenges in implementing or leveraging it effectively.
Below are important factors to ensure a CDM is implemented effectively.
Focus on outcomes to enable change
Transitioning to a CDM often requires altering existing workflows and practices, and can require a significant change in data management practices. The stakeholders and teams might resist these changes if they are accustomed to their existing data structures and processes.
In larger organisations, different departments or teams may have their own data management practices and resist adopting a common model. This siloed approach can lead to fragmentation and inefficiency.
It can be challenging to integrate CDM with complex legacy systems. Legacy systems often have data structures that do not align well with the CDM, requiring extensive data migration or transformation efforts which can be costly and prone to errors.
Robust implementation
If organisations create overly complex CDMs that are difficult to understand and maintain, it can lead to confusion among developers and data users, making it challenging to derive value from the CDM.
Some companies may be tempted to overly customise the CDM to match their specific needs. While some customisation is necessary, excessive customisation can defeat the purpose of having a standardised data model and make it challenging to maintain and update the CDM effectively.
Continuous monitoring and iteration
CDM is not a one-time project. It requires continuous monitoring, maintenance and iteration. It requires clear ownership and ongoing management and maintenance, or it will become outdated and ineffective over time. Successful CDM adoption requires a holistic approach and commitment.
How we implemented CDM
Knowledge and skill
At Macquarie Bank, we hold a deep understanding of our data and have well defined use cases. Our team has extensive experience in the lending sector, having worked with data from a wide array of lenders. We have the skill to transform data from one format to various others. As a result, we have a firm grip on our domain and excel in the efficient structuring of our data.
Current data assessment and mapping
Our previous data model, influenced by an outdated industry standard, primarily because of data provision by our service provider, incorporates certain customisations. Over time, this industry standard has undergone substantial changes, many of which are non-backward compatible, to accommodate evolving industry demands. While some of our customisations aimed to simplify processes and align with our needs, there was room to review options.
By conducting a comprehensive analysis of our existing data model and engaging with stakeholders and data consumers to understand their present and future requirements and use cases, we were able to pinpoint gaps, data silos, inconsistencies and other challenges. These insights allowed us to effectively address these issues within our CDM.
The CDM is designed to fully cover existing data sources by performing compressive data mapping exercises and iterations. The goal was to centralise and consolidate data from diverse sources into a unified model while minimising disruptions to the existing system.
Industry standard alignment
We crafted the CDM by adhering to a set of standardised entities, attributes and relationships, incorporating bespoke elements and structures. Our guiding principles include industry standards and best practices where applicable. Our CDM represents a subset of the common data standards with minimal customisations. The common data standards are helpful in resolving questions or uncertainties that arise during data modelling.
Readability and usability
We employ a logical and consistent approach to arrange data into standardised entities or categories to ensure readability. This means the data is easy to understand and navigate. We achieve this by adopting clear and intuitive naming conventions, maintaining a consistent structure with well-defined semantics and associations. Our aim is to streamline data retrieval, querying and reporting for enhanced usability and clarity.
Championed by stakeholders
When stakeholders understand and appreciate the benefit of a well-designed CDM and are committed to the adoption, it ensures that the CDM receives the essential attention and care required for successful implementation including providing resources and resolving any resistance or pushbacks. This commitment also guides coordinated efforts, fostering alignment throughout the organisation by increased awareness and engagement. It helps promote a cultural shift towards embracing CDM adoption.
Iteration and scope
We began by defining the initial CDM with a narrower scope. For example, our initial scope was to focus on loan application pre-submission customer data capture. The plan is to progressively broaden its reach over time, incorporating additional data domains and use cases. This phased approach serves multiple key purposes: it establishes a well-defined starting point, builds a sturdy foundation, promotes ongoing learning and adaptation, and helps ensure that our growth remains closely aligned with our actual business needs and capabilities.
Dealing with system with legacy components
Conclusion
Macquarie Bank initiated this process of CDM several months ago. Following the successful adoption of CDM, our conversations and collaborations have shifted towards a stronger emphasis on functional requirements. We now dedicate considerably less time to data model design and data / service contract negotiation. This shift not only enhances productivity by delivering high-quality software solutions but, more importantly, it significantly enhances the overall engineering experience.
The adoption and use of a CDM helps bring consistency, efficiency and adaptability to the management and utilisation of data within software systems. It enables data-driven decision making, automation and continuous improvement, ultimately leading to more robust and scalable software solutions.
