The Journey of Metadata at PayPal
With 2020 ending and many of us looking forward to 2021, it’s a good time to reflect on the last three-and-a-half years of PayPal’s journey with metadata.
Metadata is not new to PayPal. Attempts to create a comprehensive metadata system predates PayPal’s Unified Data Catalog (UDC), which today is the Enterprise Data Catalog at PayPal.
As an introduction, this picture shows where we are today with UDC.
Today, UDC is a central part of everything that deals with data at PayPal. Here’s how we conceived, incubated, and evolved the enterprise data catalog from scratch:
Through the multi-year journey, we’ve made a giant leap forward in terms of making UDC the official enterprise data catalog for PayPal. We’ve assimilated and deferred responsibilities of duplicate products and features that have existed traditionally. We’ve made decisions that may not reflect on the roadmap or the features available on product. Hence, it’s meaningful to walk through how we navigated the ecosystem and organization dynamics to where we are today.
Last but not least, our story isn’t complete without details of how the team started and evolved over the last three years.
We’ll be following up on metadata at PayPal with more posts in the following areas:
- UDC architecture and how it fits in PayPal’s larger data landscape
- Technical deep-dive into the mechanics of data catalog and metadata crawling at scale
- Lessons learned and fundamental problems that we’re striving to solve
Author :
Deepak Chandramouli | Engineering Lead, Unified Data Catalog @ PayPal
Special Thanks
Enterprise Data Governance partners : who are driving the strategy & vision to build a comprehensive metadata solution for the company : Subra Munaganuru, Thomas Castriota, Funmi Balogun
Enterprise Data Platforms Leadership : Sudhir Rao, Sudhir Muthuraj, Prasanna Krishna
UDC Team :
- Dheeraj Rampally | Staff Software Engineer, PayPal [ Lead | UDC foundations stack ]
- Harsh Bhimani | Staff Software Engineer, PayPal [ Lead | UDC Metadata Discovery stack ]
- Shraddha Yeole | Software Engineer, PayPal [ Unified Data Catalog Engineering ]
- Aishwarya Mohan | Software Engineer, PayPal [ Unified Data Catalog Engineering ]
- Romit Mehta | Lead Product Manager, PayPal [ Product manager | UDC |2018,2019 & early 2020 ]
- Daniel Silva | Product Manager, PayPal [ Product manager | UDC | 2020 & beyond ]
- Vladimir Bacvanski | Principal Architect, Strategic Architecture, PayPal
