Scaling Ancestry.com

Over the past 20+ years, I have been really fortunate to be part of scaling multiple zero-to-one, Direct-To-Consumer Internet businesses; as an engineer on Travelocity.com in the early days of the Internet, to an architect on YellowPages.com, to leading all of engineering & infrastructure for Archives.com and now at my current role @ Ancestry.com and AncestryDNA.com. Each of these roles gave me the opportunity to help scale a technology-powered business, innovate and work with amazing engineering teams to solve important problems for our customers.

I joined Ancestry as part of the company’s acquisition of Archives.com in 2012, at the time when Ancestry was just starting the DNA business. On my first day here, I met our Chief Scientific Officer, who later commented, “Paddy, every time we talk about a new initiative, you always add “at scale” in the end!”. She made me realize that the idea of scalability is really important to me. Indeed, this was true, since we were expecting hyper-growth in the DNA Kit sales and it was most critical for our systems to be able to keep up with the growing demands.

Ancestry is a 30+ year old Family History business that has a track record of innovation and growth. Today we serve millions of customers globally. Here are few numbers that tell the story of our scale, as our customers interact with our web properties:

• 30+ billion online historical records.
• 100+ million daily onsite searches by our customers on their discovery journeys.
• 100+ million digital family trees with several billion photos, videos and other media assets uploaded by our customers.
• 20+ million customer DNA samples, making it the largest genomics database in the world.

All of this massive content and data at Ancestry means that keeping up with the growing demands of our business and continuing to innovate and gracefully scale our product and technology organizations and the amazingly complex software and systems that help run the business is no small task! As our business, customer base and organization grew multifold, here are few of the areas of growth that have been at the root of our technology scalability challenge:

• Online Traffic growth: Over the past 10 years, we have more than doubled the daily visitors, serving millions, across all our web properties and our mobile applications.
• Data growth: We have continued to exponentially grow our core datasets, such as digitized historical records, user generated family trees, user uploaded media assets, and genomics/DNA datasets. Additionally, user activity on the site has generated humongous amounts of valuable click-stream data. All of this together, is over 20 petabytes of data and media assets that continue to grow, both in size and in their interrelationships. As each of these datasets have complex internal and cross-functional interrelationships, they further challenge us in use of the right database technologies, ability to discover meaningful insights for our customers and relevant data analytics for our product development.
• Innovations: Over the years, we have relentlessly improved the usability of our website as well as delighted our customers through several groundbreaking innovations, leveraging our unique data assets and our years of experience in the field of genealogy and bioinformatics. This includes:

1. Digital and interactive Family Trees, Historical Records Search & Recommendations, and collaboration tools for family historians.
2. Bioinformatics and Genomic Science based insights such as ethnicity predictions and physical traits predictions and genetic cousin matching.
3. Several historical data and media based insights for our customers, powered by AI and Machine Learning algorithms.

Continuing to keep up with the online traffic and the exponential data growth, growing product & technology teams and simultaneously creating many such innovations has provided us with some really hard challenges.

Our approach to scaling Ancestry’s product and technology organizations and systems that have helped us grow our business and enable millions of customers to seamlessly use our products is multi-pronged, as follows:

• A Learning Culture: At Ancestry, our mission is to empower our customers on their personal journeys of discovery, that enrich their lives as they craft family stories, preserve their research and share those stories with their families and friends. Our technology plays a huge role in enabling all of this. At Ancestry, the product and technology teams are obsessed with customer success, are highly passionate about experimenting and learning and love solving big technical challenges. We provide learning and growth opportunities for all, starting from our interns, to new hires, to seasoned staff engineers and principals, to software architects, to platform engineers, to data scientists, on to leadership at various levels. These include, internal speaker series in mobile development, data science, bioinformatics and technology platforms as well as company-wide hackathon(s) and a yearly technology conference. There is always something for everyone to learn!
• Processes and Organization: As Ancestry’s business experienced growth, so have our teams. We have more than doubled our product and technology teams in the last 10 years and we continue to grow today and bring in new talent globally. We have continually transformed our organization and adopted optimal processes to help us scale for this growth, some examples are: management of company and team goals through quarterly OKRs, matrix product and technology organization with domain, platform and infrastructure teams, optimal product development team sizes, SAFe agile software development practices, QA automation, standardized Continuous Integration/Continuous Deployment (CICD) and proactive site operations to maximize site availability and performance.
• Architectures and Technologies: Over the past several years, architectural paradigms have continued to evolve, technologies have changed multifold, and our customers are adapting to new ways of interacting with our product, from primarily using desktops in the past to many using mobile devices now. Throughout, we have followed some time-tested architectural principles for evolving & standardizing our technology stacks with architectural patterns such as multi-tiered, highly distributed, layered, micro-services based, loosely coupled, and asynchronous architectures, while efficiently leveraging cloud computing.

Currently, we have 400+ technology stacks that power our site, backend AI/ML algorithms and various supporting backend systems. These have gone through several transformations and have evolved from mostly using proprietary technologies to now mostly using open source technologies.

Personally, I am still continuing to learn through these challenges and still ending most everything with “at scale!”, as we are continuing to grow! We’re still early in our journey to scale the technologies that power our growing business, such that we can continue to serve our customers and deliver experiences that delight them and keep them coming back.

What’s Next?

This is the first article in a series of posts about scaling software product development at Ancestry. In the upcoming weeks, we will unpack the details and offer actionable steps you can take in various technology areas to more effectively scale your business.

Scaling Ancestry.com Series articles:

1. Scaling Ancestry.com: Billions and billions of images at Ancestry.
2. Scaling Ancestry.com: Providing billions of personalized hints.
3. Scaling Ancestry.com: Providing Ethnicity Inheritance insights to 22M+ customers

** Join us on our scaling journey, check out the open engineering roles! ***