Architecting Multiple Data Lakes

O'Reilly Media
Published in
9 min readJan 28, 2021


Enterprises build data lakes for a variety of reasons. Some start as single-project data puddles created by a line-of-business or project team and grow into data lakes gradually. Some start as ETL offloading projects by IT and pick up additional users and analytic use cases along the way. Others are designed from the get-go as centralized data lakes by IT and analytic teams working together. Yet others are created in the cloud as shadow IT for business teams that don’t want to wait for the official IT team.

Similarly, enterprises don’t always have their data lakes in the same public cloud. Some data lakes contain data that enterprises are not allowed to keep or not comfortable with keeping in public clouds, or regulations require having different data lakes to comply with data privacy and other local laws. Sometimes data lakes are created by different divisions or are acquired by companies in different public clouds. Other data lakes are deliberately created in different public cloud providers to take advantage of unique capabilities, data locality, vendor independence, and a variety of other reasons.

Regardless of the reasons, many enterprises end up with multiple data lakes. The question then becomes, should these be merged into one or kept separate? This piece answers this question first and then explores approaches to managing multiple data lakes by applying virtual constructs such as data federation, data fabrics, and data catalogs. No simple standard answers exist, but this chapter covers rules of thumb and best practices.

Reasons for Keeping Data Lakes Separate

The reasons for separate data lakes are usually historical and organizational. Typical reasons include the following:

Regulatory constraints
In regulated industries and for personal data, regulatory constraints often forbid the merging or commingling of data from different sources or geographies. For example, the EU has strict data privacy guidelines that are implemented differently by every country. Medical institutions usually have strict data sharing guidelines as well.

Organizational barriers
Sometimes organizational barriers to sharing data exist, mostly around budgeting and control. Financing a common data lake and deciding on common technologies and standards between warring business units with greatly differing goals and needs may prove to be an insurmountable challenge.

SLAs and governance levels
Keeping a production-quality data lake, in which data is carefully curated and has strong SLAs and governance, separate from data lakes dedicated to experimental analytics, sandboxes, and development projects can help guarantee good quality, governance, and strong SLAs for the former.

Advantages of Merging Data Lakes

If you are not constrained by the kinds of regulatory or business requirements mentioned in the previous section, you should try to restrict your organization to a single large data lake. There are several reasons for this:

Administrative and operational costs
A lake that grows twice as large doesn’t usually need a team that’s twice as large to manage it. Of course, the same team can manage multiple lakes, but if the lakes are hosted on different public clouds, teams trained on those clouds are usually required to support them. Even if a single platform is used by the enterprise, but there are multiple data lakes because of organizational and control issues, each organization tends to staff its own team so it can control its own destiny. This duplication raises costs.

Data redundancy reduction
Since the data lakes belong to the same enterprise, chances are that data lakes contain quite a bit of redundant data. By merging lakes, you can eliminate this and reduce the quantity of data stored. Data redundancy also implies ingestion redundancy: the same data is extracted and ingested multiple times from the same source. By consolidating, you can lessen the loads on the sources and the network. If all the data lakes are on the same platform, the best practice is to create shared data buckets (e.g., S3 buckets) or tables that can be updated once and shared by multiple data lakes. However, if data lakes are in different public clouds, that would not be possible.

Combining the lakes into one, or at least sticking with a single public cloud vendor, will make it easier for the enterprise to reuse the work done by one project for other projects. This includes scripts, code, models, data sets, analytic results, and anything else that can be produced in a data lake. It will also be easier to train the users on a common set of tools.

Enterprise projects
Some groups work on an enterprise scale and may require data from different organizations. These projects will greatly benefit from having a single centralized data lake instead of having to merge data from multiple lakes.

There are also other business advantages to consider, including faster time to insight. Next we’ll turn our attention to using the same cloud platform to build multiple data lakes.

Building Multiple Data Lakes on the Same Cloud Platform

One compromise that addresses some of the reasons for keeping data lakes separate, yet achieves some of the advantages of merged data lakes, is to build physically separate data lakes on the same cloud platform, such as AWS, Azure, or GCP. This allows for reuse of tools, support for enterprise projects through easy access to the data in different data lakes, and lower administration and training costs, since only one technology stack needs to be administered, and users can be trained on a single set of tools.

If you decide to keep your data lakes separate, the next section will help you address some of the challenges of multiple data lakes by turning them into a single virtual data lake.

Virtual Data Lakes

One approach that has been gaining ground is creating a virtual data lake to mask the existence of multiple physical data lakes. Instead of living with multiple data lakes or merging them into one centralized data lake, why not present them to the user as a single data lake, while managing the architectural details separately? Several approaches can accomplish this data virtualization: data federation masks the physical location of the data by presenting it as a set of virtual tables and files, data fabrics present files from different physical systems in a single filesystem, while enterprise catalogs do not hide the physical location of the data, but help users find and provision data from multiple systems.

Data Federation

Data federation refers to combining, or federating, data from different databases into a single virtual database. It has been around for at least 20 years. IBM’s DataJoiner, introduced in the early 1990s, created a “virtual” database whose tables were really views on physical tables in multiple databases. The users of DataJoiner would issue SQL queries against these virtual tables, and DataJoiner would translate them into queries that it would apply against the different databases; it would then combine the results and present them back to the user.

DataJoiner eventually evolved into IBM InfoSphere Federation Server and was matched by products from Denodo, TIBCO (Composite), Informatica, and others. More modern versions of these products support RESTful APIs and can interact with applications and filesystems as well as databases. Nevertheless, at their core, these products are all designed to create a virtual database that, under the covers, can pull data from different systems and make it look like a single table.

Applying this technology to data lakes carries several significant challenges. The biggest challenge is that you have to manually configure each virtual table and map it to the physical data sets, whether they’re files or tables. In a data lake with millions of files loaded through frictionless ingestion, that’s simply not practical. Then there’s the traditional distributed join problem: combining or joining large data sets from different physical systems requires sophisticated query optimization and lots of memory and computing power. Finally, you have the schema maintenance problem: when the schema changes, the virtual tables have to be updated as well. Since a schema is applied only when data is read (schema on read), the users may not know that the schema has changed until their queries fail. And even then, it may not be clear whether the problem was caused by a schema change, a data problem, human error, or any combination of these.

Data Fabric

Just as data lakes arose as a way to cope with the massive growth in volume and variety of data, big data virtualization applies the big data principles of schema on read, modularization, and future-proofing to create a new approach to data virtualization that can cope with the massive volume and variety of data in the enterprise. At the heart of the new approach is a virtual filesystem that represents physical data sources as virtual folders, and physical data sets as virtual data sets. This approach mirrors how staging zones are organized in data lakes. This virtualization allows the data to stay in its original data source, while being exposed to other business users.

This virtual filesystem, called a data fabric, is provided by vendors such as HPE (MapR), Cloudera (ViewFs), and Hitachi Vantara. It presents data from different public clouds and on-premises systems as a single filesystem with a single API. Any program running on any system can then access any data set in the filesystem, and the data fabric delivers it to that program for processing.

Since the programs are not aware of the location of the data, this approach may cause large performance challenges if, say, two massive files from different data lakes across the world and platforms are read by a program.

Catalogs and Data Oceans

A different approach that does not require a single data fabric with complete location transparency is to build a data catalog that provides more visibility into data location while still creating a single place for users to find and provision the data. Because a virtual data lake can be massive and potentially contain millions of data sets, a search mechanism is needed to find and navigate them. This role is usually played by a data catalog that presents all the data in the enterprise, including the data lakes.

With this approach, only the metadata (information describing the data) is in the catalog, so users can find the data sets they need quickly. Once a data set is found, it can be provisioned to give the user access either by copying it to the user’s project area or by giving the user permissions to access it in place.

Because both tables have been copied to the same physical system, the joins are local, much easier to implement, and faster to execute. Provisioning can involve an approval process whereby the user requests access for a period of time and specifies a business justification. The request is reviewed and approved by the data owners, and the data is then copied over. Finally, the copy of the data is kept up-to-date by ETL tools, customer scripts, or open source tools like Sqoop that connect to a relational database, execute a user-specified SQL query, and create an HDFS file with the results of the query.

Because the catalog is the main interface for finding and provisioning data in the data lake, it enables an elegant solution to building a virtual data lake. When users look for a data set, it really does not matter to them where it is physically located — it looks the same and can be found in exactly the same way. An intelligent provisioning system can be provided so if the user wants to use a data set in a tool, it may be provisioned in place (i.e., opened directly in the tool), while if it needs to be joined with other data sets or modified, it can be transparently copied over to the physical data lake and made accessible there.


If you are considering multiple data lakes, this piece discussed reasons to keep them separate or to merge them. If strong organizational, regulatory, or security reasons indicate keeping data separate, consider keeping all the data lakes on a single cloud platform to take advantage of learning and administering a single technology stack, easy reuse of data and tools, and easy data access for enterprise-wide projects. If you do end up with a heterogeneous architecture, whether because it is a hybrid and some data needs to remain on premises or because there are compelling reasons to use different platforms, consider different approaches to creating a single virtual data lake through a set of shared virtual tables, by building a virtual data fabric, or by using catalogs to create data oceans. Whether you have a single or multiple data lakes, always build a data catalog, so users can find and provision the data that they need. Consider other virtualization techniques if you have applications that have to use data from disparate systems or if you need to move data around for cost or compliance reasons. Both data federation and a data fabric provide homogeneous data access to the disparate data sources, freeing the applications from having to worry about heterogeneous data access, as well as location transparency that allows administrators to move data to different systems and storage tiers.

Learn faster. Dig deeper. See farther.

Join the O’Reilly online learning platform. Get a free trial today and find answers on the fly, or master something new and useful.

Learn more

Alex Gorelik has spent his career helping enterprises use data. He is a Hitachi Fellow and IBM Distinguished Engineer, as well as founder and CTO of three companies in data space: Acta — an ETL company acquired by Business Objects and now marketed as SAP Data Services, Exeros — a data discovery company acquired by IBM and marketed as Infosphere Discovery, and Waterline Data — a pioneering Big Data catalog company acquired by Hitachi Vantara and marketed as Lumada Data Catalog.



O'Reilly Media

O'Reilly Media spreads the knowledge of innovators through its books, video training, webcasts, events, and research.