Neon: Revolutionizing Transactional Databases with its Multi-Tenant Storage Engine

Published in

VAFION

5 min readSep 5, 2023

In the rapidly evolving landscape of cloud-native databases, the quest for achieving elastically scalable and cost-efficient solutions for both analytical and transactional workloads has been relentless. Neon, an innovative open-source alternative to AWS Aurora PostgreSQL, has emerged as a game-changer by introducing a groundbreaking approach to separating storage from compute nodes. This approach allows for seamless scalability and flexibility in transactional databases — a feat previously considered challenging. In this blog, we’ll dive into the revolutionary features of Neon and how it is transforming the database ecosystem.

Architecture overview

A Neon installation consists of compute nodes and the Neon storage engine. Compute nodes are stateless PostgreSQL nodes backed by the Neon storage engine.

The Neon storage engine consists of two major components:

Pageserver. Scalable storage backend for the compute nodes.

Safekeepers. The safekeepers form a redundant WAL service that received WAL from the compute node, and stores it durably until it has been processed by the pageserver and uploaded to cloud storage.

The Challenge of Elastic Scaling in Transactional Databases

Cloud-native analytical databases have long benefited from the practice of decoupling storage from compute nodes. This separation enables these databases to scale up or down based on demand, ensuring optimal performance and resource utilization. However, applying the same principle to transactional databases has been considerably more complex due to the nature of transactional workloads and data integrity concerns. Neon steps up to this challenge with its multi-tenant storage engine for PostgreSQL.

Introducing Neon’s Multi-Tenant Storage Engine

At its core, Neon’s architecture revolves around decoupling storage from compute in a transactional database environment. This is achieved through its innovative multi-tenant storage engine, which seamlessly integrates with PostgreSQL. This means that Neon retains compatibility with the familiar PostgreSQL interface while introducing groundbreaking capabilities.

On-demand scalability

Neon separated storage and compute to make on-demand scalability possible. Compute activates on an incoming connection and scales to zero when not in use.

Compute is fully client-compatible with Postgres because a Neon compute is Postgres.

Neon dynamically adjusts the allocation of compute resources based on workload.

Bottomless storage

Neon designed the storage from the ground up as a fault tolerant scale-out system built for the cloud. It integrates with cloud object stores such as S3 to offload cold data for cost optimization. Storage architecture ensures high availability, scale out, and unlimited capacity that is called “bottomless storage”.

Neon storage uses the “copy-on-write” technique to deliver data branching, online checkpointing, and point-in-time restore. This eliminates expensive full-data backup and restore operations required with traditional database-as-a-service systems.

Its storage technology is open source and written in Rust.

Data branching

Neon allows you to instantly branch your Postgres database to support modern development workflows. You can create branches for test environments and for every deployment in your CI/CD pipeline.

Branches are created using the “copy on write” technique, making them virtually free.

Leveraging AWS S3 for Long-Term Storage

Neon’s storage engine takes advantage of AWS S3 for long-term data storage. By leveraging S3’s scalable and cost-efficient storage, Neon significantly reduces the burden on the compute layer and enhances its overall elasticity. This design allows Neon to cater to workloads with varying storage and compute requirements, ensuring efficient resource utilization without compromising performance.

Elastic Scaling of Processing Power

One of Neon’s standout features is its ability to elastically scale processing power — a challenge that traditional transactional databases often struggle with. Neon achieves this by providing the ability to scale processing up or down, including the ability to scale-to-zero when the workload is light. This flexibility ensures that resources are allocated as needed, delivering cost savings and optimal performance for varying workloads.

Benefits Galore: Cheap and Fast Clones, Copy-on-Write, and Branching

Neon’s innovative architecture brings forth a multitude of benefits:

Cheap and Fast Clones: Neon’s architecture makes creating clones of databases both cost-effective and rapid. This is particularly advantageous for scenarios like testing and development, where the ability to quickly spin up new instances can accelerate the development lifecycle.

Copy-on-Write and Branching: The architecture’s copy-on-write approach ensures that data changes are efficiently managed, contributing to better data integrity and reduced overhead. Additionally, the capability for branching enables the creation of divergent versions of the database, offering new possibilities for data analysis and experimentation.

Embracing the Future with Neon

Neon’s innovative multi-tenant storage engine for PostgreSQL sets a new standard for elastically scalable and cost-efficient transactional databases. By seamlessly integrating with AWS S3 and providing the capability to scale compute resources as needed, Neon empowers organizations to adapt to changing workloads while maintaining data integrity.

As the database landscape continues to evolve, solutions like Neon demonstrate the power of pushing boundaries and rethinking conventional approaches. With its minimal changes to mainstream PostgreSQL code and its ability to harness cloud-native storage and compute paradigms, Neon paves the way for a new era of transactional databases that are not only robust and reliable but also agile and cost-effective.

Conclusion :

Neon’s architecture represents a significant leap forward in the world of cloud-native databases, redefining what’s possible for transactional workloads. Its multi-tenant storage engine, which leverages AWS S3 and enables elastic scaling of processing power, unlocks new opportunities for businesses to optimize their database operations and adapt to the dynamic demands of modern applications. As technology enthusiasts and businesses alike look to the future, solutions like Neon inspire confidence in the ongoing innovation and evolution of database management systems.

For more details contact info@vafion.com