5 Database technologies used by 2000 Wix microservices
Unlike with UI frameworks, or software industry buzzwords, data storage technologies, i.e. databases are famously slow to change, and new innovation takes longer to be adopted.
For a large organization, switching to a different OLTP database as your main workhorse happens very rarely. In recent years, with the advent of microservices, more flexibility is given for each service to choose the way it persists it’s data. Still, the risk of using unproven technology sometimes outweighs the benefits, as losing crucial business related data is unacceptable. While the technology may be proven elsewhere, it may not be proven and battle-tested in your organization yet.
At Wix, the situation is not different, as MySQL is still the main go-to database, more than 10 years after it was first used. Albeit, new and modern MySQL APIs (e.g. JSON Documents) are being used more and more.
But with 2000 microservices in production, there are of course several storage technologies other than MySQL that are used by a significant portion of services due to various reasons. The reasons range from different API use-cases (e.g. support CRUD API, allow free-text search) to different tech stacks (JVM vs. Node.js) to more demanding technical requirements (infinitely scaling storage).
5 technologies are described below, each with its own use cases at Wix, unique characteristics, strengths and weaknesses. The technologies are MySQL, MongoDB, DynamoDB, Elasticsearch and S3.
Read on to get some insights that may help you choose your next microservice storage technology.
1. MySQL
Other DBs in this category
PostgreSQL, MariaDB, SQL Server, Oracle, Db2, SQLite
When to use MySQL
- When you already widely use it in your organization
- When you want both relational tables (when you know the schema upfront) and JSON collections (Schemaless)
- Relational / normalized — when you need to optimize on writes instead of reads, to have strong read consistency
MySQL Advantages
- Maturity & Reliability — MySQL is highly used, battle tested and mature
- Fast read performance
- Improved JSON/Document support (MySQL 8)
- Cross-DC write consistency (when ProxySQL is used)
MySQL Disadvantages
- Scalability — Does not scale horizontally. Limited by amount of disk space
- Consistency and Replication Issues (when not using ProxySQL)
Example MySQL use cases at Wix
A. Pre MySQL8 JSON support — relational tables with BLOB
Almost all JVM (Scala/Java) services at Wix used to store their data in MySQL relational tables. Data was usually denormalized across multiple tables (SQL joins were rarely used). For added flexibility most of the content was kept in BLOB type in order to avoid frequent schema changes that add a lot of risk for production stability.
B. JSON support and Simple Data Layer
Simple Data Layer (SDL) is an internal infrastructure built at Wix in order to help speed-up the development time and reduce cognitive load when persisting a document data structure.
This is a very common pattern as persistence exists in CRUD methods as well as business operations. The value spans from having sensible defaults like page size through support for WixQL which is part of our API and up to wider cross cutting concerns like GDPR support, emitting domain-changed events, and more.
It supports the very common need of microservices for a non relational embedded document data structure (to have more operations done by DB instead of in-memory).
Its engine is MySQL8 with JSON datatype. MySQL was chosen mainly because of Wix DBAs long time experience with MySQL, a lot of infrastructure already in place for backups, migrations, indexing, and for the strong consistency and availability support provided by ProxySQL (see next paragraph)
C. Multi DC & replication support with ProxySQL
ProxySQL is used in order to seamlessly route write traffic to the currently active main data center’s MySQL clusters without any data loss. It also manages read distribution among replicas. It provides high availability and high performance out of the box.
2. MongoDB
Other DBs in this category
CouchDB (many relational/NoSQL DBs now offer JSON support)
When to use MongoDB
- When data schema is predicted to keep changing and evolving
- When working with dynamic JSON content
- When keeping data denormalized is not a problem. i.e. to have eventual consistency
MongoDB Advantages
- Flexibility — with Schemaless documents, the number of fields, content and size of the document can differ from one document to another in the same collection.
- Easy to scale with sharding
MongoDB Disadvantages
- High Memory Usage — a lot of denormalized data is kept in memory
- Document size limit — 16MB
- Non optimal replication solution (data cannot be re-replicated after recovery from failure). Consistency issues on traffic switch to another data center (No automatic remaster)
Example MongoDB use cases at Wix
A. Wix Data
Wix Data is a service that allows anyone with a Wix website to own a document database — “Database as a Service”). It uses MongoDB behind the scenes to store the data.
Mongo was chosen in order to be able to store any schemaless (JSON) doc. When the team compared MongoDB performance to MySql 5.7.6, the former had 2X write and read performance over the latter (albeit, this older version of MySql didn’t support JSON natively yet)
Recently the Wix Data team published a post on how to host 1 million sites per 1 MongoDB cluster
B. Wix Blog
Wix Blog is Wix’s blog publishing site-app. The backend Node.js microservices use MongoDB to store several TBs worth of blog posts.
When the project was young the schema was still evolving, and MongoDB’s schema-less nature helped ease the pain of adding new fields to a fast growing database. Data is mostly kept denormalized as MongoDB joins ($lookup) are not convenient to use.
3. Amazon DynamoDB
Other DBs in this category
Cassandra, HBase, Redis (Key-value Store)
When to use DynamoDB
- When you need a simple key value store without complex querying patterns
- When you need to store expirable data
- Low-medium throughput apps as writes are expensive and consistent reads are twice the cost of eventually consistent reads.
DynamoDB Advantages
- Fast performance in any scale (as long as enough capacity is provisioned)
- No storage limit
- Schemaless — it’s possible to define a schema for each item, rather than for the whole table
- Multi-master replication (update data in multiple regions)
- Supports TTL per item
- Built-in CDC events (DynamoDB streams)
DynamoDB Disadvantages
- Size limit — item can only reach 400KB in size
- Limited querying options (limited number of indices)
- Throttling on burst throughput (and hot keys in certain situations)
Example DynamoDB use cases at Wix
A. KVStore + Cache
DynamoDB is the back-bone for Wix KV-store cache abstraction. You can find more details in this article (search for (Dynamo)DB+CDC based Cache)
B. Wix Chat
Wix Chat is a service that lets site owners instantly communicate with visitors.
They recently migrated their database from Cassandra to DynamoDB, in order to be able to handle growing scale without needing to increase the DBA & operations teams.
In DynamoDB the Wix Chat team stores chat messages and chat message indications (sent / seen / failed). In addition, data is stored about online site visitors (page, location, entrance time, etc.) which also takes advantage of DynamoDB TTL feature.
C. Wix Contacts
Wix Contacts is a service that manages all contacts of a Wix site owner (Part of Ascend business tools suite)
DynamoDB is used to help orchestrate event-driven bulk jobs (they save the job current status and current offset/page). Each of the job stages is advanced by sending a Kafka event to the appropriate topic. Then the Relevant data is fetched from DynamoDB.
4. Elasticsearch
Other DBs in this category
Apache Solr, Splunk, Amazon CloudSearch
When to use Elasticsearch
- When you need to perform fuzzy search or have results with ranking
- When you have another data store as source of truth (populate Elasticsearch as a materialized view)
Elasticsearch Advantages
- Easy to horizontally scale with index sharding
- Rich search API
- Query for analytical data using aggregations
Elasticsearch Disadvantages
- Indexes are created with a predefined number of shards. More shards requires migration to a new index. Usually done with ReIndex API
- Performance issues when indices hit very large scale (> 1TB with hundreds of nodes and shards)
Example Elasticsearch use cases at Wix
Wix Search
Wix search allows searching inside any Wix site (out of the box) by having a generic API for indexing per site and installed apps.
It also provides custom search services for creating and managing self-owned indices by other Wix teams (basically serves as a proxy to Elasticsearch).
5. Amazon Simple Storage Service (S3)
Other DBs in this category
Google Cloud Storage, Azure Blob Storage
When to use S3
- When you need to store large binary objects/files (up to 5TB each)
- When the amount of data you need to store is large (>10TB), continues to grow daily, and may need to be retrieved (can’t be deleted)
S3 Advantages
- Supports very high throughput
- Infinite scalability — No limit on amount of storage
S3 Disadvantages
- No Query support, only key-based retrieval
- Latency is 100–200 ms for small objects. Caching can alleviate this
Example S3 use cases at Wix
A. Wix Media
Wix Media manages all the media files that are uploaded by Wix site owners. These files are stored, and then accessed when the user’s site gets a visitor.
In order to have highest possible availability and lowest possible retrieval latency, the media files are stored and replicated between both Amazon S3 and Google GCS and cross-region within the same cloud provider.
A dedicated process moves files to cheaper storage classes (e.g. from S3 Standard to S3 Standard-IA) once it finds out they haven’t been accessed for a long time. for more information you can read — Storage Lifecycle: How We Cut Cost on 55% of Our Storage Without Deleting a Single File
B. Wix Editor
The Wix Editor backend team is in charge of (among other responsibilities) storage and fast retrieval of all Wix site pages during site editing.
A site page is actually a JSON file. Every change made to a site page means a new immutable JSON file is inserted to the database.
A couple of years ago they migrated the site pages from MySQL to S3. The main reason was that S3 guaranteed infinite scale (MySQL was unable to handle a scale >10TB well). The S3 latency they measured for read/write ranged from 30 ms on median case to 400ms on p99. In order to improve this they added a write-through caching layer on top of S3.
Summary
Each database technology (and type) has advantages and disadvantages.
While there is a convergence happening in some general-purpose databases (that support various models relational / document / key-value at the same time) like MySQL and PostgreSQL,
Cloud providers like Amazon offer a dazzling array of database and storage options (e.g. Amazon has Aurora, RDS, DynamoDB, DocumentDB, Keyspaces, elastiCache, S3, elastic file system, etc…)
When choosing a database technology it is very important to consider various criteria:
- How complex are your query patterns? Do you just need retrieval by key, or also by various other parameters? Do you also need fuzzy search on the data?
- Is strong consistency required (read after write, especially when you switch writes to a different data-center) or eventual consistency is OK?
- How much storage capacity is needed?
- What is the needed throughput and latency?
- If you choose self-hosted deployment, How much experience does your DBA team have with this technology, how mature is it?
- If you choose a managed cloud solution, What are the costs? What are its limitations?
Below you will find a handy flowchart to help you choose the right database technology for your needs.
For more in-depth look at the various criteria you can read my follow-up article — How to choose the right database for your service
I would like to thank Giedrius Graževičius, Karolis Astrauka, Donatas Remeika, Ittai Zeidman, Baruch Assif Osoveskiy, David Vaks, Avraham Rosenzweig, Idan Yael, & Eyal Malron for their help in detailing their DBs use-cases and technical insights
Thank you for reading!
If you’d like to get updates on my future software engineering blog posts, follow me on Twitter and Medium.
You can also visit my website, where you will find my previous blog posts, talks I gave in conferences and open-source projects I’m involved with.
If anything is unclear or you want to point out something, please comment down below.
