Reinvent Data FlyWheel

This blog is for Data Enthusiasts who want to understand how legacy databases are migrating to more modernise data structure and architectures available in today’s world.

The Data FlyWheel comprises of following steps :

1. Break free from legacy databases
2. Move to managed databases
3. Modernise your data warehouse
4. Build data-driven apps
5. Turn data to insights ; Goto Point #1

We generally wants to break free from legacy databases to save time and cost. This also enable us to remove undifferentiated heavy lifting from legacy databases. Moving to manages databases means adding agility, global distribution and to achieve performance at scale. In this data wheel, we moving towards more modernise data warehouses. This is pivotal step because this will help increase scale, improve performance and very obvious, reduce money $. Moving to managed databases, there are build data-driven apps which provides better and faster data insights. Companies can Migrate their on-premises or self-managed database services to Open source-compatible managed database services. There is no need to re-architect existing applications and you will get getter performance, availability, scalability and security. Picture below shows — Move to Managed relational databases provided by AWS :

Data from warehouse and data driven apps help turn data to insights. Picture below shows the Trend in Data warehouse.

App architectures & patterns have evolved over the years. There has been a colossal upsurge in Architectures from MainFrame to Mircroservices.

In New modern applications has new requirements. For example, they require following parameters to be in race :

1. USERS →in Millions , in Billions

2. Data Volume →in Terabytes, in Petabytes and in Exabytes

3. Performance →in Milli Micro seconds

4. Request Rate in Million +

5. Access →in Any Device

6. Scale up, Scale Out and Scale In

7. Economics →Pay as you go

8. Developer Access →in Managed API

It’s important to understand the common data categories and use cases.

1. Relational : Features : Referential Integrity, ACID Transactions, Schema-on-write. Common use cases : Lift & Shift , ERP, CRM, Finance. Example : AWS RDS. RDS consists of Aurora (MySql, PostgreSQL), Oracle, Microsoft SQL services , MariaDB.
2. Key-Value : Features : High throughput , low-latency reads and writes, endless scale. Common use cases : Real time bidding , shopping cart, social, product catalog, customer preferences. Example : AWS DynamoDB
3. Wide column : Features : Stores large amount of data with virtually unlimited scalability. Common use cases : Industrial Equipment maintenance, fleet management , route optimisation. Example : AWS Keyspaces (with cassandra capabilities).
4. Document : Features : Store documents and quickly access querying on any attribute. Common use cases : Content Management , personalisation, mobile. Example : AWS DocumentDB (with MangoDB capabilities).
5. In Memory : Features : Query be key with microsecond latency. Common use cases : Caching , Session store, leaderboard , geospatial services, real- time analytics. Example : AWS ElasticCache (with Redis and MemcacheD)
6. Graph : Features : Quickly and easily create and navigate relationships between data. Common use cases : Fraud detection , Social networking , recommendation engine. Example : AWS Neptune
7. Time- Series : Features : Collect, store and process data sequenced by time. Common use cases : IoT applications and event tracking. Example : AWS Time-series
8. Ledger : Features : Complete, immutable and verifiable history of all the changes to application data. Common use cases : Systems of records, supply chain, health care, registrations , financial. Example : AWS QLDB.