AWS re:Invent 2018 (Big Data and Analytics Sessions) — Day 3

Only attended two sessions today one of which was unfortunately a bit too dry to do a write-up for. However check out the below recap as well as my and Xavier Clements’s morning keynote recaps:

ANT205 — Technology Trends: Data Lakes and Analytics

In this talk, Anurag Gupta, VP for AWS Analytic and Transactional Database Services, talks about some of the key trends we see in data lakes and analytics, and he describes how they shape the services we offer at AWS. Specific trends include the rise of machine generated data and semi-structured/unstructured data as dominant sources of new data, the move towards serverless, SPI-centric computing, and the growing need for local access to data from users around the world.

Links: Portal


Company wealth has been moving away from traditional material centric companies to more data centric companies.

Over the last two decades there has been a dramatic change:

There is more data than people think — data growth has been greater than 10x every 5 years — you have to plan for this data growth now not later.

Today there are more ways to analyse data than ever before, Hadoop, Elasticsearch, Presto, Spark have all come around in the last 11 years.

Data lakes help you effectively scale

  • Store exabytes of data
  • Load, transform, and catalog your date once
  • Make it available to many tools

We need to rethink what we mean by data and analytics. Today we collect data in order to help with transactions and not necessarily with the customer journey.

  • Data can be used to benefit the customer during their journey. Things like understanding the customers favourite coffee every day, what sports team they like so you can strike up a conversation when they come into the coffee shop.
  • Example regarding Amazon Echo: Important data are not necessarily the questions that have been answered, it’s the questions that haven’t been answered. This data will help improve the customer journey and make sure the next time they ask the question, Alexa has the answer.
  • Example regarding Amazon Go (supermarket stores): Relating this to supermarket membership cards, those cards give a lot of information to the supermarket around the items you purchase, patterns etc. However, Amazon Go takes this one step further. By monitoring customers throughout the store, Amazon can understand customer routes within the store, they can understand what items you picked up or put down. What items you swapped for others etc. This adds critical data that can in the future add loads of value.

Amazon data and analytics ecosystem