8/28 AWS Big Data Workshop

Hui Yi Chen
Aug 29, 2017 · 1 min read

Ingest (收集)=> Store => Process => Visualize

Ingest: Web server logs, Devices (mobile deep linking), Framework Logs(Kinesis)

  • Database => Redshift (交易類型頻繁,查詢複雜,分析使用; 不適合each raw insert, 適合batch insert, SQL => S3 => Redshift; New feature:Spectrum預熱的服務,可直接使用)
  • ETL => AWS Glue -fully managed ETL
  • AWS Kinesis => click 3 times to build-up the service. (Access log to csv)
  • DAX => for 讀取較重,讀取效率較快 (第一次讀取時間差不多,第二次開始,原本的大約是5毫秒, DAX大約是0.9毫秒)
  • Quick Sight
  • Athena / EMR => S3可直接丟進分析
)
Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade