Hadoop Noob

Debug Flink OOM in Docker Container

Recently I am planning to deploy a new Flink pipeline. I tested on my local and staging environment. When I deploy it to serve the full traffic, the TaskManagers are killed randomly. Since I have enabled externalized checkpoint, I won’t lose any data or state. But…

Presto Parquet Reader

Recently I am working on getting all our warehouse data queryable by Presto. We have lots of data in parquet format and our batch data pipelines are all spark jobs. They are normal ETL jobs. Data flows into Kafka, then Spark/Flink and finally are persisted on S3.

json vs msgpack

Which is better? It is really hard to say if we don’t give some context or constraints. Because if I could build it from scratch, I may choose neither.

So I have a cluster of Fluentd aggregators which streams data to treasure data. I need to fork the stream to kinesis…

Flink Forward 2018

It is a little late to write something about Flink Forward 2018. But I have to because we, well not me, actually co-present with Mesosphere about Flink on Mesos!

Notes on Apache Mesos Setup

Recently, I work on building a new data ingestion pipelines. I need to ingest data from kinesis and dump them on S3. Since I am familiar with flink and parquet, I decide to just use them. But before I can write and run some flink jobs, I need a cluster. I really like docker and I…

Dremio — best Parquet viewer

Sorry, I trolled on the title. Probably because I expected too much on it.

Back to early last year, Dremio came to our office and did a demo. It was a very informative talk and we asked a lot of Parquet related questions since they are contributors…

Benchmark: Spark SQL VS Presto

Cluster Setup:

Presto:

• Presto 0.152 (latest)
• 1 c3.xlarge node as coordinator. No work scheduled on master
• 3 c3.2xlarge node as worker

Recursive avro schema for parquet

I know it sounds stupid to use recursive data structure (e.g. a tree) in parquet, but sometime it happens. Why? Because you may need to consume some data which is not controlled by you.