Hadoop Noob

fluentd out_forward plugin is stuck longer than expected

json vs msgpack

Which is better? It is really hard to say if we don’t give some context or constraints. Because if I could build it from scratch, I may choose neither.

So I have a cluster of Fluentd aggregators which streams data to treasure data. I need to fork the stream to kinesis…

Debug Flink OOM in Docker Container

Recently I am planning to deploy a new Flink pipeline. I tested on my local and staging environment. When I deploy it to serve the full traffic, the TaskManagers are killed randomly. Since I have enabled externalized checkpoint, I won’t lose any data or state. But…

Notes on Apache Mesos Setup

Recently, I work on building a new data ingestion pipelines. I need to ingest data from kinesis and dump them on S3. Since I am familiar with flink and parquet, I decide to just use them. But before I can write and run some flink jobs, I need a cluster. I really like docker and I…

Flink Forward 2018

It is a little late to write something about Flink Forward 2018. But I have to because we, well not me, actually co-present with Mesosphere about Flink on Mesos!

Flink Fault tolerance using externalized checkpoint

As I am writing this, Flink already on 1.4 release and 1.5 snapshot is already out. But we are still on flink 1.3.2

I want to talk a little bit about Flink externalized checkpoint. Flink’s checkpoint is a great…

Dremio — best Parquet viewer

Sorry, I trolled on the title. Probably because I expected too much on it.

Back to early last year, Dremio came to our office and did a demo. It was a very informative talk and we asked a lot of Parquet related questions since they are contributors…