Benchmark: Spark SQL VS Presto

Cluster Setup:

Presto:

• Presto 0.152 (latest)
• 1 c3.xlarge node as coordinator. No work scheduled on master
• 3 c3.2xlarge node as worker

Dremio — best Parquet viewer

Sorry, I trolled on the title. Probably because I expected too much on it.

Back to early last year, Dremio came to our office and did a demo. It was a very informative talk and we asked a lot of Parquet related questions since they are contributors…

A Realtime Flink Parquet Data Warehouse

Recently I am working on migrating our currently pipelines (mostly pyspark) to JVM based. Our plan is to use spark for batch processing and flink for real-time processing.

Query Kafka on Presto

Recently I am working on a new data pipeline, it need to consume the Kafka data and do some transformation then persist the data on hdfs. When I finished my data pipelines, I need to start integration test on staging cluster. When some records are missing on hdfs, I need to figure out…

Presto Parquet Reader

Recently I am working on getting all our warehouse data queryable by Presto. We have lots of data in parquet format and our batch data pipelines are all spark jobs. They are normal ETL jobs. Data flows into Kafka, then Spark/Flink and finally are persisted on S3.

Recursive avro schema for parquet

I know it sounds stupid to use recursive data structure (e.g. a tree) in parquet, but sometime it happens. Why? Because you may need to consume some data which is not controlled by you.

Debug Flink OOM in Docker Container

Recently I am planning to deploy a new Flink pipeline. I tested on my local and staging environment. When I deploy it to serve the full traffic, the TaskManagers are killed randomly. Since I have enabled externalized checkpoint, I won’t lose any data or state. But…

Presto In Production

So what’s Presto

“Presto is an open source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes.”

Challenges