Anuvrat Singh

Editor of A Muggle’s Pensieve

40 Following

130 Followers

Featured

Anuvrat Singh in A Muggle’s Pensieve

Apr 5, 2017 · 4 min read

Writing Into Dynamic Partitions Using Spark

Hive has this wonderful feature of partitioning — a way of dividing a table into related parts based on the values of certain columns. Using partitions it’s easy to query a portion of data. Hive optimizes the data load operations based on the partitions.

301

12 responses

Latest

Anuvrat Singh in A Muggle’s Pensieve

Jun 5, 2018 · 2 min read

Ira Spring Trail — Mason Lake

We reached the trail-head by 9:30 on a sunny Saturday morning early June to find the parking completely…

Anuvrat Singh in A Muggle’s Pensieve

Apr 29, 2017 · 3 min read

Reuse Hive Scripts

Amazon’s Elastic Data Pipeline does a fine job of scheduling data processing activities. It spawns a cluster and executes Hive script when the data becomes available. And after all the jobs have completed the pipeline shuts down the EMR resource and exits. Since the cluster is only created and…

Anuvrat Singh in A Muggle’s Pensieve

Apr 28, 2017 · 2 min read

Hive + DynamoDb: Always Specify Region

DynamoDb is a key-value storage store. One can query DynamoDb tables from Hive using the…

1 response