What I learned after using Delta Lake 2.0 on AWS EMR along with installation steps and performance benchmarks — If you have read my previous article about getting started with Delta Lake in AWS, you would have got the fundamental context and rationale that why the offerings like Delta Lake are gaining traction and what type of use-cases they address. The article presented simple and easy steps to get…

A step-by-step tutorial to configure Apache Spark and Delta Lake on EC2 in AWS along with code examples in Python — If you have worked on engineering a datalake or lake-house solutions, chances are that you may have employed (or have heard of) de-coupled and distributed computation frameworks against scalable storage layer of your datalake platform. Though the list of such computation frameworks is growing, but Apache Spark has continued to…

By using AWS CodePipeline, CloudFormation, Lambda and Python. — If your team follows leading DevOps practices, there is quite a high likelihood that you are maintaining your infrastructure as a code (IaC). IaC approaches accelerate consistent and repeatable deployment of your infrastructure and its configuration. One of the use-cases where IaC is particularly helpful is in CICD pipelines where…

If you have been working in Data Engineering space, chances are that you would’ve been involved in processing CSV files. Even though its not the most efficient format for analytics but CSV format still enjoys quite a significant footprint in the current data landscape. It’s a widely supported format and…

My hypothesis is that if there is one thing that we all data-guys can agree on, it's that logs are crucial for any data application. When set up correctly, logs unlock a lot of insights about the operation of an application. If an application behaves normally, logs are the go-to…

Machine Learning, the ability to learn from data, has been one of the most successful and disruptive use-cases of Big Data. In the landscape of data and analytics, one has access to myriad of tool-set to undertake machine learning tasks of varying nature and complexity. However when one is operating…

If you are in data and analytics industry, you must have heard of the burgeoning trend “data-lake” which, on simpler notes, represents a storage strategy that allows organizations to store data from different sources and of different characteristics (size, format and velocity) in one place. Data-lake then becomes an enabler…

Scalable analytics in the cloud is name of the game these days. All the leading cloud providers are focusing significantly on provisioning services that streamline end-to-end lifecycle of machine learning. The trend these days is to have data ingested on data-lake (which requires its own set of considerations) and process…

An extensive tutorial on how to use Azure Cognitive Services (Text analytics API) to perform sentiment analysis using Databricks (Python, Scala) — First Section — Extracting Tweets Sentiments can manifest anywhere from reviews, news, real-life conversations, journalism to name a few. The capability to identify the polarity of sentiments accurately by employing machine learning approaches unlocks series of business use-cases that can yield immense value for businesses. Instead of humans to parse through the content to infer…