Amazon Athena stands out as a powerful query engine, offering a seamless way to perform interactive queries on data stored in Amazon S3 data lakes. This advanced tool has revolutionized the way data analysts interact with large datasets by providing a server-side, serverless solution. With Athena, the data remains in…

Hello community, Today, let’s delve into the world of data query optimization, focusing on a common issue encountered when using query engines like Athena or Presto. The Small Files Challenge in Athena and Presto Many of you might have experienced this: when working with distributed query engines like Athena or…

Cloud-based and big data-based projects require a lot of guidelines to keep all factors in check and accounted for — review the most important ones here. The aim of the following article is to share with you some of the most relevant guidelines in cloud-based big data-based projects that I’ve…

What is apache Spark Shuffling Introduction Apache Spark is an open source data processing framework widely used for large-scale data processing and analysis. It was designed to be easy to use while providing high performance for batchs and streaming data. Spark’s distributed architecture is one of its key features, allowing it to scale and process data…