PySpark: Write data frame with the specific file name on HDFS

Write PySpark data frame with specific…

What Happens When You Type Google.com in Your Browser and Press Enter?

Coding problems asked in my Data Engineering Interview

List of coding questions that I was asked to…

A Conversation with chatGPT on Data Engineering

Getting a perspective on how OpenAI understands Data…

Data Modeling Interview for Data Engineers: Part V

Flipkart Interview Experience

How to write Pytest for PySpark Application

Learn how you can write Pytest for your PySpark Application like a pro

Pytest is a popular testing framework for Python that makes it easy to write and run tests for your code. In…

SCD: Slowly Changing Dimension

A Data Warehousing Concept

PySpark: Write a dataframe with a specific filename in the S3 bucket

Write PySpark data frame with…

Apache Spark On Apple Silicon

Install Spark(Scala, Python) on Macs having M1, M1 Pro, M1 Max, and M2…

Encrypt and decrypt data frame in PySpark

Learn how to encrypt and decrypt columns of PySpark data…