Handling different file formats with Pyspark

Spark support many file formats. In this article we are going to cover following file formats:

• Text
• CSV
• JSON
• Parquet
  Parquet is a columnar file…

How to use Elasticsearch as Vector Database

Lets setup single node Elasticsearch cluster on local machine.

Pull the docker images.

docker pull docker.elastic.co/elasticsearch/elasticsearch:8.12.2
docker pull docker.elastic.co/kibana/kibana:8.12.2

Manage your secrets inside containers using AWS SSM

Let’s say, we are working on a python application which is running inside a docker container and it needs access to database hosted on AWS RDS. Storing the database credentials inside repository is not recommended. So we need a solution to…

Loading large size compressed files in BigQuery (BQ)

BQ load jobs have the size limit of 4GB for a compressed CSV file. If we try to load > 4GB compressed CSV files in BQ, we get the limit exceeded error.

Gitlab CI/CD using specific runners on AWS EC2

GitLab CI/CD can automatically build, test and deploy our applications. Runners are processes that pick up and execute CI/CD jobs for GitLab.

We can configure our gitlab CI/CD jobs to run on shared runners or specific…

Adding GCS support to EMR

Ever wondered how can we read from or write to Google Cloud Storage(GCS) from AWS EMR!

To let EMR spark application talk to GCS, we need to add cloud storage connector in application jar.