Lets setup single node Elasticsearch cluster on local machine.
Pull the docker images.
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.12.2docker pull docker.elastic.co/kibana/kibana:8.12.2
Ever wondered how can we read from or write to Google Cloud Storage(GCS) from AWS EMR!
To let EMR spark application talk to GCS, we need to add cloud storage connector in application jar.
BQ load jobs have the size limit of 4GB for a compressed CSV file. If we try to load > 4GB compressed CSV files in BQ, we get the limit exceeded error.
Containerisation is the process of creating a bundle of application code and all its dependancies.
This blog post explains various benefits of containerisation.
GitLab CI/CD can automatically build, test and deploy our applications. Runners are processes that pick up and execute CI/CD jobs for GitLab.
We can configure our gitlab CI/CD jobs to run on shared runners or specific…
Let’s say, we are working on a python application which is running inside a docker container and it needs access to database hosted on AWS RDS. Storing the database credentials inside repository is not recommended. So we need a solution to…
Spark support many file formats. In this article we are going to cover following file formats:
