Jan 17, 2022Customize Spark runtime in IBM Cloud Pak for DataThis blog is written in Collaboration with Shobhit Shukla — He is the senior developer and data scientist with extensive experience in managing Spark runtime in public cloud and private cloud and has vast experience in working with internals of conda, Apache Spark, python and Jupyter Enterprise Gateway. IBM Cloud…26 min read26 min read
Published inIBM Data Science in Practice·Aug 13, 2021Spark on Knative (true serverless)Apache Spark is an open-source, scalable, massively parallel, in-memory execution engine for analytics applications. It is an essential tool for data scientists, offering a robust platform for a variety of applications ranging from large-scale data transformation to analytics to machine learning. This is an extension to one of my old…Spark6 min readSpark6 min read
Jul 30, 2021Apache Spark Shuffle Service — there are more than one options!!The purpose of this blog is to provide a list of Shuffle Service Implementations in Apache Spark and motivation for their design choices. This blog will also cover the high level understanding of Spark Shuffle service. Credit Note: All Credit goes to actual blogs and you tube videos published already…Apache Spark6 min readApache Spark6 min read
Published inIBM Data Science in Practice·Mar 24, 2021Analyzing Geospatial data in Apache SparkGeospatial data plays a crucial role in data forecasting, spatial analytics, and reporting, especially in the fields of logistics and finance. Additionally, there are increasingly more use cases where technologies like artificial intelligence have been used in tandem with geospatial applications. The purpose of this blog is to share a…Spark6 min readSpark6 min read
Published inIBM Data Science in Practice·Jan 25, 2021Working with different versions of Apache Hive with Spark service in IBM Cloud Pak for DataApache Hive is very popular data warehouse software. It is widely used for reading, writing, and managing large datasets residing in distributed storage which could be HDFS or S3 bucket using SQL. Structure can be projected onto data already in storage. Typically you will have multiple use cases where the…Spark4 min readSpark4 min read
Dec 3, 2019Using Apache Spark on IBM Cloud Pak for DataAI is not magic — but IBM Cloud Pak for data makes is magically simple! In each phase of the AI ladder viz, collect, organize, analyze and infuse, more and more customers are using Apache Spark for all the heavy lifting of data. IBM Cloud Pak for Data is a…Big Data4 min readBig Data4 min read
Jul 5, 2019Tips for handling Images, containers and your Kubernetes clustersContainers and Kubernetes clusters has evolved and become more mature with additional security measures introduced in the image build pipeline and how to configure your Kubernetes clusters. The purpose of this blog is to give a check list which you can refer to validate your docker build pipeline or your…Docker6 min readDocker6 min read
May 28, 20191 Million Spark run-times on KuberenetesI am very excited to share that we have successfully deployed over 1 million spark run-times in last 8 months. This service internally code named as Hummingbird is backed by its compute workload running of multiple Kubernetes clusters with each having 100s of nodes in different regions. We selected Hummingbird…Kubernetes2 min readKubernetes2 min read
Published inIBM Data Science in Practice·May 1, 2019When and Why Create a Kubernetes Operator?Introduction: In order to meet the increasing requirements of running more analytical jobs to get more insights into organizations data and reap benefits from it it makes more sense to move my Big data pipeline to cloud as it meets requirements of building and deploy analytical jobs within seconds with…Docker5 min readDocker5 min read
Dec 7, 2018Efficient way to connect to Object storage in IBM Watson Studio — Spark EnvironmentsWatson Studio has integrated with a number of Spark engines since it was first released as Data Science Experience in 2016. These include IBM Apache Spark as a Service, IBM Analytics Engine, and AWS EMR. These services are still integrated with Watson Studio, but Watson studio also support a new…Big Data3 min readBig Data3 min read
