Description A partitioned table is a special table that is divided into segments, called partitions, that make it easier to manage and query data. So, you split large data in smaller units, you split compute part into smaller unit, you map smaller unit of compute to storage to do processing, a…

Overview Pub/Sub is an asynchronous messaging service designed to be highly reliable and scalable. The service is built on a core Google infrastructure component that many Google products have relied upon for over a decade. Google products including Ads, Search and Gmail use this infrastructure to send over 500 million messages…

Overview Scalability at node level can be easily manageable, however when your applications run inside container and app demands more CPU & memory then there is chance that the application is crashed or pod is restarted. DevOps engineer should consult with application engineer to understand required maximum and minimum memory &…

Overview Submitting Spark job to GCP Dataproc is not a challenging task, however one should understand type of Dataproc they should use i.e. the way how they will invoke to Dataproc. Why one should submit job directly to Dataproc Traditionally, Dataproc on GCE uses yarn for scheduling & managing resources. If your old jobs are directly submitting to master…

Overview This article will provide you a sense of understanding on how we have determined performance of various machine learning algorithms using some sample data. Auto ML or Auto Model is part of Rapid Miner AI Hub(RM AI Hub) that automatically splits the dataset into training & test datasets. Training &…

Overview Establishing connectivity between Rapid Miner Studio & GCP Dataproc is challenging when both vendors never test such scenario. As a part of modernization of Hadoop platform, we decided to explore the capability of Dataproc as well as possibility of running ETL or AI models inside Dataproc while jobs will be…

Description This article will guide you how to connect Rapid Miner Studio to Cloudera CDP public running in GCP cloud. We used CDP control plane(This is part of CDP public) that was running in us-west1 region (This region is selected as it has most of the CDP features enabled). In addition…