Tharun Kumar SekarUnderstanding Streaming Query MetricsTo optimize a Streaming Pipeline, Streaming query metrics is the right place to begin your analysis.Dec 10, 2023Dec 10, 2023
Tharun Kumar SekarDynamic Partition Upsert — SPARKIf you’re using Spark, you probably know what partitioning is, and perhaps you would have even encountered Dynamic Partitions. But even…May 17, 20222May 17, 20222
Tharun Kumar SekarHigher-Order Functions — PythonA programming language is said to support First Class Functions if it treats functions as first-class objects. By definition, a…Sep 23, 2021Sep 23, 2021
Tharun Kumar SekarSlowly Changing DimensionI wanted to learn about Slowly Changing Dimension for a long time, but i couldn’t find a clear, concise blog post for anyone not familiar…Oct 7, 20201Oct 7, 20201
Tharun Kumar SekarData Reconciliation in SparkData Reconciliation is defined as the process of verification of data during data migration. In this process target data is compared…Sep 20, 2020Sep 20, 2020
Tharun Kumar SekarSpark Parallel Job SubmissionSpark is known for breaking down a big job and running individual tasks in parallel. But, this doesn’t mean it can run two independent…Sep 6, 2020Sep 6, 2020
Tharun Kumar SekarinAnalytics VidhyaGit — BasicsGit is a powerful tool, but it has a reputation of baffling newcomers. With the right knowledge, anyone can master git. Once you start to…May 29, 2020May 29, 2020
Tharun Kumar SekarGit Squash Commit With Git RebaseWhen submitting a pull request to merge your code with Master/Develop, it’s better you squash your commits. Some applications that…May 20, 2020May 20, 2020
Tharun Kumar SekarImprove Spark Write PerformanceThe EMRFS S3-optimized committer is a new output committer available for use with Apache Spark jobs as of Amazon EMR 5.19.0. This…Apr 14, 2020Apr 14, 2020
Tharun Kumar SekarCase Classes — ScalaRepresenting data is a key part of writing programs, but it’s also mechanical: you need to define your fields, setters, getters, and other…Apr 10, 2020Apr 10, 2020
