Naidu RongaliAWS Lakeformation : Managing Cross-Account Permissions.This week I worked on creating a Glue table whose permissions needs to be managed by Lakeformation . Granted access for querying a table…Nov 10, 20231Nov 10, 20231
Naidu RongaliAWS Glue Table Serde Settings for reading Escaped CSV fileAfter running a crawler to create Glue table to read from a csv file which have values in double quotes , noticed that the serde settings…Nov 1, 2023Nov 1, 2023
Naidu RongaliAlternatives for Querying RDS PostgreSQL data from Datalake / DatawarehouseWhile analyzing various ways of getting data out of RDS PostgreSQL to Datalake, came to know about a couple of alternatives to do this.Oct 30, 2023Oct 30, 2023
Naidu RongaliSpark : How to debug Malformed CSV record ErrorWhile Unit testing a new Spark SQL pipeline , got below error.Oct 29, 2023Oct 29, 2023
Naidu RongaliPostgreSQL’s various data typesWe have JDBC driver based application for querying data from PostgreSQL DB and we wanted to unit test if it handles various data types…Oct 29, 2023Oct 29, 2023
Naidu RongaliAWS Kinesis Firehose : Adding newline at the end of each JSON row. Why and How?We recently had to develop a pipeline for publishing data to Datalake from the DynamoDB. The solution we took involves DynamoDB Kinesis…Oct 29, 2023Oct 29, 2023
Naidu RongaliMy DS Doubts : Probability Vs MLEWhile going through Advanced Linear regression lecture, came across a note from the professor which goes like this “we can go by the…Jun 3, 2023Jun 3, 2023
Naidu RongaliGDPR Fundamentals for Data engineerGDPR stands for General Data Protection Rules. These rules specify how enteprises and business should handles the data of the persons they…May 26, 2023May 26, 2023
Naidu RongaliData Science : Cheatsheat for Different Data Distributions and Correspodnig R FunctionsI recently completed Statistical Data distributions and R Progrmming courses as part of my ongoing PG Diploma in Applied Statistics. Came…May 6, 2023May 6, 2023
Naidu RongaliWhat is Feature Store, Online vs Offline Feature StoreThe term “Feature” refers to a value . It could be Categorical value like Gender or Continuius Values like Height or Cordinal Values like…Apr 28, 2023Apr 28, 2023