A practical guide for creating an effective design document I often get these questions during my “data architecture” related mentoring/training sessions. What design aspects should be covered in the design document? Is there really a need for a design document? How much detailed info should be added to the document? I'm writing this article so that it can help…

Do these 3 things daily. — How much experience do you need to become a data architect? 5 years? 10 years? 15 years? There is no correct answer! In my view, any “data person” (anyone working in the data world as a data engineer, ETL developer, ML engineer, data analyst or a tech PM) has the potential to become a data architect. YES, you can also become…

Data is evolving, so are the data architectures A couple of months back, I wrote about my journey of the last couple of decades in the data world & how things changed from building warehouses to building lake houses as a combination of lake & warehouse Off-late, things have been changing rapidly in the data world with architectures…

A practical guide to securing your data & metadata — Data Security is critical. We all know it. We all want to secure our data — whether in the lake, warehouse or lake house. And when it comes to practically implementing the data protection rules, we don't want to break it down into theoretical categories such as data security or…

Have you invested days, if not months, in reconciling your data post-migration to the cloud? Have you been doing multiple parallel executions of your ETL workloads to reconcile your data daily? Are you building accelerators & reusable frameworks for automating your data reconciliation process? If you have done all these…

Managing data access for external & internal users Introduction With modern data stack & new age tools, accessing & querying data from the data-lakes, warehouses & lake-houses has become easier & quicker. However, with all this data & multiple services, identities & principals trying to access this data — data access…

One simple step that can reduce your migration efforts, time & cost — Why I am not able to find the workflow in which this Informatica Mapping is triggered? Why can’t I see the shell script from where this Spark job is submitted? Does this PL/SQL script really gets executed in production? In case you have been in the above situations & have…

Almost a decade ago, when I worked as an ETL developer (those days, “data engineer” was not so common term), I developed multiple ETL jobs to ingest data from various source systems. As part of unit testing, I often created dummy test data to validate the transformation logic & to…

From being optional to mandatory — Introduction Like any other governance, data governance is also required to regulate practices & processes — specifically around data ingestion, storage, access & usage. Data Governance is a system or an approach to organize, control & effectively manage data so that it can be easily accessed & used for insight…