PinnedViral PatelFix data issues in existing delta tables without reloading data from source.Fixes a data issue in a Databricks Delta table by performing an update on selected columns using a specified SQL function.Mar 2Mar 2
PinnedViral PatelApply Encryption to PII Fields of Delta TablesWhen it comes to the PII (Personally Identifiable Information) or Commercially Sensitive data, It is necessary requirement of most of the…Jan 174Jan 174
PinnedViral PatelDatabricks Delta Table StatusIn this blog, I will walk you through quick POC I have implemented to track the status of the the Delta table using delta table history. I…Jan 254Jan 254
PinnedViral PatelMetadata Driven Unity Catalog PermissionsMetadata driven permission for UC SecurablesJan 15Jan 15
Viral PatelIngest/Migrate Data from IBM DB2 to Azure using Databricks at Scale.In this blog, I would like to share my experience of ingesting large amount data from IBM DB2 using ADF (Azure Datafactory) and Azure…Jul 11Jul 11
Viral PatelDatabricks Sql Warehouse ServerelessVertical Scaling Databricks Sql WarehouseMay 251May 251
Viral PatelMigrate Hive metastore (HMS) Schema to Unity CatalogWelcome to the Migration Story of HMS to Unity Catalog.May 251May 251
Viral PatelDelta Lake Database/Schema Maintenance Part-2Optimize Performance and Cost of Delta Lake at ScaleApr 6Apr 6
Viral PatelFlatten Json data/file in to PySpark Dataframe using Python function.In this blog post, I will walk you through how you can flatten complex json or xml file using python function and spark dataframe.Feb 181Feb 181
Viral PatelDelta Lake Database/Schema MaintenanceIn this blog post, I am going to walk you through the journey of delta table maintenance. The process I have impalement as POC and It has…Feb 12Feb 12