Pawan Kumar ShuklaDatabrick Auto Loader Support XML Type fileThis feature is currently in Public preview when I am writing this article. In my previous article I have mentioned to use the Auto loader…Jul 10Jul 10
Pawan Kumar ShuklaRAG-Chat with your relational database from Azure Prompt Flow with LLMMost of the organisations want to use LLM based capability to provide accessibility of data for other departments /groups as per role…Jan 151Jan 151
Pawan Kumar ShuklaRAG with your Custom Data : Azure Prompt flow + Azure AI Search(Azure Cognitive Search)RAG(Retrieval-augmented generation), use case of Vector DB.Dec 4, 2023Dec 4, 2023
Pawan Kumar ShuklaHarnessing the Power of Large Language Models with Azure ML PromptFlow : Part 1 (Introduction)In today’s world, Artificial Intelligence (AI) is revolutionizing various industries by enabling intelligent automation and data-driven…Nov 18, 2023Nov 18, 2023
Pawan Kumar ShuklaAzure Databricks workflow for ingest the csv with which have header, body and footer in…In the Data Engineering work it is not always you will get a flat format csv file. Sometime you will get the csv files with Header, Body…Sep 30, 2023Sep 30, 2023
Pawan Kumar ShuklaTips: Use _metadata instead of input_file_name() function in Databricks runtime 10.5 and aboveWe can use the _metadata, a hidden column name for DBR 10.5 and above.Sep 30, 2023Sep 30, 2023
Pawan Kumar ShuklaHow to reduce the storage cost for delta tables?As now these days industry is moving towards more on Data Products and almost every organisation is going to build their own data lake…Sep 30, 2023Sep 30, 2023
Pawan Kumar ShuklaHow to update existing delta tables mount point ?Recently we got the feedback to rename the existing mount point with new mount point. And 100 delta tables were mounted on that old mount…Jul 25, 2023Jul 25, 2023
Pawan Kumar ShuklaHow to implement complex operation with Spark Structured streaming using foreachBatch?May 28, 20231May 28, 20231
Pawan Kumar ShuklaSpark Structured Streaming : Read and process data(XML and json)from Azure Event hubFor ingest the event driven data(real time ) in bronze layer(Medallion- architecture…May 21, 2023May 21, 2023