Pawan Kumar ShuklaRAG-Chat with your relational database from Azure Prompt Flow with LLMMost of the organisations want to use LLM based capability to provide accessibility of data for other departments /groups as per role…Jan 151Jan 151
Pawan Kumar ShuklaRAG with your Custom Data : Azure Prompt flow + Azure AI Search(Azure Cognitive Search)RAG(Retrieval-augmented generation), use case of Vector DB.Dec 4, 2023Dec 4, 2023
Pawan Kumar ShuklaHarnessing the Power of Large Language Models with Azure ML PromptFlow : Part 1 (Introduction)In today’s world, Artificial Intelligence (AI) is revolutionizing various industries by enabling intelligent automation and data-driven…Nov 18, 2023Nov 18, 2023
Pawan Kumar ShuklaAzure Databricks workflow for ingest the csv with which have header, body and footer in…In the Data Engineering work it is not always you will get a flat format csv file. Sometime you will get the csv files with Header, Body…Sep 30, 2023Sep 30, 2023
Pawan Kumar ShuklaTips: Use _metadata instead of input_file_name() function in Databricks runtime 10.5 and aboveWe can use the _metadata, a hidden column name for DBR 10.5 and above.Sep 30, 2023Sep 30, 2023
Pawan Kumar ShuklaHow to reduce the storage cost for delta tables?As now these days industry is moving towards more on Data Products and almost every organisation is going to build their own data lake…Sep 30, 2023Sep 30, 2023
Pawan Kumar ShuklaHow to update existing delta tables mount point ?Recently we got the feedback to rename the existing mount point with new mount point. And 100 delta tables were mounted on that old mount…Jul 25, 2023Jul 25, 2023
Pawan Kumar ShuklaHow to implement complex operation with Spark Structured streaming using foreachBatch?May 28, 20231May 28, 20231
Pawan Kumar ShuklaSpark Structured Streaming : Read and process data(XML and json)from Azure Event hubFor ingest the event driven data(real time ) in bronze layer(Medallion- architecture…May 21, 2023May 21, 2023
Pawan Kumar ShuklaExtract the data from Databricks delta table with help of Azure Data factory Copy activity with SQL…There is many ways to generate the csv file from Databricks delta lake, I am going to explain easy, cost efficient and with no…May 19, 20231May 19, 20231
