Member-only story
DATA ENGINEERING & GENERATIVE AI
Large Models Meet Big Data: Spark and LLMs in Harmony
A step-by-step guide to use Apache Spark and large language models
Generative AI, including Large Language Models (LLMs), is revolutionizing different aspects of human life. Over the past five years, Generative AI has evolved from a research project into a real-life application for many people. As a data engineer interested in Generative AI, I have always asked myself, what does this technology bring to my work and Data Engineering applications? There are some common applications of Gen AI and LLMs for engineers such as pilot coding, assisting in documentation, and so on. But, here, I am evaluating some of the more specialized uses of Gen AI and LLMs for data engineering. If you are interested in this topic, please read this article and follow me on Medium and Linkedin to get more articles about other use cases.
LLMs: Powerful Tools for Transformations
It is not new that data engineers love structured and abstracted data. But, the world is full of unstructured and disorganized data that requires the attention of data engineers. Transformations on unstructured data are always complicated and sometimes impossible with traditional tools. Historically, one of…
