Building a Mage Pipeline using AI
While I am still not out of my thoughts on how easy it was to create a data pipeline using Mage, Mage has gone to the next level !!! Yes Mage introduced in Beta, capabilities to generate data pipelines using AI. All you have to do is to prompt what you want to build while creating a new pipeline and a pipeline is built automatically for you. How cool is that
How do I get my hands on this ??
Simple. Follow these steps and you will be all set to generate one for yourselves.
- Generate a OpenAPI Key
a. Go to platform.openai.com/account/api-keys and create a new secret key for you
b. Make sure you have enough credit balance left
c. Keep the newly created secret handy as it will be required in further steps
2. Install Mage
You will need to install the ai version of Mage to get this to work. Thanks to
for guiding me and making sure it worked for me.Follow the below steps :
a. Install Python version > 3.8.1 as mage ai version only works with Python version above 3.8.1
b. Once done, setup your virtual environment
c. Run command : pip install mage-ai[ai].
Note : openai and langchanin modules get installed as part of this installation and there is no separate module installation required
d. Run command : mage start yourprojectname and you should see the project open up on your local host
3. Setting up Mage to use AI
For Mage to use AI, you will need to setup your project to use your API key. Go to the project settings and add the key you generated from Open API as seen below
You can also enable the add_new_block_v2 feature which will enable Mage to generate a block in your pipeline using GenAI as seen below
Once the project settings are saved, you are now ready to see the power of AI in play
4. Generate data pipeline using AI
Go to new pipeline creation option and you should see the menu Using AI(beta)
When you select this option, you will get a prompt window to enter the details where you can define the name of the pipeline and the instruction to AI(prompt) on what pipeline you would like to be built
Once details are entered, you will see pipeline generation status as below
In a matter of minutes, you will see a data pipeline built with all the boiler plate code for you
Isn’t this amazing ?? Go ahead and play around with these capabilities.
Possibilities seem to be endless with Gen AI and to see AI becoming our co pilots on the data engineering space makes the feeling even better..
Kudos to the entire team of Mage for such innovations. Keep them coming !!
