Creating Generative Art Dynamic NFTs with Google Cloud Vertex AI and Chainlink Functions
Previously, we have shown how Google Cloud and Chainlink Functions can be combined to unlock a number of new innovative Web3 use cases, such as on-chain Web3 Marketing. With the mainnet-beta launch of Chainlink Functions, developers can now build and deploy production ready hybrid blockchain/cloud applications that combine the highly secure and trust minimized world of blockchains and smart contracts, with the scalable and feature rich world of Google Cloud.
This blog walks through how you can use Chainlink Functions to connect your smart contract to Google’s Clouds Vertex AI Imagen model in a secure way and generate unique artwork for dynamic NFTs.
What is Vertex AI ?
Vertex AI is a machine learning (ML) platform from Google Cloud that lets you train and deploy ML models and AI applications. Vertex AI combines data engineering, data science, and ML engineering workflows, enabling your teams to collaborate using a common toolset
Imagen on Vertex AI (image Generative AI) brings Google’s state of the art generative AI capabilities to application developers. With Imagen on Vertex AI, application developers can build next-generation AI products that transform their user’s imagination into high quality visual assets, in seconds.
Google Cloud Vertex AI would make it easy for brands and organizations to create NFT campaigns with minimal efforts on the designing of the artworks / media.
What are Chainlink Functions?
Chainlink Functions provides your smart contracts with access to a trust-minimized compute infrastructure. Your smart contract sends your code to a Decentralized Oracle Network (DON), and each DON’s oracle runs the same code in a serverless environment. The DON aggregates all the independent runs and returns the final result to your smart contract. Your code can be anything from simple computation to fetching data from API providers.
By design, blockchains cannot talk and interact with external APIs due to their deterministic nature. Running compute-heavy workloads on-chain is also gas intensive, making it very expensive. So combining AI/ML and smart contracts isn’t usually possible at the protocol level; but now, with the introduction of Chainlink Functions, developers can make use of AI/ML such as Vertex AI in a trust minimized way directly from on-chain smart contracts.
How Chainlink Functions work ?
An on-chain Chainlink Functions ‘Router’ smart contract receives the request from your smart contract, then it emits an event with all the details received. The nodes in the Chainlink Functions DON detect the event and initiate a new request. They decrypt any passed in secrets required for the JavaScript code being executed, then each node in the DON executes the passed in JavaScript code.
In the JavaScript code that the Chainlink DON receives and starts to execute, an API call is made to the exposed API running on GCP, passing in the required input parameters .
From here, the oracle nodes in the DON all communicate to each other in a peer-to-peer network, and come to consensus on the final result or output of the JavaScript. This is achieved by the DON as it aggregates all return values together into a single report, and picking the median value (in ascending order) in the report as the final value to return. So if a report contains the values [1,2,3,4,5], the value 3 will be selected as the final value to return to the smart contract. This final report is then signed by each oracle node, and a single node in the DON is chosen to submit the final result to our on-chain smart contract that initiated the request.
Because all Chainlink nodes in the DON execute the JavaScript code, the GCP API that we previously provisioned will get called X times, where X is the number of oracles in the Chainlink DON (4 in the case of Mumbai testnet).
Use case
Just as NFTs swept the scene in 2021 and 2022, AI is the talk of the town today. We’ve witnessed the evolution GenAI has contributed to copy and content, but what about art and digital collectibles? To build an NFT project you need to understand how to code, produce art, write compelling storylines, and market your project., but as AI evolves and helps professionals eliminate mundane tasks, we’re witnessing a trend of AI generated NFTs.
In this scenario we are developing an NFT campaign featuring popular monuments. To create unique digital artworks, you intend to leverage the capabilities of Generative AI. To gamify the campaign, you also intend to make the NFTs dynamic, meaning that their attributes and images can be updated based on predefined rules and real-world data. In this example we are using Random seed to dynamically update the NFT for every update request, instead of random seed you can use any real-world data like weather data, sports data etc. depending on your use case.
In this tutorial, we’ll show how you can use AI to generate unique NFTs with a simple predefined set of attributes.
Architecture Overview
GCP Stack
GenerateNFT is a Cloud Function which will create a text prompt to generate images using the Imagen Model from Vertex AI and
This function has the following functionalities
- Generate the prompt from predefined set of attributes
- Use the Generated prompt to generate Image using Vertex AI Imagen Model, know more about the Imagen on Vertex AI
- Compose the watermark logo on the image
- Upload the Image and metadata to GCS Bucket
To keep the demo simple we are using Google Cloud Storage to store the NFT Metadata and Images in a deterministic URL. In prod scenario you may want to store the NFT Metadata in a decentralized storage like IPFS and GCP Cloud Function can return the CID to chainlink functions.
We are using Imagen Model from Vertex AI to generate Images from the text Prompt.
Chainlink Function & OnChain Stack
We’ll be using Chainlink Functions as a means for the on-chain NFT contract to pass the parameters to the Vertex AI API. The request is initiated on-chain via the smart contract function call, passing in the following:
- JavaScript source code for the Chainlink DON oracle network to execute
- Optional authentication secrets, encrypted with the Chainlink DONs public key (For this demo we made the Cloud Functions Publicly callable, for prod scenario you must use some S2S auth mechanism between Chainlink Functions and Cloud Functions)
- Chainlink Functions specific config values such as subscription ID and gas limit
We have an NFT contract using the ERC721 token standard which inherits FunctionsConsumer.sol allowing our NFT contract to interact with the FunctionsRouter to initiate a request.
NFT Campaign Design
Campaign design is one of the key aspects of launching the NFT Project. Here we have taken the popular artefacts theme, you can modify these attributes according to your campaign and create unique NFTs with Google Cloud Vertex AI
Below is the sample code to generate a prompt randomly with a predefined set of attributes. To know more about Prompt and image attribute you can check this guide
Here are some sample output NFTs generated using above solution.
Send Request with text prompt to Vertex AI
Here is the full documentation on Image generation model API
Overlay logo on generated image
Store Image & metadata in GCS
GCP Cloud Function
You can find the chainlink portion of the code in this github repo https://github.com/smartcontractkit/functions-google-vertex-ai
How NFT mint / update works
- The user invokes the mint function in the NFT smart contract.
- The mint function in the smart contract places a request to the Router smart contract, which in turn invokes the Chainlink Function execution in the DON.
- The oracle nodes in the Chainlink DON call the GCP Cloud Functions API with the token ID and random seed. (There will be multiple oracle nodes invoking the GCP Functions.)
- The GCP Cloud Function generates a unique text prompt from the predefined set of attributes, calls the Imagen Vertex AI model with the prompt and seed to generate a unique image, and finally stores the image and metadata in a GCS bucket. The function then returns the token URI to the Chainlink Functions.
- The Chainlink Functions aggregate the results from the multiple oracle nodes and call the fulfill method on the smart contract.
In this case, since we are storing the metadata and images in the predefined URL, and we are using a seed to generate the image via the Vertex AI Imagen model, the API result will be deterministic. In the case that we use IPFS, we can return the CID for chainlink functions to aggregate the results.
Conclusion
The example above showcases how Chainlink Functions can be used as a way to integrate Google Cloud services such as Cloud Functions and AI/ML Platforms directly into decentralized applications, while maintaining a high level of trust minimization. By combining the unique properties and benefits of cloud services with blockchains and smart contracts, an entire new wave of exciting and innovative use cases becomes possible.
To explore how Chainlink Functions works and how you can use it to transform your Web3 developer experience, visit https://chain.link/functions. You can get started right away in the Chainlink Functions Playground too. If you want to see more community-submitted examples or submit your own, visit Use Chainlink Functions.
To know more about Generative AI support on Google Cloud Vertex AI visit https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview
