Using Boomi platform to Automate API Calls and store into GCS bucket
Boomi is an integration platform as a service (iPaaS) that connects applications and data sources. It is a low code development platform that provides API, lifecycle management and event-driven architecture features for cloud integration.
In this article, we will explore the power of boomi to integrate an API and store the data in GCS Bucket.We will be using a test API from https://jsonplaceholder.typicode.com/ for testing purpose.
Prerequisites- You need to have a boomi account.
Before starting, let us understand some useful properties in boomi.
Dynamic document property: Dynamic document property are the properties that the process developer can define and use to temporarily store additional pieces of information about a given document.
Dynamic process property: A dynamic process property (DPP) in Boomi is a process property that temporarily stores metadata data pertaining to the execution of the Boomi process. DPPs are not tied to a document, and the value can be retrieved from a document or an element in the document
Now that we know what Dynamic document and dynamic process properties are, lets start step by step:
- Create a service account and attach a private key to it.
Create a private key of type P12 in your service account in GCP, then, download the key/certificate.
2. Upload the certificate in boomi
Open www.boomi.com, login to your account, create an atom and upload the key inside the create new tab and select X.509 Certificate option. Then, import the key. After importing, the certificate looks like below image in boomi.
3. Add a Google Storage Connector shape
The next step is to add a Google Connector shape inside the process. We need to set Action type as CREATE, provide Connection details, and Operation.
4. Add Google Storage connection
Create a connection component of type OAuth 2.0. Provide Grant type as JWT Bearer Token, in Scope field provide https://www.googleapis.com/auth/cloud-platform, add the certificate created in Signature key, provide the Common Name from the certificate in both Issuer and Subject field. Add this connection into the connection field of the google storage connector shape.
You can also test the connection and check if the certificate is successfully configured. After it shows successful, we will know its configured successfully.
5. Create a Google Storage Operation
Create a new google storage operation and provide the bucket name where the data is to be stored. Click on Import operation and add the details, click on next and select import new Object and Select Object as Object Type, then click on Finish. check the error Behaviour option. After all the details is filled, the operation looks like similar below:
6. Add a HTTP Client Connector Shape
Create a new HTTP client Connection from the connection field. Add the URL of the API and provide the Authentication Type. Here in our case, Authentication Type is none. Create a new HTTP Client Operation as operation. Then, create a JSON Profile and add all the keys from the API inside data elements and add it to Request Profile. Create a same JSON Profile and add it to Response Profile.
Set the request profile type as JSON, set the content type to text/plain, set the HTTP Method as GET.
Under the Resource Path, provide the subdirectory part of the URL. Since the postid is variable here, check the ‘is replacement variable?’.
Create a process property and add bucketname and postid. Provide the default value of postid as 1.
Boomi Pipeline
Now let us create a boomi pipeline merging all these shapes.
The pipeline will start from a Start Shape of type No Data. Add a Set Properties Shape. Here, we need to add a Dynamic Document Property of postid. For its value, we will assign from the Process Property’s posyid that we earlier created. The set properties will get connected to HTTP Client Connector Shape. We will again add a Set Properties Shape. We add a Document Property of source type connectors, Connector as Google storage and Property as Bucket Name. For its property value, we will assign it from the process property. We need to provide object name. Therefore, we will create a Dynamic Process Property named GCS_objectName. In this property, we will provide the path where the data from the api will get stored. A json file will get created inside data folder named currentdatepostid.json where currentdate is the current date in MMddyyyy and postid is the postid.
Process to Test the pipeline
- After making sure all the components are added and confugured, save the process and click on Test button
- A window will open where we need to select atom and click ok
- After successfully testing, all the component’s icon in the process will have a green outline indicating no issue and the process has successfully completed.
- By clicking on the icon of each components, we can get the logs and shape source data.
Therefore, in this way, we can integrate any API with boomi and store the data into bucket.
References
