Passing gitlab artifacts from pipeline to pipeline

As a part of the Trendyol Storefront International Team, we’re creating and maintaining Trendyol International web and mobile web sites. We are using micro-frontend architecture while building our apps. Through this blog post, I’m excited to share an insightful experience that I gained during an integration that we used to automate our localization process.

In this post, we’ll discover how to effectively transfer artifacts from one GitLab pipeline to another and learn how to trigger a pipeline from another pipeline.

Context Table

Passing gitlab artifacts to another gitlab pipeline.

• How we use GitLab artifacts API
• Create a gitlab.yml file using GitLab artifacts API

Triggering gitlab pipeline

• How we use Trigger Pipelines API
• Passing trigger payload.
• Write a global environment value as an artifact.
• Create a GitLab job using via Trigger pipelines API with PAYLOAD

Flow of integration

Here’s a blueprint of how we’ll proceed today.

Now, Let’s get started.

We want to share artifacts of the gateway project to the translation-uploader project. Then we will upload those artifacts to CDN from the translation-uploader project.

To accomplish this, we will be using the Gitlab Artifacts API.

First, we will hit the get job artifacts endpoint. You will need a project ID and the specific job ID where you stored the artifacts.

PS: We will show you how to get these pieces of information later.

For now, let’s assume that our project ID is 1 and the job ID is 42.

curl --location \
     --output artifacts.zip \
     --header "PRIVATE-TOKEN: <your_access_token>" \
     "https://gitlab.example.com/api/v4/projects/1/jobs/42/artifacts"

After retrieving the zip file, you will need to unzip it to use the artifacts. You can do this by running the following command:

unzip artifacts.zip

$ ls 
/src/translations

Now let’s take a look at how to integrate this flow into the gitlab.yml file. We will have three stages:

image: node:latest

stages:
  - install
  - download-translation-artifacts
  - upload-translations-to-cdn

In the first stage, we will install our project to the GitLab runner.

install:
  stage: install
  script:
    - yarn
  only:
    - master

In the second stage, we will integrate the same structure that we just used to download the artifacts.

We will be getting the project ID and the job ID values as environment variables. (I will show you in the next section)

download-translation-artifacts:
  stage: download-translation-artifacts
  script:
    - echo "Downloading artifacts.zip"
    - "curl --location --output artifacts.zip 'https://gitlab.domain.com/api/v4/projects/'$PROJECT_ID'/jobs/'$JOB_ID'/artifacts' --header 'PRIVATE-TOKEN: '$TOKEN''"
    - echo "Unziping artifacts.zip"
    - "unzip artifacts.zip"
  artifacts:
    paths:
      - artifacts.zip
      - ./src/translations
  only:
    - master

💡 QUICK TIPS: If you define artifacts in your gitlab.yml file, GitLab will store these files as global storage and you can access these files throughout the pipeline.

In this case, we saved the ./src/translations directory because we plan to upload these values to a CDN in the next stage.

In the third stage, we will upload the values to a CDN. The upload-cdn script will read the src/translations directory and upload all files to the CDN.

upload-translations-to-cdn:
  stage: upload-translations-to-cdn
  dependencies:
    - install
    - download-translation-artifacts
  script:
    - yarn upload-cdn
  only:
    - master

Now, we can get artifacts when this pipeline is triggered with the correct project Id and job Id from another pipeline.

Time to trigger pipeline with necessary payload.

Trigger a gitlab pipeline from another gitlab pipeline

https://docs.gitlab.com/ee/ci/triggers/

To trigger a GitLab pipeline from another GitLab pipeline, you will need a pipeline trigger token. You can get this token by following these steps:

1. Main menu > Projects and find your project.
2. On the left sidebar, select Settings > CI/CD.
3. Expand Pipeline triggers.
4. Enter a description and select Add trigger.
5. You can view and copy the full token for all triggers you have created.
6. You can only see the first 4 characters for tokens created by other project members.

PS: ref is your branch name for example; master or main

curl --request POST \
     --form token=<token> \
     --form ref=<ref_name> \
     "https://gitlab.example.com/api/v4/projects/<project_id>/trigger/pipeline"

When you run this curl you will trigger the any project pipeline successfully but we need to send a data while triggering the pipeline.

To accomplish this, we will use variables attributes

For example, if you want to send the “UPLOAD_TO_S3” variable as “true” while triggering the pipeline, you can modify the curl command as follows:

curl --request POST \
  --form token=TOKEN \
  --form ref=main \
  --form "variables[UPLOAD_TO_S3]=true" \
  "https://gitlab.example.com/api/v4/projects/123456/trigger/pipeline"

After sending this request, you can see the "UPLOAD_TO_S3" variable in the job details of the triggered pipeline.

Let’s get back to our example. We want to send the project Id and the job Id

To send PROJECT_ID we can use Predefined gitlab variables.

--form "variables[PROJECT_ID]=$CI_PROJECT_ID"

But sending the job ID can be tricky because you are sending the trigger request from another job. You cannot define a global variable inside a job, so you need to save the job ID to a text file and save it as an artifact. Then, you can read the text file and use it as a variable in the trigger job.

For example, in the "fetch-translations" job of the gateway project, you can save the job ID to a text file as follows:

fetch-translations:
  stage: translations
  script:
    - echo CI_JOB_ID= $CI_JOB_ID
    - echo $CI_JOB_ID >> ARTIFACT_JOB_ID.txt
    - touch ARTIFACT_JOB_ID.txt
    - yarn fetch-translations
  artifacts:
    paths:
      - src/translations
      - ARTIFACT_JOB_ID.txt

Then, in the "snapshot-translations" job, you can read the text file and use it as a variable in the trigger command:

snapshot-translations:
  stage: deploy-prod
  only:
    - master
  script:
    - export ARTIFACT_JOB_ID=$(cat ARTIFACT_JOB_ID.txt)
    - curl --request POST --form token=$TRANSLATION_UPLOADER_TRIGGER_TOKEN --form ref=master --form "variables[PROJECT_ID]=$CI_PROJECT_ID" --form "variables[JOB_ID]=$ARTIFACT_JOB_ID" "https://gitlab.domain.com/api/v4/projects/$TRANSLATION_UPLOADER_ID/trigger/pipeline"
  when: manual

After this integration you can trigger the translation-uploader project with the translation-snapshot job and when you triggered that translation-uploader pipeline start to work.

We’ve completed the integration.✨ I hope you’ll enjoy taking a look at this real-life example and I would love to hear your thoughts on this example. Please feel free to share your feedback and ideas in the comments below. It’s always great to learn from others and continue growing together!

If you’re interested in seeing more content like this, please consider following me and Trendyol Tech.

Want to work in this team?

Be a part of something great! Trendyol is currently hiring. Visit the pages below for more information and to apply.

Have a look at the roles we’re looking for!