Integrating Grafana Notifications with GitLab Pipeline to restart Debezium tasks using Go
Hello, all. In this article, we will integrate Grafana Webhook with our mini GoLang service with the power of the GitLab pipeline. You can find the repository on GitHub.
Before the solution, we have to explain our use case. We use Kafka in our CDC process, and sometimes few Debezium connector tasks fail, and we see error messages such that
Task is being killed and will not recover until manually restarted [org.apache.kafka.connect.runtime.WorkerTask]
so we have to restart manually to recover from that failure. So we ended up with this solution, which has worked flawlessly.
1. Setting GitLab Pipeline Trigger
At the time of writing this article, it’s enough to go to Your Project > Settings > CI/CD and expand Pipeline Triggers section. After adding the trigger, you will see all the details below (token, Webhook URL).
2. Create Grafana Webhook Notification Channel
We need to get our Webhook URL from Stage 1 and paste it into the Grafana Notification Channel below. Don’t forget to select type as webhook too.
3. Setting GitLab Pipeline
You can use the default Go template as a base.
We create a debezium stage. At this stage, we indicate that this is only a trigger job. To get the Grafana Webhook payload, we read the predefined TRIGGER_PAYLOAD variable and export it as an environment variable to use within Go.
Also, we need to specify DEBEZIUM_URL. To restart failed task in Debezium, we call Debezium’s rest API, so we need baseURL to communicate with the API.
4. Handle Error
In Go, we get our environment variable specified within the GitLab pipeline.
We parse our Grafana trigger payload to find the connector failed task within it.
After that, we get all the failed tasks and restart them.
For details, you can review the repository on GitHub. Thank you for contribution Seçkin Şen, Murat Cemcem and Mert Karayel.
