Introducing Goblero: a Go Embedded Job Queue

Adil H
Adil H
Feb 13 · 6 min read

What is a Job Queue ?

Web App Flow: Client -> Http Handler -> Job Queue

Why an Embedded Job Queue ?

Enter Goblero

git clone https://github.com/didil/goblero-demo.git
go get -u github.com/didil/goblero/pkg/blero
go build .
./goblero-demo enqueue
$ ./goblero-demo enqueue
Starting Blero ...
Registering Processor 1 ...
Enqueuing jobs ...
[Processor 1] Processing job: Job #1 - data: Job Data #1
[Processor 1] Done Processing job: Job #1
[Processor 1] Processing job: Job #2 - data: Job Data #2
[Processor 1] Done Processing job: Job #2
[Processor 1] Processing job: Job #3 - data: Job Data #3
[Processor 1] Done Processing job: Job #3
[Processor 1] Processing job: Job #4 - data: Job Data #4
[Processor 1] Done Processing job: Job #4
[Processor 1] Processing job: Job #5 - data: Job Data #5
^CCaught signal interrupt. Exiting ...
Stopping Blero ...
$ ./goblero-demo
Starting Blero ...
Registering Processor 1 ...
[Processor 1] Processing job: Job #6 - data: Job Data #6
[Processor 1] Done Processing job: Job #6
[Processor 1] Processing job: Job #7 - data: Job Data #7
[Processor 1] Done Processing job: Job #7
[Processor 1] Processing job: Job #8 - data: Job Data #8
^CCaught signal interrupt. Exiting ...
Stopping Blero ...
Starting Blero ...
Registering Processor 1 ...
Registering Processor 2 ...
Registering Processor 3 ...
[Processor 3] Processing job: Job #9 - data: Job Data #9
[Processor 1] Processing job: Job #10 - data: Job Data #10
[Processor 2] Processing job: Job #11 - data: Job Data #11
[Processor 2] Done Processing job: Job #11
[Processor 3] Done Processing job: Job #9
[Processor 1] Done Processing job: Job #10
[Processor 2] Processing job: Job #12 - data: Job Data #12
[Processor 3] Processing job: Job #13 - data: Job Data #13
[Processor 1] Processing job: Job #14 - data: Job Data #14
[Processor 2] Done Processing job: Job #12
[Processor 1] Done Processing job: Job #14
[Processor 2] Processing job: Job #15 - data: Job Data #15
[Processor 1] Processing job: Job #16 - data: Job Data #16
[Processor 3] Done Processing job: Job #13
[Processor 3] Processing job: Job #17 - data: Job Data #17
^CCaught signal interrupt. Exiting ...
Stopping Blero ...

Quick Benchmark

# Core i5 laptop / 8GB Ram / SSD 
make bench
BenchmarkEnqueue/EnqueueJob-4 50000 159942 ns/op(~ 6250 ops/s)
BenchmarkEnqueue/dequeueJob-4 5000 2767260 ns/op(~ 361 ops/s)

Todo:

Adil H

Written by

Adil H

Freelance Backend Developer. Golang, Python, Node.js, GCP Certified

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade