This tutorial shows how to use the Scheduled Backups tool to configure Cloud Scheduler for creating Cloud Spanner backups on a regular basis, e.g. daily or weekly.
We will use the following GCP services:
The architecture is as follows:
We will need to do the following steps:
This tutorial will show you how to enable the session-related metrics in a demo application with the Cloud Spanner client library in Go. These metrics allow for better monitoring of the session pool and help us to quickly identify session issues:
The background is that I have an S3 hosted website but the DNS records are managed by Google Domains. Previously, it doesn’t support HTTPS, so I would like to add it now.
Two more AWS services are required:
5. Add a record set in “Custom resource records” (Google Domains):
Domain name: [FGHIJK].acm-validations.aws.
I was working on Kubernetes CronJob and I was wondering what
startingDeadlineSeconds is. There is official documentation, but I am still confused after reading it.
After looking at the source code, I think
startingDeadlineSeconds means that if a CronJob controller cannot start a job run on its schedule, it will keep retrying until
startingDeadlineSeconds is reached.
Before showing a few examples, we need to clarify some concepts:
Controller check: CronJob controller checks things (watching and syncing jobs) every 10 seconds.
Schedule: the time to execute the job according to the given schedule expression.
Job run: a job object is created…
This all begins with my need to schedule a script to crawl some stock data weekly. In a web service or application, we always have some needs to do a job at fixed times, dates, or intervals.
The most famous job scheduler is cron, which provides a utility for scheduling repetitive tasks on Linux/Unix systems. Cron Expression is commonly used to let you define when tasks should be run. You can check this website crontab.guru to configure the cron expression. There are different variants of Cron Expression used in systems, like Jenkins, Kubernetes CronJob, Fargate Scheduled Task etc. …
Actually, this post is related to my previous one about creating a dead letter queue. Now, the problem is how to read from this queue and re-publish celery tasks to a queue.
Three ways that I found can read messages from rabbitmq:
pika is a pure rabbitmq client library in Python.
The following code allows you to read messages from queues in rabbitmq from RabbitMQ Tutorial — Work Queues:
import timeconnection = pika.BlockingConnection(
channel = connection.channel()channel.queue_declare(queue='default.deadletter', durable=True)
def callback(ch, method, properties, body):
This is a summary for git workflow management, which is an important part when having a team working together with git. There are several approaches to manage the workflow:
Simple. Everything is a single master branch.
# git pull [remote repository] [branch]
$ git pull origin master
$ git commit
$ git push origin master
git pull is a shorthand of
git fetch and
git merge FETCH_HEAD.
git merge to merge the retrieved branch heads into the current branch.
A---B---C master on origin…
Last year, in my current company, we spent three months to re-architect and implement a new data capture pipeline based on some ideas how a good dataflow system should be, such as immutable rawest data, log-based message broker, recomputation, and etc. The new capture pipeline is much more scalable and reliable. With the log-based message broker, it has the ability to reprocess all raw data and generate derived data when needed.
Continue to my last post. How should we route tasks to different queues in Celery?
Here is the example:
We can specify queue, exchange, routing_key when calling
apply_async. However, it is very confusing that when calling
apply_async(routing_key='xxx'), the message goes to the default queue. If you don’t set
app.conf.task_default_queue, it will create a queue with name
Actually, the trick is to specify either
exchange + routing_key. I tried the following examples:
add.apply_async((1,2), exchange='default', routing_key='moon')
add.apply_async((1,2), exchange='default', routing_key='sunshine')
Recently, I started learning how to run async tasks in Celery, which has been used in our company product. One interesting requirement is how to create a dead letter queue in Celery?
RabbitMQ has already supported arguments,
x-dead-letter-routing-key, when creating a new queue.
I have made a simple example to create a dead letter queue:
You need a rabbitmq as a broker and redis as a backend. Run
docker-compose up -d to start containers.
celery -A tasks worker --loglevel=info# open ipython
from tasks import add, div
div.delay(2,0) # throws ZeroDivisionError
PhD. Software engineer at Google. Living in Australia.