Jenkins and Cron Jobs
Everybody knows that Jenkins is a wonderful tool for running builds, tests, deploying software, etc. But, what about running cron jobs?
Most companies have cron jobs to run the most diverse tasks and, as the number of tasks grow, the maintainability decreases exponentially. This happens because usually people create their own cron jobs manually, without a tool for managing it. So, having many cron jobs spread through your VMs is not that fun, right? Here are the main problems, in my opinion:
- Hard to know what are the tasks that you have running, you have no visibility.
- Hard to monitor…how do you know that your cron job ran? You know because you probably receive an e-mail with some kind of status, right? Not ideal.
- How to check the logs of your last running cron job, that unfortunately failed?
- What if you want to change the date and time that your cron job is supposed to run? And again…do you know where it's running? Hard to know if you have many VMs…probably you have some spreadsheet to document it? Are you sure the spreadsheet is up-to-date? Probably not. And please, don't create a spreadsheet for things like that ;)
- Something stopped to work in production….hmmmm…maybe a cron job is the responsible! Should be great if you can just check what ran in the last few hours. Can you do it easily? No? So, keep reading.
So, you have Jenkins, you are using Continuous Integration correctly, you have a job to deploy your software, maybe you have a Continuous Delivery pipeline…but you still can not answer the questions above? So, this is easy to fix, you're lucky, trust me.
If you are a Jenkins user, you know you can configure almost any kind of jobs to do. Why not adding your cron jobs there too? If you never heard about Jenkins, it's a software that is famous for running Continuous Integration, Continuous Delivery, unit tests, etc. Doesn't mean that, if you have Jenkins running, you have CI/CD, but this is subject for another post.
On Jenkins, you can define your tasks as jobs, and you can have, for example, one job to run your unit tests when you push code to your git repository. How Jenkins knows how to execute your tests? Magic? No, you can define it as a shell script or install some plugin that will do the magic for you. You have also a nice dashboard to keep track of what’s running and you can schedule jobs to be executed by defining a periodically (ex: every Saturday, at 4am) date based on a cron-style date/time (0 4 * * 6). What’s a cron job if not a scheduled job? So, this possibility was always there and you never used? No worries, I did this before too, that’s why I’m writing this post! :)
That’s it. You just have to create a job, configure what you want to be executed, when you want it to be executed and now you have a nice dashboard, logs, the possibility to send an e-mail after your job is finished (or why not a Slack message to your team channel?)…need to publish something to S3 after the job finishes? No problem, you can use a plugin for this kind of task. More than that, now your entire team can see what is being executed, what was executed last night. If some build fails, everybody will know about this if you publish the failure to slack. It’s also about improving communication, having everybody knowing about the tasks that are executed daily. It should not be something to hide. I hate communication problems, that’s why I love tools like slack, but without culture, doesn’t matter the tool…even slack can be useless.
Another good point is to use some version control for your cron job too. If it’s a custom job, it’s code, right? Do you have unit tests? You should.
You can say that you have to run some of your jobs locally, but did you think about exposing your jobs as APIs? It’s easy to deploy an API today, could be some REST interface exposed by the framework you love, within a Docker container. Jeff Bezos said, in 2002: "All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions." Now we have Amazon AWS. Many thanks to Jeff Bezos! :)
Having some jobs securely exposed as an API and/or a Jenkins job can also give more power to the developers. You, as a DevOps, don’t want to do manual tasks or to have a developer asking you to run some task. If, for example, the dev wants fresh data in another environment (staging?) to perform some tests, ideally he would be able to perform this task with one-click, without asking you.
At the end, many things are just about culture.