Project Automation with Bash

How often does it happen that you have a project set up and you need to add a new feature? Going through the same monotonous method of creating a sub-directory, creating new files with the same old boring naming convention, etc.

Let alone one project, what if your “new bug” (everyone hopes that it’s not a bug) has a dependency on multiple projects. You could make the optimizations and common out anything and everything, but what about the cumbersome process that hasn’t evolved yet.

Where am I heading with this?

Those who use frameworks to create new projects are aware of the fact that they all have built-in utilities to create a new one with default template at just the user’s whim. These utilities not only cover just the initial settings but the directories, boilerplate code, and all sorts of comments and hints.

Wouldn’t you wish if you could just

npm init -y 

and be done with everything? Yes, I understand how you feel. Been there done that.

Well, I have blabbered enough, let us skip to the good part.

What got me started, and why bash?

As I earlier mentioned, doing a repetitive task without optimization in programming is a blunder and a cardinal sin. So I just wanted to automate what kept repeating. As simple as that.

Primarily, it was navigating to a certain directory, creating a sub-directory, then adding files. Then it increased to four repositories at once. Having worked with Linux before, I recalled that bash has every utility that I needed. Creating directories with sub-directories, creating files, adding context to files, resolving dynamic data with variables, deletion, git operation, and just way too many things to mention here.

Let me get you started!

Creating a setup
Since we aren’t working with node, but rather a dummy project, we’ll just proceed with dummy folders.

Project Structure

Here, we have our parent folder: Automation, with two sub-directories: Vegetables and Fruits, which have their own file needs and identifiers. At the same level, we have our Automator which will store the script and code templates for respective files.

Tools we will be using
- Bash
- Perl
- Cat
- Eval
- Awk/ Sed
- Npx on global user
- Git (Optional)

A lot of tools are redundant and may do exactly what you want to achieve without incorporating all of them, but it is essential to consider the complexity and time before you dive into it.

I use macOS for development and it ships with ‘zsh’ and ‘bash@3.2’. It is imperative to decide on either bash or Z shell before we proceed because it makes a huge difference on what and how we use them — syntactically and functionality-wise as well. For more context, you can refer to the blog below: -

https://medium.com/@harrison.miller13_28580/bash-vs-z-shell-a-tale-of-two-command-line-shells-c65bb66e4658

Adding on to why it is complex, for one, macOS ships with FreeBSD and not the GNUBash@5, so a lot of syntaxes vary. Second, awk/ sed behave differently in both variants. So it becomes taxing to build on. Another possibility to consider is the types of users who’d be using it, for example, a group of people can either use Linux or macOS. So a solution that caters to either group is preferred.

Hence, I chose ‘bash’ over ‘zsh’. But since FreeBSD has its own ‘sed’, the functionalities differ as well. So to overcome them, I am using ‘awk’ and ‘Perl’.

Next, we shall create a script inside our ‘Automator’ directory — setup.sh. But only creating the file isn’t enough, we have to make it executable.

Create setup.sh: -
touch Automator/setup.sh
Make it executable: -
chmod +x Automator/setup.sh

Now we will open our setup.sh script and start adding logic and flow to it.
As you can see below, the first line is called a ‘shebang’, typically used to let the machine know which shell you want to use. It generally is:

For Bash
#!/bin/bash
For zsh
#!/bin/zsh

Now add the below code to set paths for different parts of the project. We have also added a trap — to record any event of interrupt [ ctrl + c ], whether it is done by the user or the machine. When the interrupt is detected we call the cleanUp function.

Note: To call a function in shell, we just use it’s name.

Now that our paths are set, let us ask the user for details.

1. Ask whether the model is Fruit or Vegetable.
2. Ask the name of the model.

We will store this info in our variables and use them later to create file names based on them, or we can even export them for others to use, etc.

Note: reading a variable only needs you to “read -r VARIABLE_NAME”, but here I am using a while loop to check if varibale has recieved some value other than null. In case it is null then do not let the user proceed until he does enter something.

There are multiple things that we can do, and various info that we could ask from users. But if we let them input anything and everything they wish, we could end up handling a lot of runtime issues. For example, if an ‘apple’ folder already exists, you wouldn't want to have another apple folder, but what if someone enters apple as ‘aple’ or ‘appple’? It would lead to creating redundant models. Hence, to limit these mistakes, we can create a list that will hold all options a user can pick from. We can then sort the list and display it as options, if the required option isn’t there, then they can enter manually.

Before we proceed, let us talk about templating with bash and the use of variables. To create a new template, create a file with the .tpl extension, and set the path for the same inside the setup. It works best if you have common out code — that reduces the number of templates and the checks you would need to use them.

So after creating the files for your templates, copy and paste your code inside the template. Any dynamic data that is either dependent on what user enters or can be built up with what user enters, can be used as variables.

So let’s say your logic makes calls to different collections of a database, and the name of the collection keeps on changing as per the feature you’re adding. Make the name of the collection a variable.

Instead of
db.collection('nameOfCollection').find({});
Use
db.collection('${COLLECTION_NAME_VARIABLE}').find({});

Using uppercase for naming the bash variables isn’t a paradigm, but better to use as it stands out from the rest of the code. For a variable to be recognized and evaluated use ‘${VARIABLE}’. Then any available variable inside the script can be used and replaced with their placeholders, using eval.

Note: $, “” , `` which are prominently used a lot in code, are also part of bash special characters. When using them in your regular code along with templating, escape them using ‘\’, that way bash won’t interfere with them.

Model Logic Template

Config File Template

Now that our templates are ready, let us evaluate and use them to add new fruit/vegetable inside our project.

Note: After everything has been implemented, in order to successfully run the script, we need to set a start point, since everything is modular. At the end of the script, we will add [[ acceptModel “$@” ]], letting shell know where to start from.

With that done, we have our automaton all set. To use it, change the directory to your parent folder, and execute:

[[ parentDirectory]]> ./automator/setup.sh

There we have our Automator doing all your boring job.

BONUS

Now that a project is done, how to handle multiple projects. One way is to have automation scripts for each project. But it is cumbersome and difficult to manage, especially if all share similar data. So, make a master script that accepts all the data, and triggers other scripts from it.

|-Master Script
    |-ProjectA Script
    |-ProjectB Script

Thus, covering all projects. To achieve that, add these lines around your variables:

set -a
variable 1
variable 2
variable 3
variable 4
.
.
.
variable n
set +a

Any data between “set -+ a”, will be shared informally across all children scripts.

Additional/Optional
You can incorporate git functionalities like some are shared above and even make a lazy PR script.

https://gitbetter.substack.com/p/automate-repetitive-tasks-with-custom

If you use node, you can install ‘prettier’ globally, and keep your inserted code formatted as per your team guidelines.

Thank you for reading all along, if you find this article helpful, upvote.

Feel free to reach out to me in case of discrepancies or suggestions.