Unix command-line interpreter under the hood

If you type echo "hello, World!"on your terminal you will get the same output of the image above. This can be quite obvious and simple, because as a programmer we have a close relationship with the terminal and surely we execute much more advanced commands than these, including long bash scripts. But have you ever wondered how all this works under the hood?

What is a Shell?

Before diving into the process, we should understand what is a shell. So answering this question, a Shell is a program that takes the command inputs written from the user’s keyboard and passes them to the machine to execute them through the kernel. It also verifies if the command inputs from the user are correct.

Terminals, kernels, and shells…

It is worth clarifying that the kernel, the terminal, and the shell are different things.

Here is an overview to understand the differences:

One thing I want to clarify in the above image is that each box is within the parent box. So for example, The “Program: terminal” is interacting with a contained shell, while the shell then spawns either a “Built-in” or “Program”. source: https://www.integralist.co.uk/posts/terminal-shell/

Main differences and important concepts:

• Kernel: a computer has a kernel, the kernel is responsible for managing the computer’s system, the kernel has no user interface, to interact with the kernel you use an intermediary “program”.
• Program: a “program” is a structured collection of instructions (machine code) that a computer can execute. Your computer has many programs. One such example would be the ‘terminal emulator’ program.
• Executables: executables (or executable binaries’) are programs. More specifically, an ‘executable’ is a file that contains a program, generally the result of a program being turned into something that can be ‘executed’ by the computer.
• Terminal: a terminal is an input/output device. Traditionally they would have been a real ‘hardware’ device that you used to interact with a computer. e.g. the computer would be a large box in a server room, and the terminal would be a monitor/keyboard connected to the computer. In modern computing, we have a ‘terminal emulator’. If you don’t want to use a GUI (graphical user interface) to interact with your computer, you can use a terminal emulator.
• Shell: a shell is a program that is accessed via a terminal emulator. The terminal accepts input, passes it to the shell, and the shell’s output is sent back to the terminal to be displayed. The shell accepts input as a set of commands.

If you want to read more about this here I leave you a good resource: https://www.integralist.co.uk/posts/terminal-shell/

What happens when you type `ls -l *.c` in the shell

Since we understand that it is a shell, it can receive a variety of commands to perform a task, one of those is ls -l *.c. If you have a little experience working with shell commands you know what will be the result, if you don’t know don’t worry here is the answer:

ls -l *.c

Let’s break this into small parts, first let’s start by reviewing the ls command. ls list information about the files in the current working directory, so if we type ls the output will be like:

ls

-l is a flag of ls command which means the files will be listed in long format:

-rw-------   1 me       me            576 Apr 17  2019 weather.txt
drwxr-xr-x   6 me       me           1024 Oct  9  2019 web_page
-rw-rw-r--   1 me       me         276480 Feb 11 20:41 web_site.tar
-rw-------   1 me       me           5743 Dec 16  2018 xmas_file.txt

----------     -------  -------  -------- ------------ -------------
    |             |        |         |         |             |
    |             |        |         |         |         File Name
    |             |        |         |         |
    |             |        |         |         +---Modification Time
    |             |        |         |
    |             |        |         +-------------  Size (in bytes)
    |             |        |
    |             |        +-----------------------       Group
    |             |
    |             +--------------------------------       Owner
    |
    +---------------------------------------------- File Permissions

So if we type ls -l the output will be:

ls -l

Lastly *.c is a wildcard and it is used to match any character or set of characters. So putting all this together ls -l *.c will list all the files ending in .c in long format.

What happened under the hood?

A shell, in a very simple way, is RPEL (Read, Parse, Execute, and Loop) process. The lifetime of a shell is summed up in a loop, which receives commands from the standard input, the shell Read a command line, Parses (tokenizes) the command, Execute the command with the corresponding flags and options, and finally, the shell waits for the user type a new command to repeat the process (Loop).

Let’s look at this in a little more detail

As we said before, the life time of a shell is summed up in a loop. When the shell enters the loop, the following things happen:

1. Print the Prompt: the first shell task may seem a bit obvious and that is because it is about printing the prompt. A prompt is text or symbols used to represent the system’s readiness to perform the next command. A prompt may also be a text representation of where the user is currently. The prompt is found behind the cursor and is highly customizable.
2. Read line: once the prompt is printed and the user types a command, the shell must read the entire line from the stdin and store it in a buffer. A function that is commonly used is getline() which read a line from the stream and returns a pointer.
3. Parse line: now that the shell has the entire line, it must separate the line into multiple elements, the reason for this is because the first element of the line refers to the command, and the following elements refer to the options and arguments. strtok() is a function that split a string into multiple tokens, it divides the initial string according to the specified delimiter. So the shell could use strtok() to split the entire command line into multiple elements.
4. Execute arguments: we already have the command and its options and arguments. But before executing the command, we must take into account that most of the commands that a shell executes are programs, but not all of them, some commands live in the shell itself, they are called built-ins. This means there are two types of commands: built-in and non-built-in.
5. Built-in commands: built-ins are the functions that live inside the shell itself, they are implemented within the shell code. So first the shell must check if the input command corresponds to a built-in, if so, the shell would make the call as if it were another function.
6. Non-built-in commands: if the command is not built-in, the shell must create another process in which the executable file has to be found on the system and executed along with its options and arguments. fork() function can be used to create a new child process that runs in parallel with the parent process. Inside the child process, we use execvp() to execute the program that matches the command.
7. Output: the output will be displayed in the terminal (suppose we are using the shell in interactive mode).

So far, we have seen the RPE process, to complete the missing piece, the above steps will be repeated until the shell reaches the End Of the File (EOF) or the user exits the shell. Now we have completed the whole RPEL process.