How A Shell Works — What happens when ls -l is typed?
At Holberton School these past few weeks, students in my cohort were assigned the task of coding our own Linux Shell in the C Programming Language that worked similarly to the sh shell written by Ken Thompson. Using some of the knowledge gained from writing this project, the subject of this article is about what happens when a user types in ls -l in the Linux shell. The shell I will be using to demonstrate is the one that we wrote for this project. We called it hsh for Holberton Shell (I think). Here is a link to our repository.
What is a Shell?
In short, the shell is a command line interface (CLI) program that takes commands entered by a user from a keyboard and gives them to the operating system to execute. The shell is like the medium through which a user communicates with the kernel, the fundamental part of a computer’s operating system that controls the actual computer hardware (typically designed by a computer engineer) and causes various (and tiny) transistors to turn on and off. There are various shells that are typically used, including sh and bash. So what happens when a command like ls -l is typed into the shell?
Very High Level Explanation
In Linux, the command ls is used to list the contents of the current working directory. The -l flag displays it in long format so that additional information is displayed including file permission configurations, size, dates, and times. That is an extremely high level explanation. Here is a picture showing what the command looks like when typed into bash.
So…What’s REALLY Happening?
The cool thing about writing your own shell is just how much you learn about how methodical the inner workings of a computer program actually are.
The basic mechanism of a shell is an infinite loop that keeps running until an exit command or end-of-file (EOF) value is sent. Each time the shell runs through one loop, one command is entered and executed. The magic happens each time the loop is run.
The following is a summary of what happens and each step will be elaborated on with respect to ls -l specifically:
1. The shell prompt is displayed
The shell prompt typically displayed is a $ as seen below. There will typically be a cursor that is blinking, waiting for the user to enter a command.
2. The shell reads the command from standard input that was entered by the user
Typically, this is done using a getline() function which saves the user-entered line from STDIN into a buffer, including the newline character.
3. The shell tokenizes the string command entered by the user
A string tokenization function is called which splits the command line into tokens. In our shell, we used a function called strtok() which took the line to tokenize and the delimiter to define token boundaries. A token is an element between a delimiter. In this case, the delimiter is typically a space character such that each token is one word. The delimiters are replaced with a \0 null byte so that each token is its own string. So for instance, for ls -l, ls is a token, and -l is another token because they are separated by spaces and each token ends with a null byte. These tokens are typically stored in an array of strings with a terminating NULL pointer for accurate parsing.
4. The shell checks if the first token (the main command itself) is an alias, and if so, replaces the alias with the actual command
The shell will typically look in its system files for defined aliases. If the ls command is an alias for something else, the shell will replace the ls token with the string for the command that ls represents so that the correct operation takes place in the subsequent steps.
5. If it is not an alias, the shell checks if the command is a built-in and executes its code accordingly if it is
The shell checks if the entered command matches any of the built-ins. Built-ins are commands that are explicitly coded into the shell itself. The actual code is not from an executable, but actually exists in the source code of the shell. Examples include exit, cd, and env. Since ls does not match any built-ins, it must be an executable. An example of a builtin (env) is shown below using the shell we coded. This builtin displays the environment variable. The environment variable is an array of strings. Each string is a specific variable associated with how the system runs.
6. The shell passes the environment variable $PATH to the process and tokenizes each directory
When the command does not match any built-ins, the shell then implements the $PATH variable. This is seen below:
The $PATH variable is basically a list of directories that the shell looks through for executables. The shell picks this specific variable from the environment variable and uses the : as a delimiter to split the variable up into tokens. The way we did this in our shell was split the environment variable up into a singly linked list.
7. The shell appends the command entered to the end of each directory and checks if the executable exists. If the executable does not exist in any $PATH directories, an error is thrown
The shell then goes through each directory in the $PATH variable and appends the ls command to the end of them. For example, the first directory is /usr/local/sbin, so the shell will combine this directory with ls to create the following: /usr/local/sbin/ls . The shell then puts this string into functions called access() (with an argument F_OK) or stat() which returns 0 if the file exists and -1 if the file does not. The shell would then check if the file it found is an executable using access() with a X_OK argument. If the file does not exist or is not executable, it moves to the next directory and does the same thing. If it reaches the end of the $PATH variable, this means the executable does not exist and an error is thrown.
8. When the executable is found, the parent process is then forked to create a child process. The child process executes the command while the parent process waits for it to finish
If the executable is found, the shell goes into an execution function. The concept of child and parent processes is then used. The following three commands are most important for this:
fork()wait()execve()
The fork() command creates a child process from a parent process. The child process is what will be used to run the command. The way this works is that two processes are now running concurrently and these two processes have two different process identification numbers (PID) which allow the system to recognize which process it actually is.
The child process and parent process run concurrently, but execute different code because of their differing PIDs. In this case, the parent must employ the wait() function so that it waits for the child to terminate before continuing running. The child on the other hand employs the execve() function which takes the executable (/bin/ls), the array of tokens ({“/bin/ls”, “-l”, NULL}), and an optional environment argument (usually NULL) to run the executable. The execve() function terminates the child upon completion so that the parent receives a signal to continue and finish running the program. After this is run, the listed contents of the current working directory are finally displayed on the terminal!
9. After running a command, the shell executes any shutdown commands programmed into it, frees any allocated memory from its operations, and re-prompts the user for input
After completion, the shell executes any shutdown commands contained within it, frees any allocated memory that was for storing strings, lists, and arrays in the heap, and initiates another iteration of the main shell loop. This causes it to re-prompt the user with the $ again as seen below.
The shell intrinsically keeps count of how many iterations through the loop it takes during one shell session. This is demonstrated below by the errors that are thrown for unknown executables. The numbers shown are the number of iterations done. This number is counted up, whether the commands executed successfully or not. Notice how the numbers skip from 3 to 5 after a successful pwd command is executed in between(iteration number 4).
This was way more complicated than just “listing the contents of a working directory”! Coding our own shell was a very interesting and truly humbling experience. It was very interesting because mostly all of the concepts that we learned about when completing our C projects came together during this project. I have learned so much about how a shell works inside and I hope this article helped shine some light on how complicated, methodical, but interesting the whole process is!
