How A Shell Works

Walking through “ls -l” step by step

Co-authored with Kevin Yook

The shell is a command line interpreter for your operating system. It can utilize a command line interface (CLI) or a graphical user interface (GUI). A CLI allows users of any operating system to type in a defined set of commands (e.g. rm to remove files, cat to combine text files, etc) and have the shell execute and print out the result of that command or an error message within less than a millisecond. It is slightly different from a GUI. Instead of typing in commands (e.g. to remove files) and seeing the terminal write it out in words (e.g. list all your files and you’ll see the unwanted one gone from the list) most people just use a mouse to click to open folders and delete files. GUI and CLI both have the same purpose to interact with the operating system but their input methods are different and some developers prefer using the CLI to interact with the shell because it’s faster. Most people just use a GUI.

There are many different versions of shells amongst different operating systems. For instance, in Unix we have the “sh” shell (created as the first Unix shell in 1971 by Ken Thompson, also the author of the B language and co-creator of GoLang) to the most common “BASH” shell (the 1989 Bourne Again Shell is an improved version of the sh and Bourne shell). Each shell’s general structure is similar. Later shells handle memory leaks better and each different version might handle different commands differently (e.g. echo “\n” prints differently in the sh and BASH shells).

Shells in general use system calls — a function that requests services from the Kernel; which is a core component of the operating system that has complete control over the system. It provides an essential interface between a process and the operating system. It connects the machine to the operating system.

So what exactly happens?

1. Shell activates: Once you open a terminal on whatever operating system you’re on, you’ll activate the CLI.
2. Prompts user: A prompt is always printed to the screen (standard output) with a blinking cursor to cue that it’s waiting on you to type something. (e.g. $)
3. Stores your input: getline() is the function used to wait for you to type something in. It will read everything you wrote until you hit enter and store that command into a space in your computer’s memory (buffer). getline performs a few checks and acts accordingly. If it reads that you typed exitor hit Ctrl-Dor Ctrl-C, the shell will immediately terminate the CLI. If it reads that a user typed in enterfor a new line, it will go back to step 2 and reprompt you for a new command.
4. Tokenizes your input: If you typed in something that doesn’t cue the termination of the shell or the reprompting for a new command to be entered, the function strtok()is used to break your stored command into separate strings (tokens.) It knows to separate tokens by the spaces (delimiter.) Any extra spaces put in by accident when the user types in the command will be ignored. In a simple command without pipes |, the first token is the function to be performed and the rest of the tokens are options (detailing how to perform the function) and required arguments (e.g. filenames.)
5. Checks if alias: The shell will first check to see if this first token is the result of one of your shortcut names (aliases.) If it is, the shell will then replace the first token with a string of tokens from your alias.
6. Checks if builtin: The shell will then check if the first token is a builtin program specific to the shell and not an executable. If it is a builtin, it will execute that builtin, print the result on your terminal, and skip down to step 11, repeating the process to reprompt you for more commands.
7. Checks if executable in PATH: If it was not a builtin, the first token is first matched to see if it exists. The directories that contain all the executable functions the shell can handle are in the PATH environmental variable. The PATHis similarly tokenized like your command so each directory is separated by their special delimiters (e.g. :) and compared. If no match is found, the shell will try again and append your command’s first token onto each PATHdirectory token and will check for a match again. In essence, the command can only be executed using it’s absolute full path. Note: After we’ve checked it exists, we also have to check that it’s an executable command. If it’s not an executable command, steps 8–10 are skipped, the shell will print an error message pertaining to that specific situation, and you’ll be reprompted with step 11.
8. Creates child process: The shell performs a system call usingfork()that creates a child process that works simultaneously along with the parent process. Once the fork is created, the rest of the program is duplicated. The goal of this child process is to execute your command. The duplication of the program is necessary because the execution (discussed in step 9) will kill the program once it’s done executing the command. The shell makes sure to handle all command executions in the child process so only the duplicate gets killed. Meanwhile, the parent should wait() (another system call) until the child is finished executing your process.
9. Execute or print error message: Once the shell detects it’s inside the child process, a system call such as execve()will execute your command and print the result of the command onto the standard output. After the execution, the system call kills the program. Since the fork duplicated the program, the child process is the one that is killed.
10. Parent finishes waiting and wakes: After the child process fully terminates, the parent is done waiting and will begin to perform again. Since the shell has returned a result of your command to the standard output, it has done it’s job. Some versions of shells will take this time to free memory allocated to certain things.
11. Reprompt and repeat: After a result of your command is printed, the shell will reprompt you (e.g. $) The shell continuously repeats steps 2–11, reading your typed in command until you hit enterand printing the result of that command onto your terminal until you cue it to exit , press CTRL-Cor CTRL-D, or press “x” to close your screen!

ls -l

Let’s go through these steps using a real example! ls -l!

1. Shell activates: You search for “terminal” on your computer and open it.
2. Prompts user: Depending on your shell, you see a prompt like $and a blinking cursor, waiting on you to type something. You type ls -land hit enter because you want to list all the files in your current directory in long format (with the file permissions, the owners of the files, the file size, the modification date, etc.)
3. Stores your input: The shell reads what you typed before the enter tab and stores ls -l in a memory space/buffer.
4. Tokenizes: The shell now sees the tokens lsand -l.
5. Checks if alias: The shell first checks to see if the first token lsis an alias. In the BASH shell, ls is an alias to ls — -color=autoso your new tokens are ls ,--color=auto , and -l.
6. Checks if builtin: ls is not a builtin.

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

7. Checks if executable in PATH: The shell then tokenizes and searches through the PATHenvironmental variable to try to find a matching executable. The BASH shell has the above PATH.It’ll go through these steps with your first token: Is “ls” /usr/local/sbin? No. Is “ls” /usr/local/bin? No. Is “ls” /usr/sbin? No. Is “ls” /usr/bin? No. Is “ls” /sbin? No. Is “ls” /bin? No. Is “ls” /usr/games? No. Is “ls” /usr/local/games? No. Since it’s none, the shell will try again by appending the first token onto the PATHdirectories and go through these steps: Does/usr/local/sbin/ls exist? No. Does /usr/local/bin/ls exist? No. Does /usr/sbin/ls exist? No. Does /usr/bin/ls exist? No. Does /sbin/ls exist? No. Does /bin/ls exist? Yes! Is this an executable? Yes!

8. Creates child process: The shell forks and creates a child process to execute this command.

9. Execute: The shell executes the command via the executable and absolute path /bin/ls. It executes according to your next tokens. When it sees a -, the lscommand will use whatever letter comes after it and interpret it as an option that adjusts it’s standard output. -lwill be interpreted as “long format.” The result of that is printed to the terminal.

10. Parent finishes waiting and wakes: Upon execution, the child process is killed. Your parent process, the shell is still running.

11. Reprompt and repeat: The shell is done with your command and will go back to the beginning of the steps and reprompt you with $waiting for another command! Cool huh?

I’m still here until you exit me!