Managing the Files and Folders in Linux (PART I)

How to manage files and folders in Linux from the terminal?

In this article, you will learn about “Managing the Files and Folders in Linux” with Hands-on scenarios.

👨🏽‍💻🧑🏻‍💻For more ARTICLES, FOLLOW📍DevOps Engineering on Cloud

Let’s get started.

Generate empty files in Linux using the touch command

The touch command in Linux is used to change file access and modification times. You can get the complete details of the touch command by running any one of the following commands from the terminal. man touch or touch --help

🚨👉🏼 You can also check the complete udemy course (Linux Shell Commands for Absolute Beginners using Ubuntu 20x)🔗Referral link

NAME
     touch – change file access and modification times
SYNOPSIS
     touch [-A [-][[hh]mm]SS] [-achm] [-r file] [-t [[CC]YY]MMDDhhmm[.SS]] [-d YYYY-MM-DDThh:mm:SS[.frac][tz]] file ...

The output of the man touch command in Linux

So, Touch generally creates a new file if the file doesn’t exist. And it will update the timestamp if the file already exists.

Create a new file using touch in Linux

The touch command is very powerful when it comes to creating multiple files at once.

You can use the below command to create multiple files at once.

touch album{1..4}_song{1..5}.mp4

Create multiple files at once using the touch command in Linux

You can extract the information from the file names by using the following command in Linux.

ls -l album* | cut -d_ -f1 | sort | uniq

The output of the above command

Piping multiple commands in Linux to filter the file names.

You can remove the multiple files which are created above at once using the rm command with the pattern.

rm album{1..4}_song{1..5}.mp4

remove multiple files at once using the rm command in Linux

If you perform the touch command on non-empty files, They will behave the same as on empty files. It will modify the time stamp of the file.

A detailed video on the touch command in Linux

Creating Directories in Linux using mkdir command

The mkdir command is a command in Linux which is used to create a new directory. The mkdir utility creates the directories named operands.

mkdir — make directories

You can get the full details of the mkdir command usage along with the control arguments by running anyone of the following command in the Linux terminal. man mkdir and mkdir --help .

The output of man mkdir command in Linux

You can create a parent directory along with the child directory at once by using the -p control argument along with the mkdir command in Linux.

mkdir -p  retail/orders/2022/08/27
-p             Create intermediate directories as required.  If this option is not specified, the full path prefix of each operand must already exist.  On the other hand, with this option specified, no error will be reported if a directory given as an operand already exists.  Intermediate directories are created with permission bits of “rwxrwxrwx” (0777) as modified by the current umask, plus write and search permission for the owner.

Create a hierarchy of directories using mkdir command in Linux

You can create multiple folders at once using the following command pattern.

mkdir -p album{1..4}/artist{1..3}

Create multiple folders and files at once using mkdir command

You can remove all the folders which start with the album by running the following command in Linux.

rm -rf album*

How to use mkdir command to create multiple hierarchies of folders and files.

Overview of awk command in Linux

You can get the complete usage of the awk command by running any one of the following commands in the Linux terminal. awk --help or man awk

The awk is pattern-directed scanning and processing language. Awk scans each input file for lines that match any of a set of patterns specified literally in prog or in one or more files specified as -f progfile. With each pattern, there can be an associated action that will be performed when a line of a file matches the pattern. Each line is matched against the pattern portion of every pattern-action statement; the associated action is performed for each matched pattern.

man awk command to get the details of awk in Linux terminal.

For practice, you can clone the following repository from GitHub.

Clone the above repository for the data to practice.

You can use the awk command to get the specific column values from the CSV file.

awk -F"," '{ print $4}' part-00000 | sort | uniq
The -F fs option defines the input field separator to be the regular expression fs.

get the data from a specific column using the awk command from CSV file.

You can print the output however you want by using the awk command for example check out the below command.

awk -F"," '{ print "Order status is " $4}' part-00000 | sort | uniq

Print the output in the requested format using the awk command.

You can also pipe the output of cat to awk command to get similar results as above.

cat part-00000 | awk -F"," '{ print "Order status is " $4}' | sort | uniq

pipe the output of cat command to awk command.

A detailed explanation of the awk command in Linux.

Overview of cp command to copy files in Linux

The Linux cp command is used to copy the contents of source_file to the target_file. You can get the complete details of the cp command by running any one of the following commands man cp or cp --help

For the hands-on practice of cp command, You can create multiple files and a folder by using the commands shown in the below image.👇🏻

Create multiple files and a folder for hands-on practice.

Detailed overview of the copy command in Linux

Using the cp command to copy files in Linux

Now let's create a /tmp/demo folder and copy the file from the /data/demo directory to /tmp/demo directory using cp command.

In the below image you can clearly see the step-by-step process of copying the files from one location to another using cp command.👇🏻

You can clear the /temp/demo folder using one command. rm -rf /tmp/demo/*

remove multiple files at once using the rm command in Linux.

A detailed explanation of copying files using the cp command in Linux.

Create multiple folders at once using awk in Linux

If you run ls -1 you will get the files present in the present working directory as shown below👇🏻

Get the files in the present working directory using ls -l

You can use the following commands piped to each other to get the unique album names from the above file names.

ls -1 | awk -F"_" '{ print $1}' | sort | uniq

piping ls command to awk command to get the file names in Linux

Now you can print the output in the required format by using print.

ls -1 | awk -F"_" '{ print "/tmp/demo/albums/" $1 }' | sort | uniq

Print the files in the required format using the print inside awk command.

Now, You can pass mkdir -p command to create the albums in the /tmp/demo/albums/ directory. You need to pipe bash at the end for the command to work.

ls -1 | awk -F"_" '{ print "mkdir -p /tmp/demo/albums/" $1 }' | sort | uniq | bash

Create multiple folders at once using the awk command with mkdir in Linux

A detailed explanation of how to use the awk command with mkdir command to create folders.

Automate file copy using awk and cp in Linux

The usual way to copy the files from one directory to another is by using cp command.

cp album1_song1.mp4 /tmp/demo/albums/album1

But, By using the awk command, you can copy multiple files to the destination folder at once. The below command is an example to show how to generate the required pattern of cp commands.

ls -1 | awk -F"_" '{ print "cp " $0 " " $1 }'

Generate the pattern of cp commands by using the awk command

ls -1 | awk -F"_" '{print "cp " $0 " /tmp/demo/albums/" $1 "/" $2}'

Generate the complete pattern of cp commands using the awk command to copy multiple files at once

Now, You can pipe this output to the bash to execute the awk command successfully.

ls -1 | awk -F"_" '{print "cp " $0 " /tmp/demo/albums/" $1 "/" $2}' | bash

Validate whether files were copied successfully or not by using the find command

A detailed explanation of using the awk command to automate the bulk file copy process.

Copying folders recursively using the cp command in Linux

From the previous section, you can observe the albums in /tmp/demo/albums the directory. You can recursively copy the folder and files in it at a stretch using cp a command with a control argument -r .

Albums we created in the previous sections using the awk command in Linux

Now you can copy albums to the present working directory which is /data/demo using the following command at a stretch.

cp -r /tmp/demo/albums .

Albums folder copied recursively to the current directory using cp command in Linux

You can use find albums to validate whether every file is copied from source to destination. find albums

Check all the files copied from source to destination in time.

A detailed explanation of how to use copy command recursively in Linux.

Copying files using cp while preserving properties in Linux

You can preserve the timestamp of the file or folder which is copied from source to destination by using the control argument -p along with the cp command in Linux.

cp -rp [source][destination]
-p    Cause cp to preserve the following attributes of each source file in the copy: modification time, access time, file flags, file mode, user ID, and group ID, as allowed by permissions.

You can go through the below video to understand the complete details of how to preserve the properties of the file while copying in Linux.

A detailed explanation of copying files using cp while preserving properties in Linux.

Moving Files in Linux using mv command

The mv command is used to move the files from the source directory to the destination directory in the Linux file system. When you use mv command the file will be removed from the source and moved to the destination directory.

You can get the complete usage of the mv command by typing mv --help or man mv

man mv command to get the details of the mv command in Linux

Lets create a folder like/tmp/albums and move the files from /data/demo to /tmp/albums/ You can create the folder by running mkdir /tmp/albums

Move the file from source to destination using mkdir

You can rename the files using mv command. Check out the below screenshot for an example.👇🏻

renaming the file using the mv command in Linux

You can move multiple files at once from the source directory to the destination directory using a pattern along with mv command in Linux.

mv album2_*.mp4 /tmp/albums

You can move multiple files at once using a pattern with the mv command in Linux

Whereas if you rename multiple files at once then the mv command is not sufficient. You need to use the awk command to generate the pattern of the mv command. You can come up with a solution by taking the reference of the “Automate file copy using awk and cp in Linux” section in this article.

A detailed explanation of how to use mv command in Linux.

Automate file move using awk and mv in Linux

You can generate the pattern of mv commands using awk command in Linux to move multiple files from one directory to another directory.

First let's create multiple folders at /tmp/albums/ directory using awk command

ls -1 | awk -F"_" '{print "mkdir -p /tmp/albums/" $1 }' | sort | uniq | bash

Create multiple folders using the awk command

Now, You can move multiple files from /data/demo to the destination directories /tmp/albums/album1 and /tmp/albums/album2with awk command as shown below.

ls -1 | awk -F"_" '{print "mv " $0 " /tmp/albums/" $1 "/" $2 }' | sort | uniq | bash

How to move multiple files at once using awk command in Linux?

A detailed explanation of using the awk command in Linux to automate the file move.

🙏🏼Thank you, for reading the article. If you find it valuable please follow our publication DevOps Engineering on Cloud

🚨👉🏼 You can also check the complete udemy course (Linux Shell Commands for Absolute Beginners using Ubuntu 20x)🔗Referral link