There a two types of manipulation commands

Redirection

This is how we combine a command with a file
example:

• cat file.txt > file_copy.txt : copy file.txt content into file_copy.txt
• ls / > directory.txt : create a new file with list of content of / directory

Pipeline

Combine two or more commands in a single line
example:

• ls /home/host/ | grep {keyword_1} | grep {keyword_2}: list all files or folder which are matched with those keywords

GREP

Grep is used for filtering files or folder based on a keyword.
basic usage :

# display all line matched
grep {keyword} {file_path}

# display all line matched
grep {keyword} {file_1} {file_2} {file_3}

# ignore keyword case
grep -i {keyword} {file_path}

# display lines without the matched line
grep -v {keyword} {file_path}

# display with line number
grep -n {keyword} {file_path}

# display a count of matching lines
grep -c {keyword} {file_path}

# display matched line for either keyword 1 or keyword 2
grep -w "keyword_1 | keyword_2" {file_path}

# display 4 lines after the keyword
grep -A {number} {keyword}

# display 4 lines before the keyword
grep -B {number} {keyword}

REGEX

We can also combine grep with regular expression. Regex is case sensitive
basic usage :

grep -E "REGEX" {file_path}

This command will display all lines which are matched with the regex pattern.

note : will talk about regex in another story

SED

Sed is Stream Editor. Sed can be used for searching a file, find and replace, insertion or deletion. We can also use sed for quickly editing a file without opening it. Sed also support regular expression pattern.

• we use s for substitution and / for the delimiter

# replace first unix word in each line with linux
sed 's/unix/linux/' {file_path}

# replace all unix word in each line with linux
sed 's/unix/linux/g' {file_path}

# replace second unix word in each line with linux
sed 's/unix/linux/2' {file_path}

# replace with case insensitive
sed 's/unix/linux/i' {file_path}

# replace all unix word except in fifth line
sed '5!s/unix/linux/g' {file_path}

# replace unix word from second and next line
sed 's/unix/linux/2g' {file_path}

# replace unix word in third line
sed '3 s/unix/linux/' {file_path}

# replace unix word in first line until third line
sed '1,3 s/unix/linux/' {file_path}

# replace unix word in start from third line
sed '3,$ s/unix/linux/' {file_path}

# display only replaced lines
sed -n 's/unix/linux/p' {file_path}

# display replaced lines twice
sed 's/unix/linux/p' {file_path}

• d for deletion

Sed also can be used for deleting a line.

# delete fifth line
sed '5d' {file_path}

# delete last line
sed '$d' {file_path}

# delete third line until sixth line
sed '3,6d' {file_path}

# delete from third line and next
sed '3,$d' {file_path}

# delete line with matched pattern
sed '/pattern/d' {file_path}

# delete line with matched case insensitive pattern
sed -i '/pattern/d' {file_path}

# delete line with matched pattern and two lines after
sed '/abc/,+2d' {file_path}

# delete every second line, start from third line
sed '3~2d' {file_path}

# delete blank line
sed '/^$/d' {file_path}

# delete blank line w
sed -i '/^#/d;/^$/d' {file_path}

• G for file spacing

# insert one blank line after each line
sed G {file_path}

# insert two blank lines after each line
sed 'G:G' {file_path}

# insert a black line above every line which matches pattern
sed '/pattern/{x;p;x;}' {file_path}

# insert a blank line below every line which matches pattern
sed '/pattern/G' {file_path}

• -n p for displaying a file

# display a file
sed -n p {file_path}

# display only fourth line
sed -n '4p' {file_path}

# display from second line to fifth line
sed -n '2,5p' {file_path}

# display only last line
sed -n '$p' {file_path}

# display from third line and next
sed -n '3,$p' {file_path}

# display entire file except second line to fourth line
sed -n '2,4d' {file_path}

# display only line with matched pattern
sed -n '/abc/p' {file_path}

# matched pattern start from fifth line
sed -n '/abc/,5p' {file_path}

# display from second line to matched pattern line
sed -n '2,/abc/p' {file_path}

• = for numbering lines

# add a number in front of each line
sed = {file_path}

CUT

As the name suggests, cut i used for cutting out the section from each line of file and writing the result to a new file. We can use this command to cut parts of a line by some of these commands :

• -b or --bytes=LIST

Select by specifying a byte, a set of bytes, or a range of bytes.

# display first, second, and third byte of each line
cut -b '1,2,3' {file_path}

# display first until third byte and fifth until seventh byte
cut -b '1-3,5-7' {file_path}

# display third and next byte
cut -b '3-' {file_path}

# display until third byte
cut -b '-3' {file_path}

• -c or --characters=LIST

Select by specifying a character, a set of characters, or a range of characters.

# display first, second, and third characters of each line
cut -c '1,2,3' {file_path}

# display first until third characters and fifth until seventh characters
cut -c '1-3,5-7' {file_path}

# display third and next characters
cut -c '3-' {file_path}

# display until third characters
cut -c '-3' {file_path}

• -f or --fields=LIST

Select by specifying a field, a set of fields, or a range of fields. This command needs specified delimiter written with -d or --delimiter{DELIMITER} .

# display all fields without specified delimiter
cut -f '1' {file_path}

# display first until third fields, with comma delimiter
cut -d\, -f '1-3' {file_path}

# display from third fields, with comma delimiter
cut -d\, -f '3-' {file_path}

# display until third fields, with comma delimiter
cut -d\, -f '-3' {file_path}

• --complement

Displays all bytes, characters, or fields except the selected.

# display all bytes except the selected
cut --complement -b '1' {file_path}

# display all characters except the selected
cut --complement -c '1' {file_path}

# display all fields except the selected
cut --complement -d\, -f '1' {file_path}

• -s or --only-delimiter

Do not print lines not containing delimiters.

# display only fields which have delimiter
cut -d\, -f '1-4' {file_path} -s

• --output-delimiter=STRING

Modify the output delimiter.

# display fields with # delimiter
cut -d\, -f '1-4' {file_path} --output-delimiter="#"

SORT

This command will do as the name implies.

# sort the lines ascending
sort {file_path}

# sort the lines descending
sort -r {file_path}

# sort and create a new file
sort -o {output_file} {original_file}

# sort the numbering lines in ascending
sort -n {file_path}

# sort the numbering lines in descending
sort -nr {file_path}

# sort and remove duplicates
sort -u {file_path}

# checks if a file is already sorted and return unsorted lines (if any)
sort -c {file_path}

# sort a table by the nth column
sort -k2 {file_path}

# sort by month
sort -M {file_path}

UNIQ

We can delete the duplicated lines with this command. Uniq basic command will return the same result with sort -u .

# print only uniq line
uniq {file_path}

# print a number how many times a line was repeated
uniq -c {file_path}

# print repeated lines, only display one duplicated line per group
uniq -d {file_path}

# print repeated lines, display all duplicated lines
uniq -D {file_path}

# prints only the unique lines
uniq -u {file_path}

# skip nth field to be skipped while comparing uniqueness
uniq -f 2 {file_path}

# skip nth character to be skipped while comparing uniqueness
uniq -s 2 {file_path}

# limit the comparison to a set number of characters 
uniq -w 2 {file_path}

# make the comparison case-insensitive
uniq -i {file_path}

TR

We can use this command to translate or delete characters.

# translate from lowercase with uppercase
cat {file_path} | tr "[a-z]" "[A-Z]"

# translate from lowercase with uppercase
cat {file_path} | tr "[:lower:]" "[:upper:]"

# translate whitespace with tab
cat {file_path} | tr "[:space:]" "\t"

# translate brackets with curly barckets
cat {file_path} | tr "()" "{}"

# translate multiple whitespace with single space
cat {file_path} | tr -s "[:space:]" " "

# delete specific A character
cat {file_path} | tr -d "A"

# delete digit character
cat {file_path} | tr -d "[:digit:]"

# delete all except digit
cat {file_path} | tr -cd "[:digit:]"

AWK

Awk is used for manipulating data and generating reports.
Fields identifier :

• $0 : entire line of text
• $1 : first field
• $2 : second field
• $6 : sixth field
• $NF : stands for “number of fields,” and represents the last field
• NR : display number of fields
• OFS : output field separator
• BEGIN : begin of the file
• END : end of the file

# view file
awk '{print}' {file_path}

# view specific lines
awk '{print $2,$3,$6}' {file_path}

# view last field each lines
awk '{print $NF}' {file_path}

# view number of fields each line
awk '{print NR}' {file_path}

# print specific lines with / separator
awk 'OFS="/" {print $2,$3,$6}' {file_path}

# add text in the beginning of the file
awk 'BEGIN {print "text"} {print $2,$3,$6}' {file_path}

# add text in the ending of the file
awk 'END {print "text"} {print $2,$3,$6}' {file_path}

# print the lines which is matched with the given pattern
awk '/pattern/' {file_path}

# print specific field after matched pattern each line 
awk '/pattern/ {print $2,$3,$6}' {file_path}

Linux is closely related to the command line. Even though some people prefer to use graphic user interface (GUI), we will using command-line interface (CLI). CLI is more effective and powerful. Some process can be done in one line of commands in seconds.

These some basic commands that I use often.

1. Navigation

• pwd : print working directory
• cd : change directory
  - cd : go to home
  - cd {dir_path} : go to directory path
  - cd .. : on directory above
  - cd - : back to last directory path
• ls : list
  - ls -a : list all (include hidden file and folder)
  - ls -l : long list
  - ls -la : long list all

2. Directory / file management

• touch : create a new file
• mkdir : create a new folder
• rm : remove
  - rm {file_name} : delete a file
  - rm -r {dir_name} : delete a folder
• rmdir : remove directory, only allows to delete empty directory
• cp : copy
  - cp {old_file} {new_file} : copy a file
  - cp -r {old_dir} {new_dir} : copy a folder and all sub folder
• mv : move
  - mv {old_file_path} {new_file_path} : move a file or folder
  - mv {old_file_name} {new_file_name} : rename a file or folder
• locate {file_name} : search a file or folder
  - locate -i {file_name} : search file with case sensitive
• find {dir_path} {file_name} : search a file or folder inside a directory
• tar : archive multiple files into a tarball
• zip : compress files into a zip archive
• unzip : extract the zipped files

3. File view and editing

• nano : view and edit with nano
• vim : view and edit with vim
• cat : view file
  - cat > {file_name} : create a new file
  - cat {file_1} > {file_2} : copy file 1to file 2
  - cat {file_1} >> {file_2} : append file 1 content to file 2
• grep {keyword} {file_name} : search through all the text in a given file
• head : view the first lines of any text file, by default it will showing first 10 lines
  - head -n 5 {file_name} : display first 5 lines of the file
• tail : view the last lines of any text file, by default it will showing first 10 lines
  - tail -n 5 {file_name} : display last 5 lines of the file

4. Manual guide

• man : manual
  - man cp : display full guide for copy command
• --help : help
  - cp --help : display short guide for copy command

5. System commands

• sudo : SuperUser Do, enables to perform tasks that require administrative or root permissions
• command1; command2; command3 : run multiple commands
• command1 && command2 && command3 : run the next commands after the first one is successful
• useradd {username} : add a new user
  - passwd {password} : set the password
• userdel : delete a user
• chmod {options} {file_name} : change the read, write, and execute permissions of files and directories
  - chmod u=rwx,g=rx,o=r {file_name}
  - u : user
  - g : group
  - o : other
  - r : read
  - w : write
  - x : execute
• chown {username} {file_name} : change or transfer the ownership of a file
• uname : print detailed information about your Linux system
• hostname : display the name of host/network
  - hostname -i : display the IP address of the network
• wget {link} : download from the internet
• ps : display information about the currently running processes, including their process identification numbers (PIDs)
• kill {pid} : terminate an unresponsive program
• df : get a report on the system’s disk space usage
  - df -h : display disk space statistics
  - df -m : display disk space statistics in megabytes
• du -h : display disk usage
• free -m → show free memory
• cat /proc/cpuinfo → display cpu info
• htop / top → monitor system resources
• history : review the commands we’ve entered before
• clear : clean out the terminal

6. Keyboard shourcut

• ctrl + C : safe kill
• ctrl + Z : sometimes there are child processes that still running in the background
• ctrl + A : moves to the beginning of the line
• ctrl + E : moves to the end of the line

Source:

• Unix cheat sheet → http://www.mathcs.emory.edu/~valerie/courses/fall10/155/resources/unix_cheatsheet.html

Let’s create a basic configuration of our Vagrantfile, instead of only use vagrant init.

• Default Vagrantfile if we run vagrant init <boxpath>

• Set minimum vagrant version

Vagrant.require_version ">= 1.3.5", "< 1.4.0"

This will help toavoid compatibility issues that may arise from using different version of vagrant.
We can specify the version that the Vagrantfile will only run if we use greater than equal version 1.3.5 and below 1.4.0.

• Configures provisioners

config.vm.provision "shell", inline: "apt-get update && apt-get install -y  docker.io "

or

config.vm.provision "shell", inline: <<-SHELL
    apt-get update
    apt-get install -y  docker.io
SHELL

We can automatically run a code to install and configure something when the virtual machine is created.
In this case, we update the package list and install docker.

• Define a Virtual Machine

config.vm.define "ubuntu1804" do |ubuntu1804|
    # ...
end

We can define a name for our VM, instead of use default if we don’t define it.

• Configure the machine

config.vm.box = "ubuntu/bionic64"

We can specify our machine we want to run. We can find available linux machine here.

• Set autoupdate for every vagrant up

config.vm.box_check_update = false

The default value is true, so every time we run the VM, it will check for updates.

• Create a forwarded port

# enable public access
config.vm.network "forwarded_port", guest: 80, host: 8080

# disable public access
config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"

We can configure the networks in the machine. If we use config.vm.network we have to specify two ports, first one is for guess and the second one is host port that the guest port can be accessed by.
By default, it will automatically enable the public access. If we want to disable public access, we need to specify the host_ip that only allow access via 127.0.0.1.

• Create a private network

config.vm.network "private_network", ip: "192.168.33.10"

We can also specify the private IP for the machine that allow us to access the VM using a static IP.

• Create a bridge network

config.vm.network :public_network, :bridge => "en0: Wi-Fi (Wireless)"

The default network connection for vagrant is NAT. We can update into bridge which is allow the machine to connect to other vm and the host, also we can use host only which only allow communication between the machin and the host.

• Crate share a folder between VM and host

config.vm.synced_folder "HOST_FOLDER_PATH", "VM_FOLDER_PATH"

We can create a sync folder that allow us to copy pasting a file between the machine and the host. We need to specify the host folder path and vm folder path , then we can easily transfer file between that machines.

• Set provider configuration

config.vm.provider "virtualbox" do |vb|
    vb.memory = "2048"
    vb.cpus = "1"
end

If we need more memory and cpu, we can configure the memory and cpus we want to use in the VM.

• Configure disk size

config.disksize.size = '25GB'

We also can configure disk size that we want to use for the machine.

Source

• Vagrant public boxpath → https://app.vagrantup.com/boxes/search
• Vagrantfile docs → https://www.vagrantup.com/docs/vagrantfile/

Hi, we will run through how I run linux in my machine.
I will use brew for the installation.

1. Make sure brew is installed

brew --version

if there is no brew installed, let’s just add it to your machine.

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

2. Install VirtualBox

Simply run this command to add VirtualBox in your machine

brew install cask virtual box

Then check the installation using this command

vboxmanage --version

3. Install Vagrant

Next, we will need Vagrant to help us to run and getting in to the linux machine

brew install vagrant

Then check the installation

vagrant --version

4. Let’s start the linux machine

These are the main things to run linux

• Creating a Virtual Machine

vagrant init <boxpath>

You can choose linux version that you want. go to this source here.
In case we want to use ubuntu1804, we can type vagrant init ubuntu/bionic64 .
This command will generate a Vagrantfile for us.

• Starting a Virtual Machine

vagrant up

• Getting into a Virtual Machine

vagrant ssh <boxname>

In this case, we only use default Vagrantfile from vagrant init <boxpath> instead of set the box name. We can simply run vagrant ssh to get into out VM. Welcome to linux machine !!

• Stoping a Virtual Machine

vagrant halt

This will stop the VM. We can simply run vagrant resume to start the VM again

• Cleaning up a Virtual Machine

vagrant destroy

This command will delete the VM, and we need to run vagrant up to create the VM.

Source:

• Vagrant cheatsheet → https://gist.github.com/wpscholar/a49594e2e2b918f4d0c4
• Vagrant public boxpath → https://app.vagrantup.com/boxes/search