OS/Subprocess Python Module
As a DevOps/System Admin everyday we encounter a task where we need to interact with OS. In this tutorial I am going to tell about some important Python module which we can use for this purpose
- os
- subprocess
OS module allow us to interact with underlying operating system in different ways
#Import the OS module
>>> import os#Let see the methods that we have access to within this module
>>> print(dir(os))[‘CLD_CONTINUED’, ‘CLD_DUMPED’, ‘CLD_EXITED’, ‘CLD_TRAPPED’, ‘DirEntry’, ‘EX_CANTCREAT’, ‘EX_CONFIG’, ‘EX_DATAERR’, ‘EX_IOERR’, ‘EX_NOHOST’, ‘EX_NOINPUT’, ‘EX_NOPERM’, ‘EX_NOUSER’, ‘EX_OK’, ‘EX_OSERR’, ‘EX_OSFILE’, ‘EX_PROTOCOL’, ‘EX_SOFTWARE’, ‘EX_TEMPFAIL’, ‘EX_UNAVAILABLE’, ‘EX_USAGE’, ‘F_LOCK’, ‘F_OK’, ‘F_TEST’, ‘F_TLOCK’, ‘F_ULOCK’, ‘MutableMapping’, ‘NGROUPS_MAX’, ‘O_ACCMODE’, ‘O_APPEND’, ‘O_ASYNC’, ‘O_CREAT’, ‘O_DIRECT’, ‘O_DIRECTORY’, ‘O_DSYNC’, ‘O_EXCL’, ‘O_LARGEFILE’, ‘O_NDELAY’, ‘O_NOATIME’, ‘O_NOCTTY’, ‘O_NOFOLLOW’, ‘O_NONBLOCK’, ‘O_RDONLY’, ‘O_RDWR’, ‘O_RSYNC’, ‘O_SYNC’, ‘O_TRUNC’, ‘O_WRONLY’, ‘POSIX_FADV_DONTNEED’, ‘POSIX_FADV_NOREUSE’, ‘POSIX_FADV_NORMAL’, ‘POSIX_FADV_RANDOM’, ‘POSIX_FADV_SEQUENTIAL’, ‘POSIX_FADV_WILLNEED’, ‘PRIO_PGRP’, ‘PRIO_PROCESS’, ‘PRIO_USER’, ‘P_ALL’, ‘P_NOWAIT’, ‘P_NOWAITO’, ‘P_PGID’, ‘P_PID’, ‘P_WAIT’, ‘PathLike’, ‘RTLD_DEEPBIND’, ‘RTLD_GLOBAL’, ‘RTLD_LAZY’, ‘RTLD_LOCAL’, ‘RTLD_NODELETE’, ‘RTLD_NOLOAD’, ‘RTLD_NOW’, ‘R_OK’, ‘SCHED_BATCH’, ‘SCHED_FIFO’, ‘SCHED_OTHER’, ‘SCHED_RR’, ‘SEEK_CUR’, ‘SEEK_END’, ‘SEEK_SET’, ‘ST_APPEND’, ‘ST_MANDLOCK’, ‘ST_NOATIME’, ‘ST_NODEV’, ‘ST_NODIRATIME’, ‘ST_NOEXEC’, ‘ST_NOSUID’, ‘ST_RDONLY’, ‘ST_SYNCHRONOUS’, ‘ST_WRITE’, ‘TMP_MAX’, ‘WCONTINUED’, ‘WCOREDUMP’, ‘WEXITED’, ‘WEXITSTATUS’, ‘WIFCONTINUED’, ‘WIFEXITED’, ‘WIFSIGNALED’, ‘WIFSTOPPED’, ‘WNOHANG’, ‘WNOWAIT’, ‘WSTOPPED’, ‘WSTOPSIG’, ‘WTERMSIG’, ‘WUNTRACED’, ‘W_OK’, ‘XATTR_CREATE’, ‘XATTR_REPLACE’, ‘XATTR_SIZE_MAX’, ‘X_OK’, ‘_Environ’, ‘__all__’, ‘__builtins__’, ‘__cached__’, ‘__doc__’, ‘__file__’, ‘__loader__’, ‘__name__’, ‘__package__’, ‘__spec__’, ‘_execvpe’, ‘_exists’, ‘_exit’, ‘_fspath’, ‘_fwalk’, ‘_get_exports_list’, ‘_putenv’, ‘_spawnvef’, ‘_unsetenv’, ‘_wrap_close’, ‘abc’, ‘abort’, ‘access’, ‘altsep’, ‘chdir’, ‘chmod’, ‘chown’, ‘chroot’, ‘close’, ‘closerange’, ‘confstr’, ‘confstr_names’, ‘cpu_count’, ‘ctermid’, ‘curdir’, ‘defpath’, ‘device_encoding’, ‘devnull’, ‘dup’, ‘dup2’, ‘environ’, ‘environb’, ‘errno’, ‘error’, ‘execl’, ‘execle’, ‘execlp’, ‘execlpe’, ‘execv’, ‘execve’, ‘execvp’, ‘execvpe’, ‘extsep’, ‘fchdir’, ‘fchmod’, ‘fchown’, ‘fdatasync’, ‘fdopen’, ‘fork’, ‘forkpty’, ‘fpathconf’, ‘fsdecode’, ‘fsencode’, ‘fspath’, ‘fstat’, ‘fstatvfs’, ‘fsync’, ‘ftruncate’, ‘fwalk’, ‘get_blocking’, ‘get_exec_path’, ‘get_inheritable’, ‘get_terminal_size’, ‘getcwd’, ‘getcwdb’, ‘getegid’, ‘getenv’, ‘getenvb’, ‘geteuid’, ‘getgid’, ‘getgrouplist’, ‘getgroups’, ‘getloadavg’, ‘getlogin’, ‘getpgid’, ‘getpgrp’, ‘getpid’, ‘getppid’, ‘getpriority’, ‘getresgid’, ‘getresuid’, ‘getsid’, ‘getuid’, ‘getxattr’, ‘initgroups’, ‘isatty’, ‘kill’, ‘killpg’, ‘lchown’, ‘linesep’, ‘link’, ‘listdir’, ‘listxattr’, ‘lockf’, ‘lseek’, ‘lstat’, ‘major’, ‘makedev’, ‘makedirs’, ‘minor’, ‘mkdir’, ‘mkfifo’, ‘mknod’, ‘name’, ‘nice’, ‘open’, ‘openpty’, ‘pardir’, ‘path’, ‘pathconf’, ‘pathconf_names’, ‘pathsep’, ‘pipe’, ‘popen’, ‘posix_fadvise’, ‘posix_fallocate’, ‘pread’, ‘putenv’, ‘pwrite’, ‘read’, ‘readlink’, ‘readv’, ‘remove’, ‘removedirs’, ‘removexattr’, ‘rename’, ‘renames’, ‘replace’, ‘rmdir’, ‘scandir’, ‘sched_get_priority_max’, ‘sched_get_priority_min’, ‘sched_getparam’, ‘sched_getscheduler’, ‘sched_param’, ‘sched_rr_get_interval’, ‘sched_setparam’, ‘sched_setscheduler’, ‘sched_yield’, ‘sendfile’, ‘sep’, ‘set_blocking’, ‘set_inheritable’, ‘setegid’, ‘seteuid’, ‘setgid’, ‘setgroups’, ‘setpgid’, ‘setpgrp’, ‘setpriority’, ‘setregid’, ‘setresgid’, ‘setresuid’, ‘setreuid’, ‘setsid’, ‘setuid’, ‘setxattr’, ‘spawnl’, ‘spawnle’, ‘spawnlp’, ‘spawnlpe’, ‘spawnv’, ‘spawnve’, ‘spawnvp’, ‘spawnvpe’, ‘st’, ‘stat’, ‘stat_float_times’, ‘stat_result’, ‘statvfs’, ‘statvfs_result’, ‘strerror’, ‘supports_bytes_environ’, ‘supports_dir_fd’, ‘supports_effective_ids’, ‘supports_fd’, ‘supports_follow_symlinks’, ‘symlink’, ‘sync’, ‘sys’, ‘sysconf’, ‘sysconf_names’, ‘system’, ‘tcgetpgrp’, ‘tcsetpgrp’, ‘terminal_size’, ‘times’, ‘times_result’, ‘truncate’, ‘ttyname’, ‘umask’, ‘uname’, ‘uname_result’, ‘unlink’, ‘unsetenv’, ‘urandom’, ‘utime’, ‘wait’, ‘wait3’, ‘wait4’, ‘waitid’, ‘waitid_result’, ‘waitpid’, ‘walk’, ‘write’, ‘writev’]
To print current working directory
>>> print(os.getcwd())/root
To change path/current working directory
>>> os.chdir(“/tmp”)>>> print(os.getcwd())/tmp
To print/list files in current directory(it return a list)
>>> print(os.listdir())[‘.bash_logout’, ‘.bash_profile’, ‘.bashrc’, ‘.cshrc’, ‘.tcshrc’, ‘original-ks.cfg’, ‘anaconda-ks.cfg’, ‘.ssh’, ‘.bash_history’, ‘.pki’, ‘.cache’, ‘.aws’, ‘.lesshst’, ‘.viminfo’,‘anaconda3’, ‘.bashrc-anaconda3.bak’, ‘.python_history’, ‘.config’]
To create directories
>>> os.mkdir(“test”)>>> print(os.listdir())[‘.bash_logout’, ‘.bash_profile’, ‘.bashrc’, ‘.cshrc’, ‘.tcshrc’, ‘original-ks.cfg’, ‘anaconda-ks.cfg’, ‘.ssh’, ‘.bash_history’, ‘.pki’, ‘.cache’, ‘.aws’, ‘.lesshst’, ‘.viminfo’, ‘pandas_for_everyone’, ‘anaconda3’, ‘.bashrc-anaconda3.bak’, ‘.python_history’, ‘.config’, ‘test’]
But now let say we want to go one level deep i.e both test3 and test4 doesn’t exist, if we are trying to make directory it will not work,to do that we need to use makedirs
>>> os.mkdir(“test3/test4”)Traceback (most recent call last):File “<stdin>”, line 1, in <module>FileNotFoundError: [Errno 2] No such file or directory: ‘test3/test4’>>> os.makedirs(“test1/test2”)
Same way if we want to remove directory(rmdir) or directory inside directory(removedirs)
>>> os.rmdir(“test5”)
>>> os.removedirs("test1/test2")
NOTE: As removing is a destructive operation please be-cautious before using removedirs
To rename a file
>>> os.listdir()[‘.bash_logout’, ‘.bash_profile’, ‘.bashrc’, ‘.cshrc’, ‘.tcshrc’, ‘original-ks.cfg’, ‘anaconda-ks.cfg’, ‘.ssh’, ‘.bash_history’, ‘.pki’, ‘.cache’, ‘.aws’, ‘.lesshst’, ‘.viminfo’, ‘pandas_for_everyone’, ‘anaconda3’, ‘.bashrc-anaconda3.bak’, ‘.python_history’, ‘.config’, ‘test’]>>> os.rename(‘test’,’test1')>>> os.listdir()[‘.bash_logout’, ‘.bash_profile’, ‘.bashrc’, ‘.cshrc’, ‘.tcshrc’, ‘original-ks.cfg’, ‘anaconda-ks.cfg’, ‘.ssh’, ‘.bash_history’, ‘.pki’, ‘.cache’, ‘.aws’, ‘.lesshst’, ‘.viminfo’, ‘pandas_for_everyone’, ‘anaconda3’, ‘.bashrc-anaconda3.bak’, ‘.python_history’, ‘.config’, ‘test1’]
To check if the path exist
>>> os.path.exists(“/etc/resolv.conf”)True>>> os.path.exists(“/etc/resolvvv.conf”)False
Same way we can check the presence of file/directory
>>> os.path.isfile(“/etc/resolv.conf”)True>>> os.path.isdir(“/etc/resolv.conf”)False
To print stats
>>> os.stat(“test1”)os.stat_result(st_mode=33188, st_ino=4194982, st_dev=51714, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1492269055, st_mtime=1492269055, st_ctime=1492269074)
To print directory tree and all the files within that directory tree we can use os.walk which is a generator and has a tuple of 3 values
- directories(dirpath)
- directories within that path(dirname)
- files within that path
>>> for dirpath,dirnames,filenames in os.walk("/test1"):... print(dirpath)... print(dirnames)... print(filenames).../test1 # First it goes to toplevel directory['test2'] # Then all the directories inside it[] # All the files below it/test1/test2[][]
OR Better way to write the same program
>>> for dirpath, dirnames, filenames in os.walk(“/etc”):
… print(“Files in %s are: “ % dirpath)
… for file in filenames:
… print(“\t” + file)
Output
Files in /etc are:
afpovertcp.cfg
aliases
aliases.dbFiles in /etc/apache2 are:
httpd.conf
httpd.conf.pre-update
magic
mime.types
For Directories
>>> for dirpath, dirnames, filenames in os.walk(“/etc”):
… print(“Directories in %s are:” % dirpath)
… for dir in dirnames:
… print(“\t” + dir)
…
Output
Directories in /etc are:
apache2
asl
cups
defaults
emond.d
mach_init.d
mach_init_per_login_session.d
mach_init_per_user.d
manpaths.d
newsyslog.d
openldap
pam.d
paths.d
periodic
pf.anchors
postfix
ppp
profile.d
racoon
security
snmp
sshDirectories in /etc/apache2 are:
extra
original
other
users
To print all the environment variables
>>> os.environenviron({‘HOSTNAME’: ‘ip-172–31–33–6.us-west-2.compute.internal’, ‘SHELL’: ‘/bin/bash’, ‘TERM’: ‘xterm-256color’, ‘HISTSIZE’: ‘1000’, ‘USER’: ‘root’, ‘LS_COLORS’: ‘rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:’, ‘SUDO_USER’: ‘ec2-user’, ‘SUDO_UID’: ‘1000’, ‘USERNAME’: ‘root’, ‘PATH’: ‘/root/anaconda3/bin:/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin’, ‘MAIL’: ‘/var/spool/mail/root’, ‘PWD’: ‘/root’, ‘LANG’: ‘en_US.UTF-8’, ‘HISTCONTROL’: ‘ignoredups’, ‘SHLVL’: ‘1’, ‘SUDO_COMMAND’: ‘/bin/bash’, ‘HOME’: ‘/root’, ‘LOGNAME’: ‘root’, ‘LESSOPEN’: ‘||/usr/bin/lesspipe.sh %s’, ‘SUDO_GID’: ‘1000’, ‘_’: ‘/root/anaconda3/bin/python3’})
If we are looking for any specific value
>>> os.environ.get(‘LOGNAME’)‘root
To join two path together
>>> a = os.path.join(“/tmp”, “testhome”)>>> print(a)/tmp/testhome
Join is specially helpful while reading the content of the file
>>> import os>>> filename = os.path.join(“/root”,”test1")>>> mode = “r”>>> with open(filename,mode) as f:… print(f.read())…reading the content of the file>>> f.close()
To get the process id
>>> os.getpid()12462
Simple Directory listing Program
>>> for file in os.listdir(“.”):
… info = os.stat(file)
… print(“%-20s : size %d” % (file, info.st_size))
Output
animals.py : size 195
batteries.py : size 38
continue.py : size 128
deepreverse.py : size 222
distance.py : size 360
evenodd.py : size 151
fib.py : size 147
findminmax.py : size 441
firstclass.py : size 598
For more info please refer
Subprocess Module
Please be careful while executing shell command via Python
As per doc “the subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to replace several older modules and functions”
os.system
os.spawn*
To list all files and directories
#Subprocess always accept list as argument>>> subprocess.call([“ls”,”-l”])total 20drwxr-xr-x 20 root root 268 Mar 26 14:39 anaconda3-rw — — — -. 1 root root 7712 Oct 20 13:05 anaconda-ks.cfg-rw — — — -. 1 root root 6927 Oct 20 13:05 original-ks.cfgdrwxr-xr-x 4 root root 80 Mar 26 14:26 pandas_for_everyone-rw-r — r — 1 root root 32 Apr 15 11:57 test10#If we try to run subprocess without passing list>>> subprocess.call("ls -l")Traceback (most recent call last):File "<stdin>", line 1, in <module>File "/root/anaconda3/lib/python3.6/subprocess.py", line 267, in callwith Popen(*popenargs, **kwargs) as p:File "/root/anaconda3/lib/python3.6/subprocess.py", line 707, in __init__restore_signals, start_new_session)File "/root/anaconda3/lib/python3.6/subprocess.py", line 1326, in _execute_childraise child_exception_type(errno_num, err_msg)FileNotFoundError: [Errno 2] No such file or directory: 'ls -l'#The correct way is>>> subprocess.call(["df","-h")]Filesystem Size Used Avail Use% Mounted on/dev/xvda2 10G 4.4G 5.7G 44% /devtmpfs 477M 0 477M 0% /devtmpfs 496M 0 496M 0% /dev/shmtmpfs 496M 63M 434M 13% /runtmpfs 496M 0 496M 0% /sys/fs/cgrouptmpfs 100M 0 100M 0% /run/user/10000
Now to save the output to some variable use check_output
>>> x = subprocess.check_output([“ls”,”-l”])>>> print(x)b’total 20\ndrwxr-xr-x 20 root root 268 Mar 26 14:39 anaconda3\n-rw — — — -. 1 root root 7712 Oct 20 13:05 anaconda-ks.cfg\n-rw — — — -. 1 root root 6927 Oct 20 13:05 original-ks.cfg\ndrwxr-xr-x 4 root root 80 Mar 26 14:26 pandas_for_everyone\n-rw-r — r — 1 root root 32 Apr 15 11:57 test1\n’