Unix Shell Commands

Resources

• • CWRU HPC Orientation Course Materials Fall 2018

  • Working with Linux

  • Google Cloud Shell

• Advanced Bash Script Guide

• Software Carpentry

• Ryan's Linux tutorial

• 37 Important Linux Commands You Should Know

Terminology

Unix commands are given in a so-called shell; the most common shell is the bash shell. The shell is simply a collection of commands for talking to the computer. Each bash command is a way to communicate with the computer; since you can mix and match commands, you can talk to the computer in more innovative ways than if you have a graphical user interface. The GUI has to define a subset of ways in which you can interact with the computer; on the commandline, you have more freedom. Of course you have to have some knowledge to make use of that freedom in a useful way!

• In unix, folders are called directories. The sum of all the directories and files is called the file system.

• The root of all directories is / (forward slash). All directories on a computer are contained within /.

• Subdirectories are separated by / (forward slash) -- for instance /usr/local/bin or /home/rooster.

• Directories can be empty or contain files and other directories.

In unix, you're always located somewhere in the file system. The unix prompt on cartago always tells you where you are -- so for instance it might say

rooster@cartago:/db/tv/2015/2015-10/2015-10-02$

That tells you the name of your unix user (rooster), the machine you are on (cartago), and your present location in the file system (/db/tv/2015/2015-10/2015-10-02). (On your local laptop, the prompt may not be configured to give this information.) When you change directories, you "move" within the file system, as it were walking from one location to another. It's important to mentally track which location you are in at any moment -- and also which machine you are on!

Basic Unix Commands Tutorial 1

Unix Shell Commands Lesson

Welcome to your fist foray into unix shell commands. In this exercise, you will be introduced to the ten most common unix commands. Once you’ve mastered these commands, you will move on to use more complicated ones to navigate and manipulate data from the NewsScape archive. But first, you must learn how to create new directories and files, and how to navigate through them.

The first thing you must know is that you will be "walking into" (ie. entering) and "out of" (ie. exiting) directories as you issue commands for them. Thus, you must think of the command line as both a portal to a particular part of an operating system, and as a building block for computational creativity. So, in order to effectively navigate through and extract the information you are searching for, you must always remember where you are.

Part of this is recognizing the difference between files and directories. Directories house files and sub-directories, forming trees. Files contain data. For our purposes today, this data is text.

So, the commands you'll be using are as follows

• ls — list files and directories (LiSt); by default, always use ls -l (to see more details)

• cp — copy a file from one directory to another (CoPy)

• echo — place text into a file (ECHOs what you type)

• mv — rename or move a directory or file (MoVe)

• cat — show contents of a file (conCATenate)

• mkdir — make a directory (MaKe DIRectory)

• cd — walk into a directory (Change Directory)

• rm — remove a file (ReMove) -- be careful so you don't remove valuable files!

• rmdir — delete a directory (ReMove DIRectory)

• man — help on a command (MANual)

Because we will be creating a tree to illustrate the hierarchical nature of Unix, take care to name your directories and files, and write your text with a theme. The bracketed examples are simply suggestions -- add your own sense of humor to it. So when you see a phrase enclosed in <brackets>, that means you should substitute the expression with your own. For example, mkdir <some directory> could be entered in your terminal as

mkdir dinosaur

Keep it simple, and you'll soon get the hang of how to talk to unix!

List Files

Begin by orienting yourself. Where are you? To answer this question, you must issue

ls

This command lists all files and directories in your current location.

To tell files and directories apart, and to see permissions and the age of the files, use

ls -l

roosters-MacBook-Air:~ yard_ley$ ls -l

total 136

-rw-r-----@ 1 yard_ley staff 68900 Jan 21 2015 Appearance_Release_Sonos_Synesthesia_Workshop_CLIFFORD.pdf

drwx------ 4 yard_ley staff 136 Oct 25 02:00 Applications

drwx------@ 3 yard_ley staff 102 Aug 30 13:47 Creative Cloud Files

drwx------@ 3 yard_ley staff 102 Jul 27 17:43 Creative Cloud Files (1)

drwx------@ 3 yard_ley staff 102 Feb 23 2015 Creative Cloud Files (unknown)

drwx------+ 137 yard_ley staff 4658 Oct 25 17:47 Desktop

drwx------+ 41 yard_ley staff 1394 Aug 12 07:29 Documents

drwx------+ 593 yard_ley staff 20162 Oct 25 17:47 Downloads

drwx------@ 54 yard_ley staff 1836 Mar 17 2015 Library

drwx------+ 5 yard_ley staff 170 Mar 17 2015 Movies

drwx------+ 7 yard_ley staff 238 Sep 17 21:23 Music

drwx------+ 7 yard_ley staff 238 Sep 20 18:44 Pictures

drwxr-xr-x+ 5 yard_ley staff 170 Dec 25 2014 Public

• Note: The '-l' in ls -l (LiSt Long) is called a 'flag' or 'switch' -- note this is a lower-case L and not a digit 1. Unix commands come with different switches like that, which lets you modify exactly how they work and what they give you. For details, see Unix shell commands #Command reference.

Remember, directories house files. Files never house directories. Instead, files contain programs or data, either as binary executables compiled from source code or as text. Take a look at the diagram below. This will give you an idea of what we will be building.

Next, Create a Directory using

mkdir <directory_name>

roosters-MacBook-Air:~ yard_ley$ mkdir Cats

Avoid using spaces in a directory name (though underscore is fine) -- use short but informative words.

Create a File

roosters-MacBook-Air:~ yard_ley$ echo "Christmas for Cats" > Catnip

And Fill it with some text

roosters-MacBook-Air:~ yard_ley$ echo "A type of Mint" >> Catnip

Check your work by Showing the Contents of the File

roosters-MacBook-Air:~ yard_ley$ cat Catnip

Christmas for Cats

A type of Mint

Navigate to a New Directory and check where you are by issuing the default command:

roosters-MacBook-Air:~ yard_ley$ ls -l

total 144

-rw-r-----@ 1 yard_ley staff 68900 Jan 21 2015 Appearance_Release_Sonos_Synesthesia_Workshop_CLIFFORD.pdf

drwx------ 4 yard_ley staff 136 Oct 25 02:00 Applications

-rw-r--r-- 1 yard_ley staff 34 Oct 25 18:09 Catnip

drwxr-xr-x 2 yard_ley staff 68 Oct 25 18:03 Cats

drwx------@ 3 yard_ley staff 102 Aug 30 13:47 Creative Cloud Files

drwx------@ 3 yard_ley staff 102 Jul 27 17:43 Creative Cloud Files (1)

drwx------@ 3 yard_ley staff 102 Feb 23 2015 Creative Cloud Files (unknown)

drwx------+ 137 yard_ley staff 4658 Oct 25 17:47 Desktop

drwx------+ 41 yard_ley staff 1394 Aug 12 07:29 Documents

drwx------+ 593 yard_ley staff 20162 Oct 25 17:47 Downloads

drwx------@ 54 yard_ley staff 1836 Mar 17 2015 Library

drwx------+ 5 yard_ley staff 170 Mar 17 2015 Movies

drwx------+ 7 yard_ley staff 238 Sep 17 21:23 Music

drwx------+ 7 yard_ley staff 238 Sep 20 18:44 Pictures

drwxr-xr-x+ 5 yard_ley staff 170 Dec 25 2014 Public

Place a Directory within Another Directory

roosters-MacBook-Air:~ yard_ley$ mv Catnip Cats/

roosters-MacBook-Air:~ yard_ley$ cd ~/Cats

roosters-MacBook-Air:Cats yard_ley$ ls -l

total 8

-rw-r--r-- 1 yard_ley staff 34 Oct 25 18:09 Catnip

roosters-MacBook-Air:Cats yard_ley$ mv Catnip Nepeta_cataria

roosters-MacBook-Air:Cats yard_ley$ ls -l

total 8

-rw-r--r-- 1 yard_ley staff 34 Oct 25 18:09 Nepeta_cataria

Create three more files and two more subdirectories. Continue to name them thematically, so they'd make sense together if this were a family tree, or a game of associations. This will help you remember how all the directories, subdirectories, files, and text fit together in Unix.

roosters-MacBook-Air:Cats yard_ley$ mkdir Famous

roosters-MacBook-Air:Cats yard_ley$ ls -l

total 8

drwxr-xr-x 2 yard_ley staff 68 Oct 25 18:20 Famous

-rw-r--r-- 1 yard_ley staff 34 Oct 25 18:09 Nepeta_cataria

roosters-MacBook-Air:Cats yard_ley$ cd

roosters-MacBook-Air:~ yard_ley$ mdkir in_Literature

-bash: mdkir: command not found

roosters-MacBook-Air:~ yard_ley$ mkdir in_Literature

roosters-MacBook-Air:~ yard_ley$ mv in_Literature Cats/

roosters-MacBook-Air:~ yard_ley$ cd ~/Cats

roosters-MacBook-Air:Cats yard_ley$ ls -l

total 8

drwxr-xr-x 2 yard_ley staff 68 Oct 25 18:20 Famous

-rw-r--r-- 1 yard_ley staff 34 Oct 25 18:09 Nepeta_cataria

drwxr-xr-x 2 yard_ley staff 68 Oct 25 18:21 in_Literature

Mistakes to avoid:

roosters-MacBook-Air:Cats yard_ley$ mv in_Literature Famous/

roosters-MacBook-Air:Cats yard_ley$ cd ~/Famous

-bash: cd: /Users/yard_ley/Famous: No such file or directory

roosters-MacBook-Air:Cats yard_ley$ cd /Famous

-bash: cd: /Famous: No such file or directory

roosters-MacBook-Air:Cats yard_ley$ cat Famous

cat: Famous: Is a directory

roosters-MacBook-Air:Cats yard_ley$ cd ~/

roosters-MacBook-Air:~ yard_ley$ cd ~/Famous

-bash: cd: /Users/yard_ley/Famous: No such file or directory

roosters-MacBook-Air:~ yard_ley$ cd ~/in_Literature

-bash: cd: /Users/yard_ley/in_Literature: No such file or directory

roosters-MacBook-Air:~ yard_ley$ cd ~/Cats

roosters-MacBook-Air:Cats yard_ley$ ls -l

total 8

drwxr-xr-x 3 yard_ley staff 102 Oct 25 18:21 Famous

-rw-r--r-- 1 yard_ley staff 34 Oct 25 18:09 Nepeta_cataria

roosters-MacBook-Air:Cats yard_ley$ cat Famous

cat: Famous: Is a directory

roosters-MacBook-Air:Cats yard_ley$ cat Famous/

cat: Famous/: Is a directory

roosters-MacBook-Air:Cats yard_ley$ ~/Famous

-bash: /Users/yard_ley/Famous: No such file or directory

roosters-MacBook-Air:Cats yard_ley$

Finally, navigate to your home directory, ~/ . List your files. What difference do you see from when you started?

Conclusion

Try adding on the command specifications below to the basic commands you just learned. Play with it. Create files and copy them from directory to directory. Make note of your mistakes throughout, and how you deal with any roadblocks you may encounter. We will be discussing them line-by-line in class.

Commands you may find useful:

ls -l ~/my_files

To list files inside all of the present subdirectories, called my_files, issue

ls -l */*

The asterixes mean 'any directory and any file' -- this is called pattern matching.

Command reference

Basic unix commands with usage examples; expand as needed.

ls: list files

To list files and directories with no further information, issue

ls

It's typically much more useful to show the file permissions and the date and time of the file, so normally you should use

ls -l

The '-l' is called a 'flag' or 'switch' -- note this is a lower-case L and not a digit 1. Unix commands come with different switches like that, which lets you modify exactly how they work and what they give you.

Let's say you have a subdirectory in your home directory called 'my_files'. To see its contents,

ls -l ~/my_files

To list files inside all of the present subdirectories, called my_files, issue

ls -l */*

The asterisks mean 'any directory and any file' -- this is called pattern matching.

To see fuller information for all files in a directory

ls -al

This will show permissions, owner, group, etc., and do so even for files and directories that begin with a period, such as .ssh.

chmod: set permissions

chmod XXXX filename

So for instance

chmod 764 somefile

gives the owner permission to read (4), write (2), and execute (1) the file: 4+2+1=7; gives members of the group permission to read and write but not execute: 4+2=6; and gives everyone with an account on the machine permission to read the file: 4-4.

For a fuller tutorial, see issue "man chmod" or see the Linux permissions tutorial. For a discussion of SGID, SUID, etc., see A Deeper Dive into Linux Permissions.

echo: put text in a file

Echo is used to add text to a file, using the > or redirect character:

echo "some text" > some_file

So for instance, this command will put text into a new file called 'buttercup':

echo "What you get if you fill a cup with butter." > buttercup

A single > will overwrite any previous content in the file. If you want to append the echoed text to the file, use double chevrons:

echo "Or maybe a cup that is sculpted out of butter?" >> buttercup

Use cat (see next) to verify you now have both texts in the file.

cat: show the contents of a file

cat <filename>

Check that the text you wrote is in the file you created:

cat buttercup

mkdir: Create a directory

mkdir <directory name>

So for instance

mkdir flowers

will create a directory called 'flowers'.

cd: change directory

To walk from one directory to another, issue

cd <directory>

Your home directory is the default, so you can just type this to get home:

cd

Your home directory is called ~/ so you can use this to reach directories inside your home directory:

cd ~/flowers

If your prompt doesn't tell you where you are in the file system, always use 'pwd' to tell you! It's very important you don't lose track of where you are.

cp: copy files

cp <filename> <new filename>

So for instance you can copy a file in your home directory to a subdirectory:

cp buttercup flowers/

Now you have two copies of the buttercup file -- one in your home directory and on in the subdirectory flowers. Can you list them both?

mv: move a file

You can use 'mv' to rename a file like this:

mv <filename> <new filename>

For instance,

mv buttercup cupofbutter.txt

Or you can move a file from one directory to another:

mv <filename> <target directory>

For instance,

mv cupofbutter.txt flowers/

To make sure that the target is a directory and not a file, you can include that final forward slash.

You can use 'mv' to rename a directory, like this:

mv <old directory name> <new directory name>

For instance,

mv flowers baking

Or you can move a directory into another directory:

mv <old directory name> <existing directory name>

For instance,

mkdir foods

mv baking foods

The directory 'baking', which contains the file 'cupofbutter.txt' if you followed the examples above, has now been moved into the subdirectory 'foods'.

rm: delete a file

rm <filename>

rmdir: delete a directory

rmdir <directory name>

man: get help on a command

man <command>

Editing a file: vi or nano. Some Red Hens use vi; some use nano. You pick!

vi (or, if you are on a RaspberryPi, vim): edit a file

See one of these sites:

• https://ryanstutorials.net/linuxtutorial/vi.php

• http://glaciated.org/vi/

nano: edit a file

• On your laptop, copy some text into your paste buffter (mark it and press Ctrl-c)

• In Terminal, type "nano practise.txt" and press Return

• Press Ctrl-v to paste the text you copied into nano

• Press Ctrl-o, type the chosen file name, and press Enter to save the file

• Make changes to the text and save it

You can get help on how to use nano here: http://www.howtogeek.com/howto/42980/the-beginners-guide-to-nano-the-linux-command-line-text-editor/

You should know the commands listed here: http://mintaka.sdsu.edu/reu/nano.html

grep: search inside files

To look for something inside one or several files, you can use grep:

grep <filename>

For instance, to examine all the Named Entities in the annotated CNN files for one day, issue

grep '|NER' *CNN*seg

for loop: loop through files

To examine each file separately, you can use a for loop (note the use of backticks -- they are not regular apostrophes):

for FIL in `ls -1 *CNN*seg` ; do echo $FIL ; sleep 1 ; done

This simply outputs the name of each file, with a one-second pause between each name. If you want individual outputs for each file that you search, you can use the for loop like this:

for FIL in `ls -1 *CNN*seg` ; do echo $FIL ; grep '|POS' *CNN*seg ; sleep 1 ; done

This will output each file name, followed by all of the parts-of-speech tags in that file, with a one-second pause between each file. Note how the semicolon is used to separate each command.

String manipulation

In the default bash shell, you also have some additional functionality, such as string manipulation. You can for instance use a string from the left to create file names with the date of the directory you are working in:

grep '|NER' *CNN*seg > ~/${PWD##*/}_CNN_NER.txt

To see how chopping works, try issuing this in any directory:

echo $PWD

echo ${PWD##*/}

The variable $PWD is a so-called environmental variable, which contains local information about your context, in this case the name of the present working directory -- the same as if you issue

pwd

Then ${PWD##*/} chops off the path, leaving just the present directory name. You can also save the output -- say of all frames in a file -- to an identically named file with a new extension, as in this for loop:

for FIL in `ls -1 *CNN*seg` ; do grep '|FRM' $FIL > ~/${FIL%.*}.frames ; done

where ${FIL%.*} chops the extension off of $FIL.

sed: stream editor

The stream editor sed is very powerful at manipulating strings on the fly. Working with sed might be an extra credit task; it's not part of basic operation. See http://www.grymoire.com/Unix/Sed.html

Networked tools

We'll also learn some tools for talking to remote computers, such as ssh, scp, rsync, traceroute, host, and whois.