Unix/Linux in an Hour

Notes

First a note about notation:

• • All commands for you to type are presented as:

$ type these commands

result of that command

Where the $ indicates that this is to be typed into your terminal. When you are logged into quest your command line prompt will look like:

netid@quser03 mycurrentdirectory $

• • Following a typed command will sometimes be another line or two which should be the results of your previous command.

Logging In

The first step in using our machines is being able to log in to them.

Windows

First, you’ll need an SSH client program, such as PuTTy. Bodeen already has both PuTTy and a Windows SSH client ("SSH Secure Shell") installed.

If you are using PuTTy:

• • Open PuTTy!

  • In the "Host Name (or IP address)" field, type MyNetID@quest.northwestern.edu

  • Make sure the port is 22

  • Make sure connection type is "SSH"

If you are using SSH Secure Shell:

• • Open SSH Secure Shell!

  • In "Host Name" enter: quest.it.northwestern.edu

  • In "User Name" enter: MyNetID

Either way, you will be prompted for a password. Your password will be the same as your NetID password for Caesar, Canvas, etc.

Mac / Linux

Much easier: just open a Terminal window, then type:

$ ssh mynetid@quest.it.northwestern.edu

You will be prompted for a password. Your password will be the same as your NetID password for Caesar, Canvas, etc.

When you log into Quest for the first time, you may be prompted for a default email address. Enter your email address and press enter. You may also be told that you do not have an ssh key set up yet. Your screen should look something like this:

It doesn't appear that you have set up your ssh key.

This process will make the files:

/home/netid/.ssh/id_rsa.pub

/home/<netid/.ssh/id_rsa

/home/netid/.ssh/authorized_keys

Generating public/private rsa key pair.

Enter file in which to save the key (/home/netid/.ssh/id_rsa):

ssh keys can be used to login to Quest without a password. However, they require you to connect from the same computer every time, and are not worth our trouble right now. So, press enter several times until you see your command prompt:

netid@quser02 ~ $

Basic Unix Usage

The first thing you probably wondered is, where did my slick interface with the Start Menu and Taskbar and Recycle Bin go? You don’t get them here; you have to navigate everything from the command line. Never fear! A typical Unix root file system (located at /) like the one on quest looks like this:

Because you are not a system administrator (a stroke of luck for you, I promise), you will be living and playing exclusively in the home directory, where all your files will be saved. In fact, when you log in, you are taken to the directory /home/yourusername by .default.

Navigating Directories

Let’s start by making an example folder called "wahoo". To do that, type (mkdir = "make directory")

$ mkdir wahoo

Now, how do I see that directory? (ls = "list")

$ ls

wahoo

You should see your new folder. Fine, but how do I get into it? (cd = "change directory")

$ cd wahoo

Now what is in this directory?

$ ls

Nothing! Because we are in a new, and empty, directory. Very boring, so let’s make a file in “wahoo.”

$ touch boring

Now list directory contents again.

$ ls

boring

You can copy this file:

$ cp boring stillboring

$ ls

boring stillboring

or move it (there is no such thing as renaming a file in linux; the operation is called moving. "moving" is slightly different from "renaming"):

$ mv stillboring yawn

$ ls

boring yawn

If you want to go up one folder in the file system hierarchy,

$ cd ..

Also, at any time, if you want to go to the root of your home directory (even plain "cd" works the same way),

$ cd ~

Also, at any time, if you want to go back to the directory you were in:

$ cd -

You can also type the exact path of where you want to go.

$ cd ~/wahoo

This command, when issued from any location on the machine will take you to the wahoo folder in your home directory.

Another very important tool is Unix’s “auto-complete” feature. Say you had a folder called verylongfoldernameyoudontwanttotype. You could get there by typing $ cd verylo

(or however much of the directory name was required to distinguish it from the other files in the current directory) and then pressing the <TAB> key. Try this with your directory named 'wahoo'

$ cd ~/wah

Now press <TAB>. Taadaa!

Manipulating Files

Even in the well-controlled sandbox theory, things go wrong more often than right. We like to say “Garbage in, garbage out.” You will want to clean up the garbage once in a while. Say you want to delete the wahoo folder, because your WAHOO code simulation didn’t quite work out. Try:

$ rm wahoo

Tricked you: it didn’t work! You can delete files that way (but not directories):

$ rm wahoo/boring

This will wipe out the boring file. However, to delete directories, you need to modify the rm command with the –r flag, which stands for recursive (meaning it deletes everything in that directory permanently):

$ rm –r wahoo

Be very careful with rm. There’s no Recycle Bin/Trash here from which you can recover your files. A deletion is permanent!

Deleting individual files and folders can get tedious when you really have a lot of garbage to get rid of. Do:

$ touch asdf1

$ touch asdf2

$ touch asdf3

Now, you could delete these creatively named files one at a time, or you can use a wildcard in your rm command:

$ rm asdf*

File Editing

Unfortunately, none of the codes we will be using will provide you with a beautiful interface in which you can click the “calculate my final project” button. They are all text- and command line-based. Unix has several well-known text editors, and the one we will be explaining is vi (but feel free to use emacs, nano, etc. if you’re more comfortable with them). To get an example file to play with, do

$ mkdir pwscf

$ cp /projects/e20438/intro_to_linux/cu_test.pw.in pwscf/

$ cd pwscf

This is an example input file for the PWSCF DFT code we’ll be using later in the quarter. To see it in the vi text editor,

$ vi cu_

and then press TAB to use that cool autocomplete we learned earlier!

In vi you can move around the file by using the up and down arrow keys. That suffices for small files, but for big ones you could be there all day. You will also want to use:

<CTRL>f - go forward one page

<CTRL>b - go backwards one page

<SHIFT>g - go to the bottom of the file

gg - go to the top of the file

:7 - go to line number 7

$ - go to the end of the current line

0 or ^ - go to the beginning of the current line (ignoring leading white-space)

w - go forward one word

b - go backwards one word

/wordtolookfor - search file for wordtolookfor

n - go to next instance of wordtolookfor

<SHIFT>n - go to previous instance of wordtolookfor

u - undo changes

<CTRL>r - redo changes

That’s all well and good, but what if you want to edit the file? By default, when you open something in vi, you’re in “NORMAL” mode and you can’t make changes. Pressing “i” puts you in INSERT mode where you can make changes. NOTE! In INSERT mode, all the commands above don’t work; they will type characters and edit your file instead! To get out of insert mode and go back to navigation mode, just press <ESC>.

Now, edit the line:

"celldm(1) = 6.73,"

so that it says:

"celldm(1) = 6.75,"

to change the lattice parameter we would be using for Cu. How do you save the file? In the NORMAL mode,(note, the ":" brings up the command prompt in vi, and must be pressed before any of the following commands can be issued)

:w - writes changes to the file.

:q - quits.

:wq - saves and quits.

What if you mess up? It’s easy to do in vi; if you start trying to enter commands while in edit mode you can wipe out lots of stuff really fast. Just quit without saving with

:q! - quits without saving

You can find much more information about vi through vi's built in help system. You can access the help system by typing :help . The help documentation contains tons of information, the most useful to you might be the quickreference and the tutorial. To exit the help system and return to editing your file, type :q.

If you want to know more about how vi/vim works and/or practice commands/editing/etc. once you login to Quest, do

$ vimtutor

This opens up a vim tutorial that goes through most useful basic features of vim. Vim is an *extremely* powerful editor once you learn its tricks and trades. Going through the entire tutorial should take you only 30 minutes or so and is highly recommended!

Sometimes, you would probably only want to look at what is inside the file without requiring to edit it. You could do

$ less cu_test.pw.in

to see the contents of a file, navigate through it using arrow keys (or Vi navigation keys!) and use "q" to quit.

To see only the top "N" lines of a file, do

$ head -n N cu_test.pw.in

Similarly, to see the last "N" lines of a file, do

$ tail -n N cu_test.pw.in

Submitting Jobs

To run the codes you’ll use in this course, you need to submit requests for processing time on quests compute nodes. If resources are available, your job will run immediately; If not, you will have to wait. The way you tell our computers you have something for them to calculate is with a .q file. Get an example with

$ cp /projects/e20438/intro_to_linux/cu_test.q ~/pwscf

$ cd pwscf

$ vi cu_test.q

The important aspects of this .q file are:

#SBATCH -N 1

#SBATCH -p short

#SBATCH --ntasks-per-node=2

#SBATCH -t 2:00:00

#SBATCH -A e20438

#SBATCH --job-name="cu_test.q"

Here you request the number of nodes (N) to use; the queue type (short); how many CPUs you want to run on (ntasks-per-node, with the maximum of 6); the maximum time this job should be allowed to run (hours:minutes:seconds) and what allocation this run should be charged to (e20438; you don't ever need to change this line for this class unless you intend to use your own/your group's allocation on Quest); and the name of your job.

NOTE: you can only set walltimes that are so long. Take a look here: https://kb.northwestern.edu/quest-partitions-queues to understand what walltimes can be set for what number of cores and what priority each case would have (remember that a lower priority number means a higher priority!).

In general, going through the "Getting Started" section here: https://kb.northwestern.edu/page.php?id=70706 is very useful and highly recommended!

mpirun -np $NPROCS pw.x < cu_test.pw.in

This is where you tell the computer what program you’re running and what input file it should use. The bolded text should never needs to be changed. The underlined text is the part that you will edit to change what you want the job to do. Here: - "pw.x" is the name of the executable that needs to be run

- "<" tells the executable that the name of the input file follows it (this "redirection" is executable dependent! may not always work)

- "cu_test.pw.in" is the input file for the executable

In addition, the following could have been added to the line to redirect the output:

- ">" redirects the standard output from the executable (not the standard error; take a look here to learn more: https://en.wikipedia.org/wiki/Standard_streams) into the name of the file following it

- "cu_test.pw.out" is the name of the output file

So, the line would look something like this:

mpirun -np $NPROCS pw.x < cu_test.pw.in > cu_test.pw.out

To submit the job, you would do:

$ sbatch cu_test.q

When your job is successfully submitted, usually the job scheduler prints the ID of the submitted job, e.g., 186554 or some number like that (useful to track jobs when you have many in the queue/running).

To see a list of (everyone’s) jobs running and waiting,

$ squeue

To see only yours, try

$ squeue -u yournetid

or

$ qstat -u yournetid

To see a list of all the jobs running in our class, try

$ squeue -A e20438

To delete one of your jobs (either running or waiting),

$ scancel jobnumber

If you don't redirect the output to a file inside your batch script (using "> [output_filename]"; see above) the default name of the output file is usually "[name_of_the_batch_script].o.[job_ID]", e.g., "cu_test.q.o.186554" for the example above.

Congratulations! You're now an expert on using a Linux machine!

“You’ve taken your first step into a larger world”

--Obi-Wan Kenobi