Filesizes

ls -lrt | awk '{ total += $5 }; END { print total }'

Rsync

Options

-v : verbose

-r : copies data recursively (but don’t preserve timestamps and permission while transferring data

-a : archive mode, archive mode allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships and timestamps

-z : compress file data

-h : human-readable, output numbers in a human-readable format

Copy a Directory from Local Server to a Remote Server

rsync -avz localdir/ user@remotehost:/remotedir

Copy/Sync a Remote Directory to a Local Machine

rsync -avzh user@remotehost:/remotedir /localdir

See more examples at: Techmint Rsync examples

Find

Basic construction:

find <dir> <something>

Useful options

-mtime (modification time); e.g -1 is one day ago

        can be between with -mtime -1  -mtime +2

-name '*.txt'

-type -[d,f] ; f=file, d=directory

-L follows symbolic links

-exec ls -l {} \;

-empty; find files of zero length or empty directories

Combining commands

-name '*txt' -exec grep “something” {} \; 

equivalent to

find -name ‘*txt’ | xargs grep “something”

The CEDA command line arrivals deleter

arrivals_deleter

Cut

Linux command cut is used for text processing. You can use this command to extract portion of text from a file by selecting columns.

Select Column of Characters

$ cut -c2 test.txt

Select Column of Characters using Range

$ cut -c1-3 test.txt

Select Column of Characters using either Start or End Position

$ cut -c3- test.txt

$ cut -c-8 test.txt

$ cut -c- test.txt

Select a Specific Field from a File

$ cut -d':' -f1 /etc/passwd

Select Multiple Fields from a File

$ grep "/bin/bash" /etc/passwd | cut -d':' -f1,6

$ grep "/bin/bash" /etc/passwd | cut -d':' -f1-4,6,7

Select Fields Only When a Line Contains the Delimiter

$ grep "/bin/bash" /etc/passwd | cut -d'|'  -f1

$ grep "/bin/bash" /etc/passwd | cut -d'|' -s -f1

Select All Fields Except the Specified Fields

$ grep "/bin/bash" /etc/passwd | cut -d':' --complement -s -f7

Change Output Delimiter for Display

grep "/bin/bash" /etc/passwd | cut -d':'  -s -f1,6,7 --output-delimiter='#'

Change Output Delimiter to Newline

$ grep bala /etc/passwd | cut -d':' -f1,6,7 --output-delimiter=$'\n'

Combine Cut with Other Unix Command Output

ps axu | grep python | sed 's/\s\+/ /g' | cut -d' ' -f2,11-

http://www.thegeekstuff.com/2013/06/cut-command-examples/

Permissions

setuid bit

• in chmod: octal value 4000, or symbolic `u+s`
• effect: on an executable file; runs with the uid that owns the file rather than the uid of the invoking process; on a directory, no effect
• in `ls -l` output: shows as `s` in the user execute bit column if the user execute bit is also set, otherwise shown as `S` in that column

setgid bit

• in chmod: octal value 2000, or symbolic `g+s`
• effect: on an executable file; runs with the gid that owns the file rather than the gid of the invoking process; on a directory, files created in the directory get given the same group ownership as the directory
• in `ls -l` output: shows as `s` in the group execute bit column if the group execute bit is also set, otherwise shown as `S` in that column

sticky bit

• in chmod: octal value 1000, or symbolic `+t`
• effect: on a file, no effect; on a directory, files in the directory can only be deleted/renamed by the owner of the file or by the owner of the directory or by root (allows users with write permission on the directory to create files without interfering with other users' files - commonly set on `/tmp`)
• in `ls -l` output: shows as `t` in the "other" execute bit column if the "other" execute bit is also set, otherwise shown as `T` in that column

(Thanks to Alan Iwi for this information)

Doing more

Running nohup with redirect of stdout

nohup ./{script} > {filename}.out 2>&1&

Data transfer when passwordless log in not working

scp {data} {username@loc:/destination} 

Stop process: CTRL+z 

Set to run in background: bg {Process_ID} 

Release process from session: disown -h {Process_ID}

Removing characters from strings

Remove from end of string

a=test01.nc

b=${a%nc}

echo b

=> test01.

To remove from start of string use #

Incremental loops

for (( y=1; y<=10; y++)); do 

 ....

done

Counter loops

Loops using alphabet and typeset to set length of a counter string

$ typeset -Z2 counter

$ counter=00

$ for ens in {a..z}; do 

$ ((counter=counter+1))

$ echo $counter

$ echo $ens

$ done

Passing a file of arguemnts to a shell script

cat {argsfile.txt} | xargs {script.sh}

Symbolic links

Create a symbolic link with the command

$ ln -s <source> <linkname>

source is the directory to link to

linkname is the symbolic link name to the source

Latex compilations

dvips {filename}.dvi

ps2pdf {filename}.ps {filename}.pdf

Printer options

lp -d{PRINTER} -o {filename} (-o fit to page)

Software

Python: setup epd 

CDO: setup cdo 

Copying files

cp 

cp -p: preserves permissions.

scp

rsync -a(v)zhe ssh <data> <destination>

Commands Alphabetically

U USB

usbcheck

usbeject

L LaTeX

latex <filename>

bibtex <filename>

xdvi <filename>

dvips <file>.dvi

ps2pdf <file>.ps <file>.pdf

P Printing

a2ps <file> | lp -d<printer>

xpp

rlpstat

rcancel

NetCDF

ncdump -c <file> | more

ncdump -h <file>

Tarring and Zipping

CREATE: tar cf <file>.tar <files>

RESTORE: tar xf <file>.tar

CREATE: tar -xcvf <file>.tgz <files>

RESTORE: tar -zxvf <file>.tgz

S Searching

grep

whence

P Processes

ps -u <userid>

ps kill [-9]

bg: &, CTRL+z "bg"

fg: "fg" 

F Filesizes

ls -l

ls -ltr

ls -lq

ls -lh

ls -F (directories)

ls -F <dir> look in dir

ls -sh (size in h converts to human readable)

du -sh */

df -h