bash

CUT command

grep X321 MS*txt | cut -f1 | cut -c17-

grep "/bin/bash" /etc/passwd | cut -d':' -f1-4,6,7

http://www.segv.me/ - linux/pyhon/postgresql notes

http://habrahabr.ru/post/260335/ load expained

http://www.ghacks.net/2009/04/04/get-to-know-linux-the-etcinitd-directory/

'chkconfig' allows the checking the startup level of services and gives the ability to set the service. Running 'chkconfig' will setup the scripts in the needed "rcX.d" directory.

other command of use:

/sbin/service "stop|start|restart|etc"

this command is the same as of typing "/etc/init.d/ ....

Recursive grep -r : Example ignore case and show only file names:

grep -ril needle folder

http://wiki.bash-hackers.org/commands/builtin/read

http://www.cyberciti.biz/faq/unix-linux-bash-split-string-into-array/

Reading line by line and splitting line using read ( another option is cut)

#!/bin/bash

FILE=$1

while read line; do

echo "This is a line : $line"

read testrunid fname <<< $line

echo $fname

done < $FILE

filename="$1"while read -r line do name=$line echo "Name read from file - $name"done < "$filename"

cat a b | sort | uniq > c # c is a union b cat a b | sort | uniq -d > c # c is a intersect b cat a b b | sort | uniq -u > c # c is set difference a - b

• • Summing all numbers in the third column of a text file (this is probably 3X faster and 3X less code than equivalent Python):

awk '{ x += $3 } END { print x }' myfile

select from the middle of the file

sed -n '10000000,+20p' filename

sed -n '10000000,10000020p' filename

tail -n 10000 print the last 10000 lines.

tail -n +10000 start print at 10000th line

head -n 10000010 filename | tail 10

timeout - kill command if it runs longer then given time http://ss64.com/bash/timeout.html

Color diff site-by-site

http://www.jefftk.com/icdiff

http://habrahabr.ru/post/221273/ bash tips from Google

For example, to determine whether a particular regular expression regex was present somewhere in a file options, we might apply grep(1) with its POSIX -q option to suppress output and just use the exit value:

grep -q regex options

An approach sometimes taken is then to test the exit value with the $?parameter, using if to check if it’s non-zero, which is not very elegant and a bit hard to read:

# Bad practice grep -q regex options if (($? > 0)); then printf '%s\n' 'myscript: Pattern not found!' >&2 exit 1 fi

Because the if construct by design tests the exit value of commands, it’s better to test the command directly, making the expansion of $? unnecessary:

# Better if grep -q regex options; then # Do nothing : else printf '%s\n' 'myscript: Pattern not found!\n' >&2 exit 1 fi

We can precede the command to be tested with ! to negate the test as well, to prevent us having to use else as well:

# Best if ! grep -q regex options; then printf '%s\n' 'myscript: Pattern not found!' >&2 exit 1 fi

ls does not all to hide direcories, by find does:

find FIX -maxdepth 1 -type f -not -name 't_*'

delete empty files

find . -type f -size 0 -delete

System monitoring scripts

http://devo.ps/blog/2013/03/06/troubleshooting-5minutes-on-a-yet-unknown-box.html

https://github.com/BashtonLtd/whatswrong/blob/master/whatswrong

http://www.fduran.com/blog/quick-linux-server-review-for-mortals/

http://habrahabr.ru/company/selectel/blog/221143/

https://github.com/rory/SystemAutopsy

http://word.bitly.com/post/74839060954/ten-things-to-monitor?h=2

bash best practice

https://news.ycombinator.com/item?id=7815190

https://news.ycombinator.com/item?id=7981633

http://habrahabr.ru/post/221273/

http://yellerapp.com/posts/2014-06-03-bash-for-ci.html

http://redsymbol.net/articles/unofficial-bash-strict-mode/

chroot

Disable bell in bash: add the following line in ~/.inputrc :

set bell-style none

~/.bashrc

http://www.pixelbeat.org/programming/shell_script_mistakes.html

http://bash.cumulonim.biz/BashPitfalls.html

http://blog.existentialize.com/dont-pipe-to-your-shell.html

http://jeroenjanssens.com/2013/09/19/seven-command-line-tools-for-data-science.html

https://news.ycombinator.com/item?id=6412190

http://www.reddit.com/r/programming/comments/1pixp1/use_multiple_cpu_cores_with_your_linux_commands/

https://news.ycombinator.com/item?id=6360320

http://www.leancrew.com/all-this/2011/12/more-shell-less-egg/

http://tldp.org/LDP/abs/html/

http://voxel.dl.sourceforge.net/project/linuxcommand/TLCL/09.12/TLCL-09.12.pdf

http://nixshell.wordpress.com/

http://habrahabr.ru/post/158971/

http://habrahabr.ru/post/165317/ Ctr-C Ctr-V

http://mmb.pcb.ub.es/~carlesfe/unix/tricks.txt

http://www.fizerkhan.com/blog/posts/What-I-learned-from-other-s-shell-scripts.html

http://stackoverflow.com/questions/592620/check-if-a-program-exists-from-a-bash-script

APP_ROOT=`dirname "$0"`

http://www.davidpashley.com/articles/writing-robust-shell-scripts/

There is never any reason to do

somecommand

if [ $? -eq 0 ]

... Or variants with ((...)) or whatever.

Just do:

if somecommand

Because if regards 0 as true:

if run some command; then

do this other thing

fi

remove blank lines: grep -v '^$' input.txt

http://www.ibm.com/developerworks/linux/library/l-10sysadtips/index.html

http://www.pixelbeat.org/docs/linux_commands.html

http://www.tuxradar.com/content/command-line-tricks-smart-geeks

http://www.opennet.ru/docs/RUS/bash_scripting_guide/

traps http://redsymbol.net/articles/bash-exit-traps/

http://www.reddit.com/r/programming/comments/15mnxe/how_exit_traps_can_make_your_bash_scripts_way/

https://github.com/visionmedia/histo#readme stream->hitogram

Always put double quotes around variable substitutions, otherwise characters like spaces and *appearing in the value are interpreted by the shell. I.e., write "$c", not $c

http://pypi.python.org/pypi/anytop/0.2.0

cut -c 1-1 /usr/share/dict/words | tr [:upper:] [:lower:] | anytop

How to make the 'cut' command treat several sequential delimiters as one?

cat text.txt | tr -s ' ' | cut -d ' ' -f 4

awk '{ printf $4; }'

http://blog.andrewhays.net/love-your-terminal

http://news.ycombinator.com/item?id=5337558

http://www.reddit.com/r/programming/comments/19uiim/unix_tricks_you_should_be_using/

Getting specific lines from file

Assuming you need line #20 only

sed '20 p' -n

Assuming you need lines from 20 to 40

sed -n '20,40p' file_name

or

awk 'FNR>=20 && FNR<=40' file

Search all files with given content in all subfolders

grep -r ERROR --exclude-dir=./git ./

nohup grep -r ERROR --exclude-dir=./git --include=\*.{cpp,c,cc,h,hpp} ./ > ~/all_errors &

http://stackoverflow.com/questions/221921/grep-exclude-include-syntax-do-not-grep-through-certain-files

Find

http://www.openlogic.com/wazi/bid/188094/How-to-Find-Anything-Under-Linux

http://www.danielmiessler.com/study/find/

find . -type f ! -name *.c -exec rm {} \; # removes all files but *.c

find . -name "*.r" -exec grep "needle" '{}' \; -print

Remove old files

find /path/to/files* -mtime +5 -exec rm {} \; -- 5 days old

find . -exec grep -l “string to find” {} \;

If you also want to print an excerpt of the matched text,

find . -exec grep -H “string to find” {} \;

Search recursively for “expr” in all *.c and *.h files:

find -name '*.[ch]' | xargs grep -E 'expr'

SET OPERATION IN SHELL

http://www.catonmat.net/blog/set-operations-in-unix-shell/

http://www.catonmat.net/blog/set-operations-in-unix-shell-simplified/

for file in *.txt.gz; do echo -n $file; zcat $file | wc -l; done

find /dir -name "prefix*" -exec md5sum {} \;.

find /dir -name "prefix*" | xargs md5sum

grep pattern $(find . -name '*.pl' -or -name '*.pm' -or -name '*.pod' | grep -v .svn)

Concatenate lines with training backslash:

sed ':a; /\\$/N; s/\\\n//; ta'

Echo the path one item per line (assumes GNU tr):

echo $PATH | tr : '\n'

Bash history expansion

!$ last command argument

!* all the arguments from the previous command

http://superuser.com/questions/210846/how-can-i-get-a-specific-argument-from-a-previous-command-in-bash

http://stackoverflow.com/questions/4009412/bash-first-argument-of-the-previous-command

https://news.ycombinator.com/item?id=5565689

Proccess management

pidof -- find the process ID of a running program.

killall - kill processes by name

pgrep looks through the currently running processes and lists the process IDs which matches the selection criteria

pkill will send the specified signal (by default SIGTERM) to each process instead of listing them

on stdout

scp directory recursively

scp -C -p -r user@hostname:/folder destination_folder

-C means compress, -p means to preserve file times, -r means to recursively copy directories.

comm

-1 suppress lines unique to FILE1

-2 suppress lines unique to FILE2

-3 suppress lines that appear in both files

Find lines which are common to both sorted files

comm -12 a.txt b.txt

Find lines which are unique to first file (a.txt) only

comm -23 < (sort a.txt) <(sort b.txt)

Find lines which are unique to second file (b.txt) only

comm -13 <(sort a.txt) <(sort b.txt)

Ctrl-R

Что бы история не терялась, в .bashrc добавляется такое:

export HISTCONTROL=ignoredups

export HISTFILESIZE=100000

export HISTSIZE=100000

shopt -s histappend #историю дополнять, а не перезатирать

export PROMPT_COMMAND='history -a' #это выполняется после каждой команды, в данном случае сейвит историю

# [Press Ctrl+R from the command prompt, which will display the reverse-i-search prompt] (reverse-i-search)`red': cat /etc/redhat-release [Note: Press enter when you see your command, which will execute the command from the history] # cat /etc/redhat-release Fedora release 9 (Sulphur)

Sometimes you want to edit a command from history before executing it:

# [Press Ctrl+R from the command prompt, which will display the reverse-i-search prompt] (reverse-i-search)`httpd': service httpd stop [Note: Press either left arrow or right arrow key when you see your command, which will display the command for you to edit, before executing it] # service httpd start

3. Repeat previous command quickly using 4 different methods

Sometime you may end up repeating the previous commands for various reasons. Following are the 4 different ways to repeat the last executed command.

• 1. Use the up arrow to view the previous command and press enter to execute it.

  2. Type !! and press enter from the command line

  3. Type !-1 and press enter from the command line.

  4. Press Control+P will display the previous command, press enter to execute it

4. Execute a specific command from history

In the following example, If you want to repeat the command #4, you can do !4 as shown below.

# history | more 1 service network restart 2 exit 3 id 4 cat /etc/redhat-release # !4 cat /etc/redhat-release Fedora release 9 (Sulphur)

5. Execute previous command that starts with a specific word

Type ! followed by the starting few letters of the command that you would like to re-execute. In the following example, typing !ps and enter, executed the previous command starting with ps, which is ‘ps aux | grep yp’.

Top for Network:

iftop

Top for Input/Output (I/O):

iotop

Get SSL website Certificate:

openssl s_client -connect www.google.com:443 < /dev/null

List processes with Port 80 open:

lsof -i tcp:80

Edit a remote file directly in vim:

vim scp://user@remote//path/to/file

Linux init and shutdown scripts

http://www.techrepublic.com/article/customizing-the-linux-bootup-and-shutdown-processes/1056248

http://nixshell.wordpress.com/2007/07/25/understanding-init-scripts/

script ( let say named "crond") would normally have two links in the appropriate /etc/rcX.d directory like below:

lrwxrwxrwx. 1 root root 17 Oct 22 1992 S90crond -> ../init.d/crond

lrwxrwxrwx. 1 root root 17 Oct 22 1992 K09crond -> ../init.d/crond

Where the "S" would be the startup for the script and the "K" would be the stop (kill).

BASH SCRIPTING

http://mywiki.wooledge.org/BashFAQ

http://mywiki.wooledge.org/BashPitfalls

http://mywiki.wooledge.org/BashGuide

http://mywiki.wooledge.org/CategoryUnix

http://mywiki.wooledge.org/ProcessManagement

http://www.servercobra.com/bash-how-to-write-better-scripts/

http://sysadvent.blogspot.com/2011/12/day-9-data-in-shell.html

http://www.davidpashley.com/articles/writing-robust-shell-scripts.html

Adding the following lines at the beginning of the script can be very useful: http://www.puschitz.com/pblog/

export PS4='$0.$LINENO+ ' exec > /tmp/script.log exec 2>&1

http://fvue.nl/wiki/Bash:_Error_handling

http://www.davidpashley.com/articles/writing-robust-shell-scripts.html

set -x # enables debugging

set -u # stop if any $FOO variable is not defined

set -e # stop execution if return of any command != 0 (set -o errexit)

Tracing shell scripts with time stamps: set -x PS4='+\t ' Here \t stands for time stamp.

nohup my.sh &

(time my_command_here) > logfile 2>&1

$# tells the number of arguments available;

$@ list of arguments

for inputFile in "$@"

do

..

done

Disk Usage

https://peteris.rocks/blog/disk-space-usage-in-linux/

Largest 5 files:

find . -type f -exec ls -s {} \; | sort -n -r | head -5

find ~ -size +100M

Для вывода отсортированного списка файлов размером больше 500Мб: find . -type f -size +500M -exec ls -l {} \; | sort -k5 -n -r| less Для вывода отсортированного по размеру списка директорий: find . -type d -maxdepth 1 -exec du -s {} \;|sort -k1 -r -n| less Для выявления конечной директории с большими файлами можно снять ограничение на рекурсивную проверку: find . -type d -exec du -s {} \;| sort -k1 -r -n| less

http://dev-random.net/finding-folder-size-with-du-and-disk-space-left-with-df-in-linux/#more-495

df -lh reports free disk space usage

Disk Full?

http://ubuntuforums.org/showthread.php?t=1122670

Then / is almost full try this to find the contributors:

du -h /tmp

du -h /var/tmp

du -h /var/log

sudo du / 2>/dev/null |sort -gr| more

sudo du -h --max-depth=1 / | grep '[0-9]G'

sudo find / -xdev -size +100M -exec du -k {} \; | sort -nr | head -20

sudo df -i # will tell the status of inode usage

http://www.linfo.org/du.html

du -h filename

find . -name *.trc -exec du -h {} \; -- size of .trc files

du -s - stays for summary:

du -s -h B*

-------------

199G B10

12G B11

9.9G B20

280K BM6

120K BM8

18G BM9

108K BMT

1.8G BUS

du -sh ./dirname | sort -h

When you first login, bash reads these initialization files in order (if they exist):

http://www.thegeekstuff.com/2008/10/execution-sequence-for-bash_profile-bashrc-bash_login-profile-and-bash_logout/

/etc/profile -- systemwide profile applies to all users

Then, it looks for these files and executes the FIRST one it finds (note that the file names start with a period):

$HOME/.bash_profile

$HOME/.bash_login

$HOME/.profile

For interactive non-login shells, it executes:

$HOME/.bashrc

At logout, it executes:

$HOME/.bash_logout

http://www.solipsys.co.uk/new/BashInitialisationFiles.html

http://news.ycombinator.com/item?id=4369485

tr -d '|,' < file # to remove characters from file

grepping pid from ps output

using tr http://linux.die.net/man/1/tr

ps -ef | grep python | grep SCALE | tr -s ' ' | cut -d ' ' -f2

ps -ef | grep python | grep SCALE | awk '{ print $2; }'

AWK

ps -u fffadm -o pid,rss:8,cmd | awk 'NR>1 {A+=$2} {print} END{print "Total RSS: " A}'

Creating daemon using bash

http://commandlinemac.blogspot.com/2008/12/bourne-again-daemon.html

http://www.mmo2play.com/?page=tutorials&tut=ld1v2

http://cr.yp.to/daemontools.html

http://blog.n01se.net/blog-n01se-net-p-145.html

Preventing running several instances in parallel

flock taskset

http://unix.stackexchange.com/questions/22044/correct-locking-in-shell-scripts

Run the last command as root

sudo !!

python -m SimpleHTTPServer

Serve current directory tree at http://$HOSTNAME:8000/

python -m SimpleHTTPServer 8080

Looking command in bash history

Once you’ve found the command using Ctrl-R you have several options:

• 1. Run it verbatim – just press Enter

  2. Edit it before running – you can use arrow keys or different key bindings to navigate to the point you want to edit

  3. Cycle through other commands that match the letters you’ve typed – press Ctrl-R successively

  4. Quit the search and back to the command line empty-handed – press Ctrl-G

to edit a command with a specific history number: $ fc 127

VIM

:w !sudo tee %

Save a file you edited in vim without the needed permissions

^foo^bar

Runs previous command but replacing

PIPE

http://habrahabr.ru/post/195152/

http://www.catonmat.net/blog/unix-utilities-pipe-viewer/

http://vincebuffalo.com/2013/08/08/the-mighty-named-pipe.html

ADDING THE time INTO log USING THE shell:

someprogramhere 2>&1 | while read -r line; do echo "$(date '+%F %T'): $line"; done

find most recent files

find . -mmin -5

LOOP

for i in `seq 2 $max`

do

echo "$i"

done

max=10

for ((i=2; i<=$max; ++i )) ;

do

echo "$i"

done

BASH: running jobs in parallel

http://coldattic.info/shvedsky/s/blogs/a-foo-walks-into-a-bar/posts/16

#!/bin/bash

#for i in `seq 0 9`; do

for i in $(seq 0 100); do #same as above

echo $i &

done

wait

#----------------------------------------------------

$PID=`pidof java`

if [ "$PID" ]; then

while [ $(kill -0 "$PID") ]; do

sleep 1

done

fi

KILL

http://unix.stackexchange.com/questions/8916/when-should-i-not-kill-9-a-process

https://news.ycombinator.com/item?id=7792973

before you attempt to kill -9 a process run ps -ef to see what his parent is and attempt the -15(TERM) or -2(INT) and lastly -9(KILL) on his parent.

kill -11 (SIGSEGV). is essentially the same as -9, with the exception that it creates a coredump

The kill command is normally used to send signals to a particular process,

but by using 0 as a signal you are only checking whether a process with such a pid exists

without sending a signal.

escaping * in bash -> put " " around the variable

SQL='SELECT * FROM X"

cmd '$SQL"

example of loop and escaping "

CMD="time hdbsql -j -f -u system -p $PWD -i $INS "

COUNTER=0

while [ $COUNTER -lt 31 ]; do

let COUNTER=COUNTER+1

if [ $COUNTER -lt 10 ];then

FILTER=2005070$COUNTER

else

FILTER=200507$COUNTER

fi

$CMD "UPDATE A SET \"/B49/S_INVCD_QTY\" = 50 WHERE CALDAY = '$FILTER'"

$CMD "ALTER SYSTEM START PERFTRACE FUNCTION_PROFILER"

START=`date +%s`

$CMD "MERGE DELTA OF A"

FINISH=`date +%s`

ELAPSED=`expr $FINISH - $START`

echo "Elapsed time: $ELAPSED"

$CMD "ALTER SYSTEM STOP PERFTRACE"

FILE=/tmp/${COUNTER}_${ELAPSED}.tpt

rm -f $FILE

$CMD "ALTER SYSTEM SAVE PERFTRACE INTO FILE '$FILE'"

done

find xargs

http://offbytwo.com/2011/06/26/things-you-didnt-know-about-xargs.html

why this does not work?: find . -name *.trc | xargs du -h

Find files with specific string

find . -name a*.txt | xargs grep -l SomeStringInFile

The most robust method is this:

find [whatever] -print0 | xargs -0 yourCommandHere

The -print0 flag tells find to use \0 (null character) delimiters between filenames,

and the -0 flag tells xargs to expect these \0 delimiters.

find /home/adam/ -size +100M # big files

find /home/adam/ -size -100M # small files

Total files size using AWK

ls -lh /large/hualin/data/*.tbl | awk '{sum=sum+$5} END{print sum}'

File descriptors: 1 for stdout, 2 for stderr, 0 for stdin

ls -al 1>file1.txt 2>file2.txt

Save the output to a file and at the same time echo everything to the screen:

ls -al | tee blah.txt

Redirect standard error to point to the same place as stdout and then we pipe the whole mess to tee in order to both save and echo at the same time:

ls -al 2>&1 | tee blah.txt

command 2>&1 > file # <---- do not use this http://en.wikipedia.org/wiki/Redirection_%28computing%29

command > file 2>&1 # use this

BASH variablesBASH http://tldp.org/LDP/abs/html/

BASH shortcuts

ctrl-l clears the screen

ctrl-u erases the whole line

ctrl-a brings cursor to the beginning of the line

ctrl-e brings cursor to the end of the line

ctrl-k erases from cursor to the end of the line

ctrl-w erases the last word – the the previous space

ctrl-y pastes what you erased

ctrl-f erases the last word – to the previous slash

esc-. pastes last argument of previous line

esc-, completes, shows each match one-by-one every time it is pressed