.bash_profile
alias app = 'cd ~/my_often_visited_dir'
https://habrahabr.ru/company/ruvds/blog/328346/
union of 2 pipes : get head and tail together
tail -5 T.csv | (head -5 T.csv && cat ) | cut -d',' -f1
Защита от одновременного запуска нескольких копий скрипта:
flock -n /tmp/lock.txt -c "/home/user1/test.sh > test.log"
https://habrahabr.ru/company/ruvds/blog/326826/
https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
http://jvns.ca/blog/2017/03/26/bash-quirks/
https://habrahabr.ru/company/ruvds/blog/323330/
https://likegeeks.com/write-shell-script/
https://likegeeks.com/linux-bash-scripting-awesome-guide-part5/
https://mkaz.tech/geek/command-line-basics-and-tips.html
https://www.blockloop.io/mastering-bash-and-terminal
https://news.ycombinator.com/item?id=13400350
https://bluepenguinlist.com/2016/11/04/bash-scripting-tutorial/
https://habrahabr.ru/company/mailru/blog/311762/
https://leanpub.com/the-tao-of-tmux/read Tmux
https://habrahabr.ru/post/309518/ execution scrip via browser Rundeck
inotify flawors
https://github.com/cortesi/modd file change watcher
https://github.com/cortesi/devd local web server
https://news.ycombinator.com/item?id=13856623
Why grep is fast?
https://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html
https://news.ycombinator.com/item?id=12350890
https://news.ycombinator.com/item?id=11832941
https://habrahabr.ru/post/281601/ bash + logs
grep '[^[:blank:]]' remove empty lines
curl ifconfig.me ipinfo.io/ip IP address
https://news.ycombinator.com/item?id=11322007
Replacing ^M with new line in file:
tr '\r' '\n' < inputfile > outfile
find . -name '*.py' -print0 | xargs -0 grep 'try'
The -print0 argument tells find to use the NULL character to terminate each name that it prints out. The -0 argument tells xargs that its input tokens are NULL-terminated. This avoids issues with characters that otherwise would be treated as special, like quotes.
Add prefix and suffix to every line:
awk '$0="prefix"$0' file > new_file
cat file | while read line; do echo prefix $line suffix; done
Add leading number substitute tab with space and add leading and trailing text
cat -n swell_1662_IQC | cut -f1,2,3 | tr -s '\t' ',' | awk '{print "before" $0 "after";}'
cat -n swell_1662_IQC | cut -f1,2,3 | tr -s '\t' ',' | awk '{print "insert into tmp_1662_IQC(id, sample,barcode) values(" $0 ");";}'
but if the inserted values are not numeric then we need to insert ' '
http://stackoverflow.com/questions/9899001/how-to-escape-single-quote-in-awk-inside-printf
cat -n swell_1662_IQC | cut -f1,2,3 | tr -s '\t' ',' | awk -F "," '{print "insert into tmp_1662_IQC(id, sample,barcode) values("$1 "," $2 ",'\''" $3 "'\'');";}'
How to put single quote inside awk print - use variable:
awk -v q=\' '{print "or filename like "q $0"%" q}';
Combining several outputs into single output line (use echo -n and &&cat):
echo 1 | (echo -n 2 && cat)
12
Calculate the frequency of word in column#11 in tab separated file
awk -F '\t' '{A[$11]++}END{for(i in A)print i,A[i]}' $file
this version produces the tab separated output
awk -F '\t' 'BEGIN {OFS = FS} {A[$11]++}END{for(i in A)print i,A[i]}' $file
http://blog.garage-coding.com/2016/02/05/bash-fifo-jobqueue.html
https://news.ycombinator.com/item?id=11048847
http://www.theunixschool.com/p/awk-sed.html
http://cb.vu/unixtoolbox.xhtml
http://habrahabr.ru/company/ua-hosting/blog/273201/ VNC server
http://phili.pe/posts/free-concurrency-with-gnu-parallel/
https://zeroturnaround.com/rebellabs/5-command-line-tools-you-should-be-using/
http://www.davidpashley.com/articles/writing-robust-shell-scripts/
https://www.reddit.com/r/programming/comments/3j2nxj/5_command_line_tools_you_should_be_using/
https://github.com/jlevy/the-art-of-command-line/blob/master/README-ru.md
http://redsymbol.net/articles/unofficial-bash-strict-mode/
http://www.kfirlavi.com/blog/2012/11/14/defensive-bash-programming
https://news.ycombinator.com/item?id=10736584 DEFENSIVE BASH PROGRAMMING
https://news.ycombinator.com/item?id=10124461
Use tr for splitting the comma-separated string to have one line per word
head -1 myfile.csv | tr ',' '\n'
http://codeyarns.com/2015/07/31/path-environment-variable-in-sudo/
echo $PATH | tr -s ':' '\n'
python -c "import os; print os.environ['PATH'].replace(':', '\n');"
http://www.procoefficient.com/blog/shell-script-to-setup-automatic-deployment-using-git/
Status of all services
sudo service --status-all
watch -n X cmd simply re-executes cmd every X seconds:
watch -n 1 "mysql -h 127.0.0.1 -u root -proot -e 'show processlist;'"
sudo !! run previous command under sudo
!^ — первый аргумент предыдущей команды;
!$ — последний аргумент предыдущей команды;
!* — все аргументы предыдущей команды;
!-2 — вторая с конца команда.
nl - for numeration
To list only the files in current directory:
$ find . -maxdepth 1 -type f | sort
To list only the directories in current directory:
$ find . -maxdepth 1 -type d | sort
curl -v -g https://192.168.56.10/ -u mlubinsky:password
curl localhost:8998/sessions/0 | python -m json.tool
https://github.com/jlevy/the-art-of-command-line
https://github.com/jeroenjanssens/data-science-at-the-command-line
http://multithreaded.stitchfix.com/blog/2015/04/09/stitch-fix-heart-unix/
set -o errexit set -o pipefail set -o nounset
CUT command
grep X321 MS*txt | cut -f1 | cut -c17-
grep "/bin/bash" /etc/passwd | cut -d':' -f1-4,6,7
http://www.segv.me/ - linux/pyhon/postgresql notes
http://habrahabr.ru/post/260335/ load expained
http://www.ghacks.net/2009/04/04/get-to-know-linux-the-etcinitd-directory/
'chkconfig' allows the checking the startup level of services and gives the ability to set the service. Running 'chkconfig' will setup the scripts in the needed "rcX.d" directory.
other command of use:
/sbin/service "stop|start|restart|etc"
this command is the same as of typing "/etc/init.d/ ....
Recursive grep -r : Example ignore case and show only file names:
grep -ril needle folder
http://wiki.bash-hackers.org/commands/builtin/read
http://www.cyberciti.biz/faq/unix-linux-bash-split-string-into-array/
Reading line by line and splitting line using read ( another option is cut)
#!/bin/bash
FILE=$1
while read line; do
echo "This is a line : $line"
read testrunid fname <<< $line
echo $fname
done < $FILE
filename="$1"while read -r line do name=$line echo "Name read from file - $name"done < "$filename"
cat a b | sort | uniq > c # c is a union b cat a b | sort | uniq -d > c # c is a intersect b cat a b b | sort | uniq -u > c # c is set difference a - b
Summing all numbers in the third column of a text file (this is probably 3X faster and 3X less code than equivalent Python):
awk '{ x += $3 } END { print x }' myfile
select from the middle of the file
sed -n '10000000,+20p' filename
sed -n '10000000,10000020p' filename
tail -n 10000 print the last 10000 lines.
tail -n +10000 start print at 10000th line
head -n 10000010 filename | tail 10
timeout - kill command if it runs longer then given time http://ss64.com/bash/timeout.html
Color diff site-by-site
http://habrahabr.ru/post/221273/ bash tips from Google
For example, to determine whether a particular regular expression regex was present somewhere in a file options, we might apply grep(1) with its POSIX -q option to suppress output and just use the exit value:
grep -q regex options
An approach sometimes taken is then to test the exit value with the $?parameter, using if to check if it’s non-zero, which is not very elegant and a bit hard to read:
# Bad practice grep -q regex options if (($? > 0)); then printf '%s\n' 'myscript: Pattern not found!' >&2 exit 1 fi
Because the if construct by design tests the exit value of commands, it’s better to test the command directly, making the expansion of $? unnecessary:
# Better if grep -q regex options; then # Do nothing : else printf '%s\n' 'myscript: Pattern not found!\n' >&2 exit 1 fi
We can precede the command to be tested with ! to negate the test as well, to prevent us having to use else as well:
# Best if ! grep -q regex options; then printf '%s\n' 'myscript: Pattern not found!' >&2 exit 1 fi
ls does not all to hide direcories, by find does:
find FIX -maxdepth 1 -type f -not -name 't_*'
delete empty files
find . -type f -size 0 -delete
System monitoring scripts
http://devo.ps/blog/2013/03/06/troubleshooting-5minutes-on-a-yet-unknown-box.html
https://github.com/BashtonLtd/whatswrong/blob/master/whatswrong
http://www.fduran.com/blog/quick-linux-server-review-for-mortals/
http://habrahabr.ru/company/selectel/blog/221143/
https://github.com/rory/SystemAutopsy
http://word.bitly.com/post/74839060954/ten-things-to-monitor?h=2
bash best practice
https://news.ycombinator.com/item?id=7815190
https://news.ycombinator.com/item?id=7981633
http://habrahabr.ru/post/221273/
http://yellerapp.com/posts/2014-06-03-bash-for-ci.html
http://redsymbol.net/articles/unofficial-bash-strict-mode/
chroot
Disable bell in bash: add the following line in ~/.inputrc :
set bell-style none
~/.bashrc
http://www.pixelbeat.org/programming/shell_script_mistakes.html
http://bash.cumulonim.biz/BashPitfalls.html
http://blog.existentialize.com/dont-pipe-to-your-shell.html
http://jeroenjanssens.com/2013/09/19/seven-command-line-tools-for-data-science.html
https://news.ycombinator.com/item?id=6412190
http://www.reddit.com/r/programming/comments/1pixp1/use_multiple_cpu_cores_with_your_linux_commands/
https://news.ycombinator.com/item?id=6360320
http://www.leancrew.com/all-this/2011/12/more-shell-less-egg/
http://voxel.dl.sourceforge.net/project/linuxcommand/TLCL/09.12/TLCL-09.12.pdf
http://nixshell.wordpress.com/
http://habrahabr.ru/post/158971/
http://habrahabr.ru/post/165317/ Ctr-C Ctr-V
http://mmb.pcb.ub.es/~carlesfe/unix/tricks.txt
http://www.fizerkhan.com/blog/posts/What-I-learned-from-other-s-shell-scripts.html
http://stackoverflow.com/questions/592620/check-if-a-program-exists-from-a-bash-script
APP_ROOT=`dirname "$0"`
http://www.davidpashley.com/articles/writing-robust-shell-scripts/
There is never any reason to do
somecommand
if [ $? -eq 0 ]
... Or variants with ((...)) or whatever.
Just do:
if somecommand
Because if regards 0 as true:
if run some command; then
do this other thing
fi
remove blank lines: grep -v '^$' input.txt
http://www.ibm.com/developerworks/linux/library/l-10sysadtips/index.html
http://www.pixelbeat.org/docs/linux_commands.html
http://www.tuxradar.com/content/command-line-tricks-smart-geeks
http://www.opennet.ru/docs/RUS/bash_scripting_guide/
traps http://redsymbol.net/articles/bash-exit-traps/
http://www.reddit.com/r/programming/comments/15mnxe/how_exit_traps_can_make_your_bash_scripts_way/
https://github.com/visionmedia/histo#readme stream->hitogram
Always put double quotes around variable substitutions, otherwise characters like spaces and *appearing in the value are interpreted by the shell. I.e., write "$c", not $c
http://pypi.python.org/pypi/anytop/0.2.0
cut -c 1-1 /usr/share/dict/words | tr [:upper:] [:lower:] | anytop
How to make the 'cut' command treat several sequential delimiters as one?
cat text.txt | tr -s ' ' | cut -d ' ' -f 4
awk '{ printf $4; }'
http://blog.andrewhays.net/love-your-terminal
http://news.ycombinator.com/item?id=5337558
http://www.reddit.com/r/programming/comments/19uiim/unix_tricks_you_should_be_using/
Getting specific lines from file
Assuming you need line #20 only
sed '20 p' -n
Assuming you need lines from 20 to 40
sed -n '20,40p' file_name
or
awk 'FNR>=20 && FNR<=40' file
Search all files with given content in all subfolders
grep -r ERROR --exclude-dir=./git ./
nohup grep -r ERROR --exclude-dir=./git --include=\*.{cpp,c,cc,h,hpp} ./ > ~/all_errors &
Find
http://www.openlogic.com/wazi/bid/188094/How-to-Find-Anything-Under-Linux
http://www.danielmiessler.com/study/find/
find . -type f ! -name *.c -exec rm {} \; # removes all files but *.c
find . -name "*.r" -exec grep "needle" '{}' \; -print
Remove old files
find /path/to/files* -mtime +5 -exec rm {} \; -- 5 days old
find . -exec grep -l “string to find” {} \;
If you also want to print an excerpt of the matched text,
find . -exec grep -H “string to find” {} \;
Search recursively for “expr” in all *.c and *.h files:
find -name '*.[ch]' | xargs grep -E 'expr'
SET OPERATION IN SHELL
http://www.catonmat.net/blog/set-operations-in-unix-shell/
http://www.catonmat.net/blog/set-operations-in-unix-shell-simplified/
for file in *.txt.gz; do echo -n $file; zcat $file | wc -l; done
find /dir -name "prefix*" -exec md5sum {} \;.
find /dir -name "prefix*" | xargs md5sum
grep pattern $(find . -name '*.pl' -or -name '*.pm' -or -name '*.pod' | grep -v .svn)
Concatenate lines with training backslash:
sed ':a; /\\$/N; s/\\\n//; ta'
Echo the path one item per line (assumes GNU tr):
echo $PATH | tr : '\n'
Bash history expansion
!$ last command argument
!* all the arguments from the previous command
http://stackoverflow.com/questions/4009412/bash-first-argument-of-the-previous-command
https://news.ycombinator.com/item?id=5565689
Proccess management
pidof -- find the process ID of a running program.
killall - kill processes by name
pgrep looks through the currently running processes and lists the process IDs which matches the selection criteria
pkill will send the specified signal (by default SIGTERM) to each process instead of listing them
on stdout
scp directory recursively
scp -C -p -r user@hostname:/folder destination_folder
-C means compress, -p means to preserve file times, -r means to recursively copy directories.
comm
-1 suppress lines unique to FILE1
-2 suppress lines unique to FILE2
-3 suppress lines that appear in both files
Find lines which are common to both sorted files
comm -12 a.txt b.txt
Find lines which are unique to first file (a.txt) only
comm -23 < (sort a.txt) <(sort b.txt)
Find lines which are unique to second file (b.txt) only
comm -13 <(sort a.txt) <(sort b.txt)
Ctrl-R
Что бы история не терялась, в .bashrc добавляется такое:
export HISTCONTROL=ignoredups
export HISTFILESIZE=100000
export HISTSIZE=100000
shopt -s histappend #историю дополнять, а не перезатирать
export PROMPT_COMMAND='history -a' #это выполняется после каждой команды, в данном случае сейвит историю
# [Press Ctrl+R from the command prompt, which will display the reverse-i-search prompt] (reverse-i-search)`red': cat /etc/redhat-release [Note: Press enter when you see your command, which will execute the command from the history] # cat /etc/redhat-release Fedora release 9 (Sulphur)
Sometimes you want to edit a command from history before executing it:
# [Press Ctrl+R from the command prompt, which will display the reverse-i-search prompt] (reverse-i-search)`httpd': service httpd stop [Note: Press either left arrow or right arrow key when you see your command, which will display the command for you to edit, before executing it] # service httpd start
Sometime you may end up repeating the previous commands for various reasons. Following are the 4 different ways to repeat the last executed command.
Use the up arrow to view the previous command and press enter to execute it.
Type !! and press enter from the command line
Type !-1 and press enter from the command line.
Press Control+P will display the previous command, press enter to execute it
In the following example, If you want to repeat the command #4, you can do !4 as shown below.
# history | more 1 service network restart 2 exit 3 id 4 cat /etc/redhat-release # !4 cat /etc/redhat-release Fedora release 9 (Sulphur)
Type ! followed by the starting few letters of the command that you would like to re-execute. In the following example, typing !ps and enter, executed the previous command starting with ps, which is ‘ps aux | grep yp’.
Top for Network:
iftop
Top for Input/Output (I/O):
iotop
Get SSL website Certificate:
openssl s_client -connect www.google.com:443 < /dev/null
List processes with Port 80 open:
lsof -i tcp:80
Edit a remote file directly in vim:
vim scp://user@remote//path/to/file
Linux init and shutdown scripts
http://www.techrepublic.com/article/customizing-the-linux-bootup-and-shutdown-processes/1056248
http://nixshell.wordpress.com/2007/07/25/understanding-init-scripts/
script ( let say named "crond") would normally have two links in the appropriate /etc/rcX.d directory like below:
lrwxrwxrwx. 1 root root 17 Oct 22 1992 S90crond -> ../init.d/crond
lrwxrwxrwx. 1 root root 17 Oct 22 1992 K09crond -> ../init.d/crond
Where the "S" would be the startup for the script and the "K" would be the stop (kill).
BASH SCRIPTING
http://mywiki.wooledge.org/BashFAQ
http://mywiki.wooledge.org/BashPitfalls
http://mywiki.wooledge.org/BashGuide
http://mywiki.wooledge.org/CategoryUnix
http://mywiki.wooledge.org/ProcessManagement
http://www.servercobra.com/bash-how-to-write-better-scripts/
http://sysadvent.blogspot.com/2011/12/day-9-data-in-shell.html
http://www.davidpashley.com/articles/writing-robust-shell-scripts.html
Adding the following lines at the beginning of the script can be very useful: http://www.puschitz.com/pblog/
export PS4='$0.$LINENO+ ' exec > /tmp/script.log exec 2>&1
http://fvue.nl/wiki/Bash:_Error_handling
http://www.davidpashley.com/articles/writing-robust-shell-scripts.html
set -x # enables debugging
set -u # stop if any $FOO variable is not defined
set -e # stop execution if return of any command != 0 (set -o errexit)
Tracing shell scripts with time stamps: set -x PS4='+\t ' Here \t stands for time stamp.
nohup my.sh &
(time my_command_here) > logfile 2>&1
$# tells the number of arguments available;
$@ list of arguments
for inputFile in "$@"
do
..
done
Disk Usage
https://peteris.rocks/blog/disk-space-usage-in-linux/
Largest 5 files:
find . -type f -exec ls -s {} \; | sort -n -r | head -5
find ~ -size +100M
Для вывода отсортированного списка файлов размером больше 500Мб: find . -type f -size +500M -exec ls -l {} \; | sort -k5 -n -r| less Для вывода отсортированного по размеру списка директорий: find . -type d -maxdepth 1 -exec du -s {} \;|sort -k1 -r -n| less Для выявления конечной директории с большими файлами можно снять ограничение на рекурсивную проверку: find . -type d -exec du -s {} \;| sort -k1 -r -n| less
http://dev-random.net/finding-folder-size-with-du-and-disk-space-left-with-df-in-linux/#more-495
df -lh reports free disk space usage
Disk Full?
http://ubuntuforums.org/showthread.php?t=1122670
Then / is almost full try this to find the contributors:
du -h /tmp
du -h /var/tmp
du -h /var/log
sudo du / 2>/dev/null |sort -gr| more
sudo du -h --max-depth=1 / | grep '[0-9]G'
sudo find / -xdev -size +100M -exec du -k {} \; | sort -nr | head -20
sudo df -i # will tell the status of inode usage
du -h filename
find . -name *.trc -exec du -h {} \; -- size of .trc files
du -s - stays for summary:
du -s -h B*
-------------
199G B10
12G B11
9.9G B20
280K BM6
120K BM8
18G BM9
108K BMT
1.8G BUS
du -sh ./dirname | sort -h
When you first login, bash reads these initialization files in order (if they exist):
/etc/profile -- systemwide profile applies to all users
Then, it looks for these files and executes the FIRST one it finds (note that the file names start with a period):
$HOME/.bash_profile
$HOME/.bash_login
$HOME/.profile
For interactive non-login shells, it executes:
$HOME/.bashrc
At logout, it executes:
$HOME/.bash_logout
http://www.solipsys.co.uk/new/BashInitialisationFiles.html
http://news.ycombinator.com/item?id=4369485
tr -d '|,' < file # to remove characters from file
grepping pid from ps output
using tr http://linux.die.net/man/1/tr
ps -ef | grep python | grep SCALE | tr -s ' ' | cut -d ' ' -f2
ps -ef | grep python | grep SCALE | awk '{ print $2; }'
AWK
ps -u fffadm -o pid,rss:8,cmd | awk 'NR>1 {A+=$2} {print} END{print "Total RSS: " A}'
Creating daemon using bash
http://commandlinemac.blogspot.com/2008/12/bourne-again-daemon.html
http://www.mmo2play.com/?page=tutorials&tut=ld1v2
http://cr.yp.to/daemontools.html
http://blog.n01se.net/blog-n01se-net-p-145.html
Preventing running several instances in parallel
flock taskset
http://unix.stackexchange.com/questions/22044/correct-locking-in-shell-scripts
Run the last command as root
sudo !!
python -m SimpleHTTPServer
Serve current directory tree at http://$HOSTNAME:8000/
python -m SimpleHTTPServer 8080
Looking command in bash history
Once you’ve found the command using Ctrl-R you have several options:
Run it verbatim – just press Enter
Edit it before running – you can use arrow keys or different key bindings to navigate to the point you want to edit
Cycle through other commands that match the letters you’ve typed – press Ctrl-R successively
Quit the search and back to the command line empty-handed – press Ctrl-G
to edit a command with a specific history number: $ fc 127
VIM
:w !sudo tee %
Save a file you edited in vim without the needed permissions
^foo^bar
Runs previous command but replacing
PIPE
http://habrahabr.ru/post/195152/
http://www.catonmat.net/blog/unix-utilities-pipe-viewer/
http://vincebuffalo.com/2013/08/08/the-mighty-named-pipe.html
ADDING THE time INTO log USING THE shell:
someprogramhere 2>&1 | while read -r line; do echo "$(date '+%F %T'): $line"; done
find most recent files
find . -mmin -5
LOOP
for i in `seq 2 $max`
do
echo "$i"
done
max=10
for ((i=2; i<=$max; ++i )) ;
do
echo "$i"
done
BASH: running jobs in parallel
http://coldattic.info/shvedsky/s/blogs/a-foo-walks-into-a-bar/posts/16
#!/bin/bash
#for i in `seq 0 9`; do
for i in $(seq 0 100); do #same as above
echo $i &
done
wait
#----------------------------------------------------
$PID=`pidof java`
if [ "$PID" ]; then
while [ $(kill -0 "$PID") ]; do
sleep 1
done
fi
KILL
http://unix.stackexchange.com/questions/8916/when-should-i-not-kill-9-a-process
https://news.ycombinator.com/item?id=7792973
before you attempt to kill -9 a process run ps -ef to see what his parent is and attempt the -15(TERM) or -2(INT) and lastly -9(KILL) on his parent.
kill -11 (SIGSEGV). is essentially the same as -9, with the exception that it creates a coredump
The kill command is normally used to send signals to a particular process,
but by using 0 as a signal you are only checking whether a process with such a pid exists
without sending a signal.
escaping * in bash -> put " " around the variable
SQL='SELECT * FROM X"
cmd '$SQL"
example of loop and escaping "
CMD="time hdbsql -j -f -u system -p $PWD -i $INS "
COUNTER=0
while [ $COUNTER -lt 31 ]; do
let COUNTER=COUNTER+1
if [ $COUNTER -lt 10 ];then
FILTER=2005070$COUNTER
else
FILTER=200507$COUNTER
fi
$CMD "UPDATE A SET \"/B49/S_INVCD_QTY\" = 50 WHERE CALDAY = '$FILTER'"
$CMD "ALTER SYSTEM START PERFTRACE FUNCTION_PROFILER"
START=`date +%s`
$CMD "MERGE DELTA OF A"
FINISH=`date +%s`
ELAPSED=`expr $FINISH - $START`
echo "Elapsed time: $ELAPSED"
$CMD "ALTER SYSTEM STOP PERFTRACE"
FILE=/tmp/${COUNTER}_${ELAPSED}.tpt
rm -f $FILE
$CMD "ALTER SYSTEM SAVE PERFTRACE INTO FILE '$FILE'"
done
find xargs
http://offbytwo.com/2011/06/26/things-you-didnt-know-about-xargs.html
why this does not work?: find . -name *.trc | xargs du -h
Find files with specific string
find . -name a*.txt | xargs grep -l SomeStringInFile
The most robust method is this:
find [whatever] -print0 | xargs -0 yourCommandHere
The -print0 flag tells find to use \0 (null character) delimiters between filenames,
and the -0 flag tells xargs to expect these \0 delimiters.
find /home/adam/ -size +100M # big files
find /home/adam/ -size -100M # small files
Total files size using AWK
ls -lh /large/hualin/data/*.tbl | awk '{sum=sum+$5} END{print sum}'
File descriptors: 1 for stdout, 2 for stderr, 0 for stdin
ls -al 1>file1.txt 2>file2.txt
Save the output to a file and at the same time echo everything to the screen:
ls -al | tee blah.txt
Redirect standard error to point to the same place as stdout and then we pipe the whole mess to tee in order to both save and echo at the same time:
ls -al 2>&1 | tee blah.txt
command 2>&1 > file # <---- do not use this http://en.wikipedia.org/wiki/Redirection_%28computing%29
command > file 2>&1 # use this
BASH variablesBASH http://tldp.org/LDP/abs/html/
BASH shortcuts
ctrl-l clears the screen
ctrl-u erases the whole line
ctrl-a brings cursor to the beginning of the line
ctrl-e brings cursor to the end of the line
ctrl-k erases from cursor to the end of the line
ctrl-w erases the last word – the the previous space
ctrl-y pastes what you erased
ctrl-f erases the last word – to the previous slash
esc-. pastes last argument of previous line
esc-, completes, shows each match one-by-one every time it is pressed
export PS1="\u@\H>" to display user and host
bash: Ctr-A and Ctrl-E are moving cursor to beginning/end of line;
set -o vi allows use vi mode after Esc:
0- move to beginning and $ move end of line
^ move to first char in line; gg - move to beginning of file G - end of file
list of BASH key bindings that I use on a daily basis:
http://typedef.ru/2010/01/time-saving-in-bash/
ctrl-x + ctrl-e — пишем скрипт в виме (или другом дефолтном редакторе), по выходу — скрипт выполняется
There is time command to measure time; also there is another way:
#!/bin/bash
START=`date +%s`
sleep 5
FINISH=`date +%s`
ELAPSED=`expr $FINISH - $START`
echo "Elapsed time: $ELAPSED"
generate number in given range
for i in `seq 100 520`; do echo $i; done
calculating in bash
for i in {2..17}
do
if [ $i -eq 7 -o $i -eq 13 ]; then
continue
fi
FILE=$OUT/$i
n=`$HADOOP fs -lsr $FILE | grep part | wc -l`
size=`$HADOOP fs -dus $FILE | cut -f2`
if [ $n -gt 0 ]
then
echo -e $i "\t" $n "\t" $size "\t" $(($size/$n))
fi
done
bash tips
JOB CONTROL http://www.gnu.org/software/bash/manual/bashref.html#Job-Control
http://www.reddit.com/r/linux/comments/mi80x/give_me_that_one_command_you_wish_you_knew_years/
http://gds.livejournal.com/42541.html
http://rosettacode.org/wiki/Category:UNIX_Shell
http://www.catonmat.net/blog/bash-one-liners-explained-part-three/
.bash_profile and .bashrc
http://www.joshstaiger.org/archives/2005/07/bash_profile_vs.html
http://www.reddit.com/r/programming/comments/9he4v/share_your_bashrc_vimrc_or_other_settings_file_i/
Functions in bash
http://sysadvent.blogspot.com/2011/12/day-9-data-in-shell.html
http://superuser.com/questions/106272/how-to-call-bash-functions
Place the shell function definitions in a shell start up file (for example, .bash_profile ). This way, the shell function is always available for you from the command line. Another option: source this file
How to make function be available to all the child process launched from the parent process:
http://stackoverflow.com/questions/1885871/exporting-a-function-in-shell
function to add to your ~/.bashrc shell if you want to replace a string in multiple files at once.
It ignores everything in .git, otherwise you get 'bad index file sha1 signature' errors.
function gsed () {
if [ -z "$3" ]
then
echo "== Usage: gsed search_string replace_string [path]"
else
egrep --exclude-dir=.git -lRZ "$1" $3 | xargs -0 -l sed -i -e "s/$1/$2/g"
fi
}
Add it to the bottom of ~/.bashrc
To replace all occurences of 'old_name' with 'new_name', recursively for the current directory
gsed "old_name" "new_name" .
kill -l
1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM 16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP 21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ 26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO 30) SIGPWR 31) SIGSYS 34) SIGRTMIN 35) SIGRTMIN+1 36) SIGRTMIN+2 37) SIGRTMIN+3 38) SIGRTMIN+4 39) SIGRTMIN+5 40) SIGRTMIN+6 41) SIGRTMIN+7 42) SIGRTMIN+8 43) SIGRTMIN+9 44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13 48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12 53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9 56) SIGRTMAX-8 57) SIGRTMAX-7 58) SIGRTMAX-6 59) SIGRTMAX-5 60) SIGRTMAX-4 61) SIGRTMAX-3 62) SIGRTMAX-2 63) SIGRTMAX-1 64) SIGRTMAX