Scripts

Sources: linux academy, cyberciti, linuxjournal

Manage File/Directory.

mkdir x  #Makes a directory named 'x' in current folder.

mkdir -p /tmp/y/x  #Makes a directory named '/tmp/y/x', '/tmp/y' and '/tmp' if they don't exist.

rm -r x  #Remove Directory/file named 'x' & its contents.

rm -rf x #Remove Directory/file & its contents without confirmations.

ls #Lists all files/directories in current directory.

ls -ltr #List files sorted in reverse order of time of modification.

7za a outputFile.7z file1tocompress file2tocompress... # Compress/Append file using 7za

tar -zcf final.tar.gz final.tsv #Compress file using tar

xz -d file.txz #Extract .txz files.

tar -xf file.tar #Extract .tar files.

7za x file.7z #Extract .7z file.

7za x file.7z specificFileToExtract #extract specific file from 7z file.

unzip file.zip #Extract .zip file

gunzip file.gz #Extract .gz file

gunzip --keep file.gz #Extract .gz without loosing input file

gunzip -c --keep file.gz | awk -F'\t' '{print $1,$2}' #Pass gunzip result to default outputstream (Console).

7za d file.7z fileToRemove #Remove specific file from 7z file

7za l file.7z #Lists content of .7z file

find / -type f -mtime +30 -print | xargs -I {} mv {} /storage/archive/dbdumps #Find and move

less file.txt  #opens file in read-forward only mode. hit q to quit

more file.txt #opens file in read only mode, and allows forward/backward scroll. hit q to quit

head -10 file.txt #reads top 10 records from file.

tail -10 file.txt #reads last 10 records from file.

tail -f file.txt #reads file and follows it for any appended row in file.

cat file.txt #displays entire content of file.

cat > file.txt #Cat can also be used to write to file. It creates new or write to existing.. hit Ctrl+d to exit writing. use double output-redirection for append.

Internet in CLI???

To download content from any URL using either wget or curl:

wget "Source URL" -O downloadPathFile.txt

curl -O "Source URL"

What if download fails in mid? -c

wget -c "Source URL"

Sending payload using wget or curl. #User agent can be obtained by hitting various site. eg- http://www.whatsmyua.com/ Sources: stackoverflow

wget -O --post-data='{"x":1, "y":"hello"}' --header=Content-Type:application/json --user-agent='<USER AGENT>' "Destination URL"

curl --header "Content-Type: application/json" --request POST --data '{"x":1, "y":"hello"}' "Destination URL"

Connect to another server:

ssh <USER>@<IP> [-p<PORT_REQUIRED_IF_OPENSSH_ON_REMOTE_IS_USING_OTHER_THAN_22>] [-i<FILE.PEM IF_PEM_FILE_PROVIDED_INSTEAD_OF_PASSWORD>]

Execute Command on remote server: same as above and supply command at last: 

ssh <USER>@<IP> "ls -l" #Lists files in user's home directory available on remote server.

grep : easily locate character sequence in huge file and more

grep -oh "word" toFindInFile.txt | wc -l # -oh will print each occurance in new line. wc -l counts lines finally giving occurrence of word in file.

grep -c "word" toFindInFile.txt #Count lines that contains a word.

grep -n "word" toFindInFile.txt #Print Line numbers, where word is found.

grep -A2 -B2 "word" toFindInFile.txt OR grep -2 "word" toFindInFile.txt #Print before/after lines where word is found.

grep --color=[always|never|auto] word toFindInFile.txt #Highlight matches.

grep -v "word" toFindInFile.txt #Print lines that Does NOT contain word.

grep -f fileWithLinesToSearch.txt toFindInFile.txt #Find lines from one file into another.

grep --lines-buffered "abc"

grep "esac" * -R #Find pattern/text in all file/folder/subfolder recursively.

Writing shell script:

Variable declaration: Its simple, think a name and value, place a equal sign in between, that's all. But there are definitely few rules. Spaces are not allowed any where. Name must not start with number or special character . What if we have space in our value? enclose them within quotes. following are valid declarations:

a=5

b="xyz"

c="x yz"4

d1='x yz'5

e=`date` #commands within back ticks are executed and result is stored in variable 'e'

f=$(date) #Alternate to back ticks. 

Values in declared variables can be accessed by variable names prefixed with $ sign. e.g- echo $a Or, echo ${a} #this will print value stored in variable a.

Arrays declaration: 

arr=(value1 value12 value123)

${arr[@]} # All of the items in the array

${!arr[@]} # All of the indexes in the array

${#arr[@]} # Number of items in the array

${#arr[0]} # Length of zero'th item 

Loop Constructs:

• For

for i in {0..10}; do echo $i; done # prints 0 to 10, available since bash 4.0

for i in {0..10..2}; do echo $i; done # prints 0 to 10 at step 2

for each in ${array[@]}; do echo $each; done ##prints each entry in array

*Input Field Separator(IFS) can be used to token/split any char sequence. Its recommended to take backup of IFS before altering it and restore to default(\n) after usage.

ifs_bkp=$IFS #Backup of IFS

a=ABC#DEF#GHI #defines '#' separated Text

IFS=# #Specifies '#' as IFS

for each in $a; do echo $each ; done ##prints each tokens

IFS=$ifs_bkp #Restore IFS to default.

• While

while [ TRUE ]; do echo "hi"; sleep 2; done #infinite loop at interval of 2 seconds.

while read line; do echo $line; done < toFindInFile.txt #for each line from file.

i=0; while [ -le 5 ]; do echo ; i=`expr + 1 `; done

Conditional Constructs:

• if

* Double [[]] allows to write expressions[e.g- [[ "This is a cat." == *a* ]] ]

*[[ "209809" =~ ^[0-9]+$ ]] && echo "True" || echo "False"

*String condition check ' =~ ' can be used within [[]] only

if [ "abc" == "" ]; then { echo "true"; } fi

if [ "abc" == "" ]; then { echo "true"; } else { echo "false"; } fi

if [ $a -eq 1 ]; then { echo "First true"; } else if [ $a -eq 2 ];

   then { echo "Second True"; } else { echo "None True"; } 

   fi

fi

[ "" == "" ] && {echo "True"} || {echo "False"}  #Is a shortend way of writing conditions. && represents 'THEN' block and || 'ELSE' block

• Switch case. note - closing circular braces after a value, and ;; as break statement while * for default case.

case $animal in

tiger ) echo "$animal is a wild animal." ;;

cow ) echo "$animal is a domestic animal." ;;

*) echo "$animal is unknown to me" ;;

esac

Logical Operators:

*AND operator is '-a' and OR is '-o' if multiple condition. e.g.

[ "209809" == "^[0-9]+$" -a 2 -gt 1 ] && echo "True" || echo "False"

[ "209809" == "^[0-9]+$" -o 2 -gt 1 ] && echo "True" || echo "False"

Comparison Operators: 

== #Used with text, for equality comparision

!= #Used with text, for non-equality comparision

-gt #Used with numbers, for Greater than check

-lt #Used with numbers, for Lesser than check

-le #Used with numbers, for Lesser than Or Equal check

-ge #Used with numbers, for Greater than Or Equal check

-eq #Used with numbers, for Equal check

-ne #Used with numbers, for Not Equal check

Other conditional switches:

*All switches can be prefixed with "! " for negative check.

*File conditions are : a,b,c,d,e,f,g,G,h,H,k,L,N,O,p,r,s,S,t,u,w,x

*String conditions : >,<,-n,-z,=~

*Miscellaneous conditions: -o

* && can also be used to put to commands together.eg- mkdir test && cd test

[ -f /tmp/a ] #returns true if file '/tmp/a' exists else false.

[ -d /tmp/a ] #returns true if '/tmp/a' is a directory else false.

[ file1 -nt file2 ] # newer than

[ file1 -ot file2 ] # older than

[ -z ] # checks is var is empty, returns true if empty else false.

Runtime value Input: Reading user inputs using read. -p - to prompt a message.

read -p "Enter your Name: " name ; #accepts input from user and store in 'name' variable.

Ouput : 

echo "Hello World" # prints hello world on screen. 

echo -e "rohit\tverma\n33" # prints rohit[TAB]verma and 25 on next line. -e=express/expand

echo -n "hello world" #prints hello world and remains on same line.

echo ${varName,,} #double comas prits value in varName is printed

echo ${varName^^} #double carat prints value in varName is printed in upper case

Arguments:

$0 #Command name itself

$1 #First argument after command, $N for rest arguments

$# #number of arguments supplied

$! #Process ID of last process

$@ #All arguments supplied

Function Declaration:

x(){

echo $#;

echo $0;

}

Function call:

x Rohit Verma

Print a sequence of numbers.

seq 0 1 10 #prints 0 to 10 at step 1

echo -e " 1rohit\n2ram\n 3rahul\n4robin\namit\nmanoj" | sort # sorts given sequence

seq 0 5 | sort -r # sorts sequence 0 to 5 in reverse order.

echo -e "abc\ndef\nab\nabc\nghs" | sort | uniq #unique sorted list.

echo Hello World | rev #reverse text.

cut usage.

*Default delimeter is \t TAB. Single Character delimeter can be specified using '-d' option

echo -e "Name=Rohit\tAge=31\nName=Rahul\tAge=33" | cut -f1 | cut -d'=' -f2

echo "This is a cat, and it mews." | cut -d' ' -f4,7 --output-delimiter=' sounds '

tr usage: Changing case of Text using tr command.

echo Hello | tr [:lower:] [:upper:] #Converts all lower case chars to upper case

echo Hello | tr [:upper:] [:lower:] #Converts all upper case chars to lower case

echo Hello | tr -s l #squeezes multiple occurrences of char to 1

echo "hello world" | tr -s " " #Squeeze spaces

Using Tee:

echo "Hello World" | tee /dev/tty | sed s/e/E/g

compgen -c # find find available commands and keywords.

cat /etc/shells #List all available shells.

Maths bc expr ;)

echo "5*3 + 3 - 19/5" | bc #return 17 dropping decimals.

echo "5*3 + 3 - 19/5" | bc -l #return decimal values up to 20.

echo "scale=3; 5*3 + 3 - 19/5;" | bc -l #return decimal values up to 3.

expr " 5 \* 3 + 3 - 19 \/ 5 "

Simple example (test.sh)

date > /tmp/dates

while [ .TRUE. ]

do

        date >> /tmp/dates

        sleep 2

done

# How to export variable in parent shell from within any script.. e.g.- I want to set JAVA_HOME and put same on PATH first for running jdk8 where jdk 7 is on default path.

1.    Create script file in /usr/bin named J8

    export JAVA_HOME=/opt/jdk1.8.0_161

    export PATH=$JAVA_HOME/bin:$PATH

2.    Run script as follows: chmod 777 J8 if required...

    . J8

#Grep and kill all process with name flume

kill -9 ` ps -ef |grep flume | tr -s " " | cut -d ' ' -f2 | tr '\n' ' '`