Quick Cheat Sheet for VM setup

Terminal tools and VPN: 

Windows: putty : https://www.putty.org/

  MAC/Linux: Terminal (default)   :  ssh, sftp

VPN tool:  Window/Linux: https://openvpn.net/    MAC : https://tunnelblick.net/

KUVPN: certificate:  https://vpn.ku.ac.th/

Hadoop installation

 install guide ubuntu for multinode 

https://www.edureka.co/blog/every-hadoop-component/

https://tecadmin.net/setup-hadoop-on-ubuntu/

suggestion: java JDK 1.8

adding the following in .bashrc:

 export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop

Create use sudo user hadoop 

Create user/sudoer

-Add user (do for both VM)

sudo adduser hadoop

-Make user sudoer

sudo usermod -aG sudo hadoop

Edit host list/Change hostname

-Change host name

sudo hostname <new-server-name-here>

-edit the /etc/hostname file and update hostname:

sudo  vi /etc/hostname

-edit the /etc/hosts file and update the lines that reads your old-host-name:

sudo  vi /etc/hosts

From:

127.0.0.1     old-host-name

To:

127.0.0.1     new-server-name-here

Save and close the file.

Make passwordless

-edit  /etc/host

add line

10.3.135.170 server1 # assume this one is your master ip

10.3.135.169 server2 # assume this

Create ssh key and make passwordless

After creating the account, it also requires to set up key-based ssh to its own account. To do this,

execute following commands. On the master node,

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

chmod 0600 ~/.ssh/authorized_keys

Now, SSH to localhost with hadoop user. This should not ask for the password but the first time it

will prompt for adding RSA to the list of known hosts.

you should be able to ssh to other nodes as well.

First, copy ssh key to every of nodes including itself. (10.3.135.170)

Assume your slave node is 10.3.135.169 using the command. Eg.

ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@server2

ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@server1

Open port and port forwarding

Sometimes, you may need to open firewall port  eg. 9870, 8088 of your VM to your localmachine.

- open firewall for the traffic to go through on the VM under the firewall.

-UFW:  command to allow firewall port

(https://www.itzgeek.com/how-tos/linux/ubuntu-how-tos/install-apache-hadoop-ubuntu-14-10-centos-7-single-node-cluster.html)

-perform port forward

https://www.ssh.com/ssh/tunneling/example

eg. for VM 10.3.134.8 

ssh  -N -L  9870:10.3.134.8:9870 hadoop@10.3.134.8 -vv

then you open webbrowser to see name node eg. 

http://localhost:9870

Disable ipv6 (make permanent)  

OR temporarily

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1

sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

sudo sysctl -w net.ipv6.conf.lo.disable_ipv6=1

sudo timedatectl set-timezone Asia/Bangkok