MapR Sandbox

Sanity test the Sandbox

List the mapr cli service running on the MapR Sandbox.

[mapr@maprdemo ~]$ maprcli service list

logpath displayname name memallocated state

/opt/mapr/hbase/hbase-1.1.8/logs HBaseThriftServer hbasethrift Auto 2

/opt/mapr/httpfs/httpfs-1.0/logs Httpfs httpfs Auto 2

/opt/mapr/spark/spark-2.2.1/logs/ SparkHistoryServer spark-historyserver Auto 2

/opt/mapr/oozie/oozie-4.3.0/logs Oozie oozie Auto 2

/opt/mapr/hive/hive-2.1/logs/mapr HiveServer2 hs2 Auto 2

/opt/mapr/logs/cldb.log CLDB cldb 256.0 2

/opt/mapr/logs/hoststats.log HostStats hoststats Auto 2

/opt/mapr/hive/hive-2.1/logs/mapr HiveMetastore hivemeta Auto 2

/opt/mapr/logs/mfs.log FileServer fileserver 1609.0 2

/opt/mapr/hadoop/hadoop-2.7.0/logs ResourceManager resourcemanager 1073.0 2

/opt/mapr/hadoop/hadoop-2.7.0/logs JobHistoryServer historyserver 107.0 2

/opt/mapr/hue/hue-3.12.0/logs/ HueWebServer hue Auto 2

/opt/mapr/logs/nfsserver.log NFS Gateway nfs 4

/opt/mapr/hadoop/hadoop-2.7.0/logs NodeManager nodemanager 215.0 2

/opt/mapr/apiserver/logs/apiserver.log APIServer apiserver 1000.0 2

If you get the following error while listing the services, probably mapr-zookeeper service is not functioning properly.

$ maprcli service list

ERROR (10009) - Could not connect to CLDB and no Zookeeper connect string provided

Restart the services. Wait for 3 minutes before testing again for the service status.

$ sudo service mapr-zookeeper restart

$ sudo service mapr-warden restart

As you see nfs service has state = 4, which indicates the NFS service of MaprFS is down. You are restart the service as below.

$ maprcli node services -nodes maprdemo -name nfs -action restart

MCS is published by the apiserver. In MCS, which is available on https://localhost:8443 (note, it is https).

Run Spark in Jupyter notebook

Download an install anaconda for python 3.6+ (64 bit). Run as mapr user.

https://www.anaconda.com/download/#linux

wget https://repo.anaconda.com/archive/Anaconda3-5.2.0-Linux-x86_64.sh

bash Anaconda3-5.2.0-Linux-x86_64.sh -f -b

echo "export PATH=$HOME/anaconda3/bin:\$PATH" >> ~/.bashrc

source ~/.bashrc

Open a terminal and run the following commands

export PATH=/home/mapr/anaconda3/bin:$PATH

rm -rf ~/.jupyter/jupyter_notebook_config.py

jupyter notebook --generate-config

echo "c.NotebookApp.ip = '*'" >> ~/.jupyter/jupyter_notebook_config.py

echo "c.NotebookApp.open_browser = False" >> ~/.jupyter/jupyter_notebook_config.py

echo "c.NotebookApp.password = u'sha1:298cdb938f91:e6462632083df74cb2f68d4b55b0e7a7d7e0e45b'" >> ~/.jupyter/jupyter_notebook_config.py

mkdir -p ~/notebooks

cd ~/notebooks

jupyter notebook

Keep the terminal running. It runs the notebook server.

Do a port forwarding for port 8889. Open Virtual box settings > Network > NAT adapter advanced setting.

Open a browser on the host machine and open http://localhost:8889

Use password, use pass123 to login.

To get started with Spark with Jupyter following this doc.

https://blog.einext.com/apache-spark/pyspark-on-jupyter-notebook

import os, sys

SPARK_HOME = "/opt/mapr/spark/spark-2.2.1"

sys.path.insert(0, os.path.join(SPARK_HOME, "python", "lib", "py4j-0.10.4-src.zip"))

sys.path.insert(0, os.path.join(SPARK_HOME, "python"))

from pyspark.sql import SparkSession

spark = SparkSession.builder.enableHiveSupport().getOrCreate()

sc = spark.sparkContext

spark.read.text("file:///etc/passwd").show(10, False)

It should show an ouput something like below

+----------------------------------------------+

|value |

+----------------------------------------------+

|root:x:0:0:root:/root:/bin/bash |

|bin:x:1:1:bin:/bin:/sbin/nologin |

|daemon:x:2:2:daemon:/sbin:/sbin/nologin |

|adm:x:3:4:adm:/var/adm:/sbin/nologin |

|lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin |

|sync:x:5:0:sync:/sbin:/bin/sync |

|shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown |

|halt:x:7:0:halt:/sbin:/sbin/halt |

|mail:x:8:12:mail:/var/spool/mail:/sbin/nologin|

|operator:x:11:0:operator:/root:/sbin/nologin |

+----------------------------------------------+

only showing top 10 rows