Simple recipe to run Hadoop at TACC

The examples given before show you how to obtain a Hadoop cluster instance, how to upload files to the parallel file system (in your home directory and /scratch), and how to run word count on the login node. This is a simple recipe that pulls from each of these steps to direct you explicitly through how to go from zero to using an actual cluster instance to do word count on a file of interest to you.

Simple recipe to run Hadoop at TACC.

Login to Longhorn: simply ssh to longhorn.tacc.utexas.edu

Upload your files (jar and data files) to $SCRATCH using a FTP client

Submit your job using qsub

How do I configure my job?
How do I use qsub?
How do I know if my job is accepted and running?
How do I terminate my job?
- How do I log in to the new hadoop cluster instance without VNC?
  - $ qstat
  - job-ID prior name user state submit/start at queue slots ja-task-ID
  - -----------------------------------------------------------------------------------------------------------------
  - 55727 1.01472 Hadoop-tes jbaldrid r 02/23/2011 09:35:17 hadoop@c202-102.longhorn 32
  - $ ssh c202-102

Try hadoop dfsadmin -report to check if you have got all the nodes without errors.

To view your HADOOP status, VNC to TACC (optional).
- How do I VNC to TACC?
Copy your files from $SCRATCH to HDFS.
- command to change directory to $SCRATCH: cds
- upload command: hadoop fs -put <local files> <HDFS destination>
Run Hadoop: hadoop jar <jar file> <class name> <arguments>
Read your result
- How to read binary result in text format
- download command: hadoop fs -get <HDFS files> <Local destination>

For all you usage allocation information, you may log on to the TACC Longhorn Portal.

Page updated

Google Sites

Report abuse