Word Count Example Part II - Run a Jar on Hadoop

• Let's count the number of words in Shakespeare's Othello (attached below).
  • You may use your own document collection (any text file) for this job.
  • Upload your file to the server using FTP. (Click here for how to)
  • Connect to TACC (SSH) and submit a job, (Click here for how to: Log in to Longhorn | Submit a job request)
• Upload your file to HDFS

hr4757@c201-121:~/Project$ hadoop fs -put othello.txt othello.txt

hr4757@c201-121:~/Project$ hadoop fs -ls

Found 2 items

drwxr-xr-x - hr4757 supergroup 0 2011-01-26 16:04 /user/hr4757/Blog08_repack

-rw-r--r-- 2 hr4757 supergroup 156171 2011-01-26 16:44 /user/hr4757/othello.txt

• Find your jar file that you uploaded in PART I.
• Run wordcount
  • input file: othello.txt
  • output folder: othello

hr4757@c201-124:/scratch/01603/hr4757$ hadoop jar wordcount.jar org.myorg.WordCount othello.txt othello

11/02/19 14:59:18 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

11/02/19 14:59:18 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

11/02/19 14:59:18 INFO input.FileInputFormat: Total input paths to process : 1

11/02/19 14:59:19 INFO mapred.JobClient: Running job: job_201102191400_0001

11/02/19 14:59:20 INFO mapred.JobClient: map 0% reduce 0%

...

• Read the Result

hr4757@c201-121:~/hadoop$ hadoop fs -cat othello/*