Embedded Files
source: http://hanishblogger.blogspot.com/2012/12/how-to-run-examples-of-hadoop.html
In hadoop-version directory there is always a jar containing all examples. We can run them directly by passing required arguments.
Let say we have hadoop-3.2.1 then there is jar of all examples i.e. /home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar
or in cloudera
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar
<https://www.cloudera.com/documentation/other/tutorial/CDH5/topics/ht_usage.html>
<https://www.cloudera.com/documentation/other/tutorial/CDH5/topics/Hadoop-Tutorial.html>
Download any text file for giving it as input to wordcount program. Copy this file into any directory lets say it is: /home/hadoop/dft/4300.txt
$ cd /home/hadoop
$ mkdir dft
$ cd dft
$ cp /home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar .
$ wget https://raw.githubusercontent.com/bwhite/dv_hadoop_tests/master/python-streaming/word_count/input/4300.txt
Now since this data on your local machine so copy these data files into HDFS. Run the following command:
$ hdfs dfs -copyFromLocal /home/hadoop/dft/4300.txt /dft
** make sure you have created /dft in hadoop file system first.
$ hdfs dfs -mkdir /dft
Check the contents of your directory:
$ hdfs dfs -ls /dft
$ hdfs dfs -cat /dft/4300.txt
Please edit /home/hadoop/hadoop/etc/hadoop/mapred-site.xml
to contain the following path:
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property> <name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property> <name>mapreduce.application.classpath</name> <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/common/*,$HADOOP_MAPRED_HOME/share/hadoop/common/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/yarn/*,$HADOOP_MAPRED_HOME/share/hadoop/yarn/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/hdfs/*,$HADOOP_MAPRED_HOME/share/hadoop/hdfs/lib/*</value> </property>
<property> <name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property> <name>mapreduce.map.env</name> <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value> </property>
<property> <name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
</configuration>
Now run your program:
$ cd /home/hadoop/dft
$ hadoop jar ./hadoop-mapreduce-examples-3.2.1.jar wordcount /dft /dft-output
15/04/25 17:34:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/04/25 17:34:28 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
15/04/25 17:34:31 INFO input.FileInputFormat: Total input paths to process : 1
15/04/25 17:34:31 INFO mapreduce.JobSubmitter: number of splits:1
15/04/25 17:34:32 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1429946598372_0001
15/04/25 17:34:32 INFO impl.YarnClientImpl: Submitted application application_1429946598372_0001
15/04/25 17:34:32 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1429946598372_0001/
15/04/25 17:34:32 INFO mapreduce.Job: Running job: job_1429946598372_0001
^C[hadoop@localhost ~]$ hadoop jar ./hadoop-mapreduce-examples-2.6.jar wordcount /dft /dft-output
15/04/25 17:34:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/04/25 17:34:50 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
15/04/25 17:34:55 INFO input.FileInputFormat: Total input paths to process : 1
15/04/25 17:34:56 INFO mapreduce.JobSubmitter: number of splits:1
15/04/25 17:34:56 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1429946598372_0002
15/04/25 17:34:57 INFO impl.YarnClientImpl: Submitted application application_1429946598372_0002
15/04/25 17:34:57 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1429946598372_0002/
15/04/25 17:34:57 INFO mapreduce.Job: Running job: job_1429946598372_0002
15/04/25 17:35:59 INFO mapreduce.Job: Job job_1429946598372_0002 running in uber mode : false
15/04/25 17:35:59 INFO mapreduce.Job: map 0% reduce 0%
15/04/25 17:36:12 INFO mapreduce.Job: map 100% reduce 0%
15/04/25 17:36:27 INFO mapreduce.Job: map 100% reduce 100%
15/04/25 17:36:28 INFO mapreduce.Job: Job job_1429946598372_0002 completed successfully
15/04/25 17:36:28 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=725025
FILE: Number of bytes written=1661195
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1573143
HDFS: Number of bytes written=527522
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=10875
Total time spent by all reduces in occupied slots (ms)=11743
Total time spent by all map tasks (ms)=10875
Total time spent by all reduce tasks (ms)=11743
Total vcore-seconds taken by all map tasks=10875
Total vcore-seconds taken by all reduce tasks=11743
Total megabyte-seconds taken by all map tasks=11136000
Total megabyte-seconds taken by all reduce tasks=12024832
Map-Reduce Framework
Map input records=33055
Map output records=267975
Map output bytes=2601773
Map output materialized bytes=725025
Input split bytes=99
Combine input records=267975
Combine output records=50091
Reduce input groups=50091
Reduce shuffle bytes=725025
Reduce input records=50091
Reduce output records=50091
Spilled Records=100182
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=323
CPU time spent (ms)=4940
Physical memory (bytes) snapshot=269115392
Virtual memory (bytes) snapshot=4150345728
Total committed heap usage (bytes)=137498624
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1573044
File Output Format Counters
Bytes Written=527522
Getting the Output:
$ hdfs dfs -ls /
$ hdfs dfs -ls /dft-output
Finally, let's take a look at the output
$ hdfs dfs -cat /dft-output/part-r-00000 | less
If we wanted to copy the output file to our local storage (remember, the output is automatically created in the HDFS world, and we have to copy the data from there to our file system to work on it):
$ hdfs dfs -copyToLocal /dft-output/part-r-00000 .
To remove the output directory from hdfs
$ hdfs dfs -rm -r /dft-output
Note: The hadoop program WordCount will not run another time if the output directory exists. It always wants to create a new one, so we'll have to remove the output directory regularly after having saved the output of each job.
You may want to start job history server:
mr-jobhistory-daemon.sh start historyserver
Compile Wordcount
(https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#Source+Code)
Make sure you setup the following in .bashrc
export HADOOP_HOME=/home/hadoop/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export JAVA_HOME="$(jrunscript -e 'java.lang.System.out.println(java.lang.System.getProperty("java.home"));')"
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
export CLASSPATH=./
export CLASSPATH=$CLASSPATH:`hadoop classpath`:.:
export PATH=$JAVA_HOME/bin:$PATH
export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar
Then compile WordCount.java
$mkdir wordcount_classes
$javac -d wordcount_classes WordCount.java
$ jar cvf wc.jar -C wordcount_classes/ .
Run it
hadoop jar wc.jar WordCount /dft /dft-output2
Other examples:
(/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar)
For running hadoop's pi example:
(wget https://code.google.com/p/mrs-mapreduce/source/browse/examples/pi/Pi.java)
Enter below command:
# hadoop jar hadoop-mapreduce-examples-3.2.1.jar pi 10 10
Note: last two are arguments which define numbers of map task and reduce task respectively. We can specify any no.'s here.
For running hadoop's sudoku example:
Enter below command:
# bin/hadoop jar hadoop-mapreduce-examples-3.2.1.jar sudoku puzzle1.dta
In mac:
(https://kimlogan.github.io/2016/01/01/first-hadoop-program/)
With mapreduce in python, you create data input directory.
hadoop dfs -mkdir /data
hadoop dfs -put 4300.txt /data
Then create mapper.py and reducer.py
then run:
sudo -u cchantra hadoop jar /usr/local/Cellar/hadoop/3.1.1/libexec/share/hadoop/tools/lib/hadoop-streaming-3.1.1.jar -input /data -output /output -mapper mapper.py -reducer reducer.py -numReduceTasks 1
2019-01-05 16:58:45,881 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-01-05 16:58:46,998 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2019-01-05 16:58:47,177 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2019-01-05 16:58:47,177 INFO impl.MetricsSystemImpl: JobTracker metrics system started
2019-01-05 16:58:47,205 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2019-01-05 16:58:47,633 INFO mapred.FileInputFormat: Total input files to process : 1
2019-01-05 16:58:47,724 INFO mapreduce.JobSubmitter: number of splits:1
2019-01-05 16:58:48,003 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local562376939_0001
2019-01-05 16:58:48,005 INFO mapreduce.JobSubmitter: Executing with tokens: []
2019-01-05 16:58:48,225 INFO mapreduce.Job: The url to track the job: http://localh
:
Map-Reduce Framework
Map input records=33055
Map output records=267975
Map output bytes=2065823
Map output materialized bytes=2601785
Input split bytes=87
Combine input records=0
Combine output records=0
Reduce input groups=50091
Reduce shuffle bytes=2601785
Reduce input records=267975
Reduce output records=50091
Spilled Records=535950
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=26
Total committed heap usage (bytes)=802160640
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1573044
File Output Format Counters
Bytes Written=527522
2019-01-05 16:58:52,264 INFO streaming.StreamJob: Output directory: /output
Then see output:
sudo -u cchantra hadoop dfs -cat /output/part*
:
zodiacal 2
zoe)_ 1
zones: 1
zoo. 1
zoological 1
zouave's 1
zrads, 2
Page updated
Google Sites
Report abuse