Hadoop contains the suit of stress testing tools in the distribution.
$ find /usr/lib/hadoop-mapreduce/ -name hadoop-*test*.jar
Find all tests
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar
TestDFSIO is designed to use 1 map task per file. (1:1 mapping from files to map tasks)
Write test
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 100
16/12/26 02:41:27 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/12/26 02:41:27 INFO fs.TestDFSIO: Date & time: Mon Dec 26 02:41:27 PST 2016
16/12/26 02:41:27 INFO fs.TestDFSIO: Number of files: 10
16/12/26 02:41:27 INFO fs.TestDFSIO: Total MBytes processed: 1000.0
16/12/26 02:41:27 INFO fs.TestDFSIO: Throughput mb/sec: 7.658316548856231
16/12/26 02:41:27 INFO fs.TestDFSIO: Average IO rate mb/sec: 10.4342679977417
16/12/26 02:41:27 INFO fs.TestDFSIO: IO rate std deviation: 7.105071350812148
16/12/26 02:41:27 INFO fs.TestDFSIO: Test exec time sec: 134.375
16/12/26 02:41:27 INFO fs.TestDFSIO:
Write speed significantly vary depending on the replication factor. Below are some indicative numbers
RF 1: 190 MB/s
RF 2: 25 MB/s
RF 3: 13 MB/s
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 100
16/12/26 02:47:44 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
16/12/26 02:47:44 INFO fs.TestDFSIO: Date & time: Mon Dec 26 02:47:44 PST 2016
16/12/26 02:47:44 INFO fs.TestDFSIO: Number of files: 10
16/12/26 02:47:44 INFO fs.TestDFSIO: Total MBytes processed: 1000.0
16/12/26 02:47:44 INFO fs.TestDFSIO: Throughput mb/sec: 46.15100609193281
16/12/26 02:47:44 INFO fs.TestDFSIO: Average IO rate mb/sec: 70.32638549804688
16/12/26 02:47:44 INFO fs.TestDFSIO: IO rate std deviation: 50.15305958356481
16/12/26 02:47:44 INFO fs.TestDFSIO: Test exec time sec: 72.964
16/12/26 02:47:44 INFO fs.TestDFSIO:
TeraSort - combines testing of hadoop and mapreduce layer
Iron Out hadoop configuration
Determine whether your mapreduce related parameters are set to proper values
This test is run in combination of three mapreduce jobs
teragen > terasort > teravalidate
Generate test data using teragen. (4 hours on 4 node)
$ find /usr/lib/hadoop-mapreduce/ -name hadoop-*examples*.jar
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen <number of records 10000000000 /user/cloudera/terasort/input
Sort data using terasort. (5 hours on 4 node)
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar terasort /user/cloudera/terasort/input /user/cloudera/terasort/output
Validate terasort output
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teravalidate /user/cloudera/terasort/output /user/cloudera/terasort/validate
LoadTest NameNode
Generate HDFS related requests with small payloads
Purpose: put a high HDFS management stress on the NameNode
Can simulate requests for creating, reading, renaming and deleting files
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar nnbench \
-operation create_file \
-maps 12 \
-reduces 6 \
-blockSize 1 \
-bytesToWrite 0 \
-numberOfFiles 1000 \
-replicationFactorPerFile 3 \
-readFileAfterOpen true \
-baseDir /user/cloudera/NNBnech-`hostname -s`
Loop small Job a number of times
Checks whether small job runs are responsive and running efficiently on the cluster
Complementary to terasort
Focus on mapreduce layer, impact on HDFS is limited
Run 50 small jobs
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar mrbench -baseDir /user/cloudera/MRbench -numRuns 50
Big Bench Test
https://github.com/intel-hadoop/Big-Data-Benchmark-for-Big-Bench
Reference:
http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/
http://www.slideshare.net/BartVandewoestyne/big-data-benchmarking
https://github.com/jbenninghoff/cluster-validation