When you're tasked with moving very large files over a high latency network, existing tcp tunable parameters need to be updated to get the best available bandwidth. The idea is to use very large frames when pushing large files over a long-distance network. Unfortunately, existing tools such as ftp or scp don't take advantage of increased buffer sizes. In fact, you'll generally see pretty poor performance when pushing GB sized files over a long-distance network.
The solution is to update your TCP kernel parameters, increase the maximum buffer size, and to use a file transfer tool that allows you to specify the send and receive buffer sizes. The MonAlisa project has made available tools for grid computing. In particular, a program, fdt.jar, Fast Data Transfer, will allow to you move very large files at lightning speed over long-distance networks.
Put these lines into sysctl.conf, then run sysctl -p
Ensure you update both the server and the client system with these values.
# increase TCP max buffer size
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# increase Linux autotuning TCP buffer limits
# min, default, and max number of bytes to use
# set max to at least 4MB, or higher if you use very high BDP paths (Bandwidth Delay Products)
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
"I obtained the following data from the monalisa project tuning page"..........................
We suggest to use newer linux distributions, or if this is not possible, update at least the kernel(2.6.20+). The newer kernels provide adequate TCP settings. We suggest to use the following settings to improve the TCP throughput, especially over long RTT links:
Increase the TCP buffers (newer kernels have this buffers increased byy default). Add the following lines in /etc/sysctl.conf to make the changes permanent accross reboots:
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 65536 8388608
net.core.netdev_max_backlog = 250000
After adding them just run the following commabd as root:
#sysctl -p /etc/sysctl.conf
The settings above will set a maximum of 8 MBytes buffers.
we suggest to use at least 4Mbytes maximum TCP buffers and a maximum of 16Mbytes should be enough. You should use reasonable values. Don't set very high values for this parameters, and especially don't set the same value for all the fields in net.ipv4.tcp_*. It's also a good practice to have the same value for net.core.r(w)mem_max with the last value in the net.ipv4.tcp_r(w)mem. Do not modify the net.ipv4.tcp_mem parameter. It is computed by the kernel at the system boot.
The TCP congestion protocol (if available).
To check if it is available for your kernel version:
$/sbin/sysctl net.ipv4.tcp_congestion_control
Set it to cubic if kernel version 2.6.20+, and to scalable if older kernels.
#sysctl -w net.ipv4.tcp_congestion_control=cubic
To make this persistent accross reboots add the following line in /etc/sysctl.conf
net.ipv4.tcp_congestion_control=cubic
You may try to experiment various TCP stacks. You can list all of them using:
$/sbin/sysctl net.ipv4.tcp_available_congestion_control
Increase txqueuelen size your ethernet card
#ifconfig eth2 txqueuelen 50000
If your network infrastructure supports jumbo frames you may set the MTU size to 9000. Please notice that this setting might broke your AFS installation (newer versions of OpenAFS supports jumbo frames)
#ifconfig eth2 mtu 9000
You may also try to disable the TCP timestamps. On some kernel versions this setting disables the automatic window scalling:
#sysctl -w net.ipv4.tcp_timestamps=0
To make this setting permanent add the following line in /etc/sysctl.conf:
net.ipv4.tcp_timestamps=0
Obtain the ftd.jar program from http://monalisa.cern.ch/FDT/download.html
Invoke the server:
# java -jar ./fdt.jar -XX:MaxDirectMemorySize=512m -bs 16M
NOTE: You will need at least a 1.5 jre
Invoke the client:
# java -jar ./fdt.jar -XX:MaxDirectMemorySize=512m -c RemoteServerName Filename-2-Send -d Directory-2-store-in-on-Remote
This will put the file using 16MB buffers. You should be able to achive significant improvement over using just plain old scp.
Here is an example dialog:
On the server (remote) system:
(Startup):
===========================================================================
[root@scmdr-lnx svn_copy]# java -jar /root/fdt.jar -XX:MaxDirectMemorySize=512m -bs 16M
FDT [ 0.9.6-200912150642 ] STARTED ...
Dec 22, 2009 2:04:53 PM lia.util.net.common.Config <init>
INFO: FDT started in server mode
READY
Dec 22, 2009 2:04:54 PM lia.util.net.copy.FDTServer doWork
INFO: FDTServer start listening on port: 54321
=============================================================================
Dec 22, 2009 2:06:36 PM lia.util.net.copy.transport.ControlChannel initStreams
INFO: NEW CONTROL stream for 0e30e2cc-8d2a-4901-a750-0a74b8cf39b3 initialized
Dec 22, 2009 2:06:36 PM lia.util.net.copy.transport.ControlChannel$ControlChannelPingerTask <init>
INFO: [ ControlChannelPingerTask ] initialized
Dec 22, 2009 2:06:36 PM lia.util.net.copy.FDTWriterSession doPreprocess
INFO: [ FDTWriterSession ] Preprocessing started
Dec 22, 2009 2:06:36 PM lia.util.net.copy.FDTWriterSession doPreprocess
INFO: [ FDTWriterSession ] No pre processing filters defined/processed.
Dec 22, 2009 2:06:43 PM lia.util.net.copy.disk.DiskWriterTask run
INFO: DiskWriterTask [ partitionID=267438073, writerID= 0, tid=11 ] since: Tue Dec 22 14:06:43 EST 2009 STARTED. hasToRun() = true
22/12 14:06:46 Net In: 93.279 Mb/s Avg: 93.279 Mb/s
22/12 14:06:51 Net In: 89.456 Mb/s Avg: 91.358 Mb/s
22/12 14:06:56 Net In: 82.288 Mb/s Avg: 88.335 Mb/s
22/12 14:07:01 Net In: 60.975 Mb/s Avg: 81.491 Mb/s 08.59% ( 03m 12s )
22/12 14:07:06 Net In: 65.460 Mb/s Avg: 78.284 Mb/s 10.16% ( 03m 17s )
22/12 14:07:11 Net In: 83.228 Mb/s Avg: 79.108 Mb/s 11.72% ( 03m 11s )
22/12 14:07:16 Net In: 89.539 Mb/s Avg: 80.598 Mb/s 14.84% ( 03m 01s )
22/12 14:07:21 Net In: 66.058 Mb/s Avg: 78.778 Mb/s 17.19% ( 03m 00s )
22/12 14:07:26 Net In: 85.448 Mb/s Avg: 79.520 Mb/s 18.75% ( 02m 55s )
22/12 14:07:31 Net In: 72.267 Mb/s Avg: 78.794 Mb/s 22.66% ( 02m 48s )
22/12 14:07:36 Net In: 95.594 Mb/s Avg: 80.320 Mb/s 24.22% ( 02m 42s )
22/12 14:07:41 Net In: 86.201 Mb/s Avg: 80.810 Mb/s 27.34% ( 02m 34s )
22/12 14:07:46 Net In: 86.263 Mb/s Avg: 81.230 Mb/s 29.69% ( 02m 28s )
22/12 14:07:51 Net In: 77.220 Mb/s Avg: 80.942 Mb/s 32.81% ( 02m 22s )
22/12 14:07:56 Net In: 68.073 Mb/s Avg: 80.084 Mb/s 33.59% ( 02m 22s )
22/12 14:08:01 Net In: 90.759 Mb/s Avg: 80.751 Mb/s 36.72% ( 02m 14s )
22/12 14:08:06 Net In: 85.660 Mb/s Avg: 81.040 Mb/s 39.06% ( 02m 09s )
22/12 14:08:11 Net In: 77.364 Mb/s Avg: 80.835 Mb/s 41.41% ( 02m 04s )
22/12 14:08:16 Net In: 64.431 Mb/s Avg: 79.972 Mb/s 42.97% ( 02m 02s )
22/12 14:08:21 Net In: 79.900 Mb/s Avg: 79.967 Mb/s 46.09% ( 01m 55s )
22/12 14:08:26 Net In: 100.863 Mb/s Avg: 80.965 Mb/s 48.44% ( 01m 49s )
22/12 14:08:31 Net In: 117.987 Mb/s Avg: 82.647 Mb/s 52.34% ( 01m 39s )
22/12 14:08:36 Net In: 77.818 Mb/s Avg: 82.437 Mb/s 53.91% ( 01m 36s )
22/12 14:08:41 Net In: 68.082 Mb/s Avg: 81.839 Mb/s 56.25% ( 01m 31s )
22/12 14:08:46 Net In: 98.318 Mb/s Avg: 82.498 Mb/s 59.38% ( 01m 24s )
22/12 14:08:51 Net In: 103.788 Mb/s Avg: 83.316 Mb/s 61.72% ( 01m 18s )
22/12 14:08:56 Net In: 84.274 Mb/s Avg: 83.351 Mb/s 64.84% ( 01m 12s )
22/12 14:09:01 Net In: 91.118 Mb/s Avg: 83.629 Mb/s 66.41% ( 01m 09s )
22/12 14:09:06 Net In: 106.931 Mb/s Avg: 84.432 Mb/s 70.31% ( 01m 00s )
22/12 14:09:11 Net In: 84.295 Mb/s Avg: 84.427 Mb/s 72.66% ( 55s )
22/12 14:09:16 Net In: 61.808 Mb/s Avg: 83.697 Mb/s 74.22% ( 52s )
22/12 14:09:21 Net In: 83.744 Mb/s Avg: 83.699 Mb/s 77.34% ( 46s )
22/12 14:09:26 Net In: 135.534 Mb/s Avg: 85.269 Mb/s 80.47% ( 39s )
22/12 14:09:31 Net In: 95.080 Mb/s Avg: 85.557 Mb/s 82.81% ( 34s )
Dec 22, 2009 2:09:32 PM lia.util.net.copy.FDTWriterSession handleEndFDTSession
INFO: [ FDTWriterSession ] Remote FDTReaderSession for session [ 0e30e2cc-8d2a-4901-a750-0a74b8cf39b3 ] finished ok. Waiting for our side to finish.
22/12 14:09:36 Net In: 98.782 Mb/s Avg: 85.935 Mb/s 87.50% ( 24s )
22/12 14:09:41 Net In: 84.263 Mb/s Avg: 85.888 Mb/s 89.06% ( 21s )
22/12 14:09:46 Net In: 68.740 Mb/s Avg: 85.425 Mb/s 90.62% ( 18s )
22/12 14:09:51 Net In: 84.013 Mb/s Avg: 85.388 Mb/s 92.97% ( 14s )
22/12 14:09:56 Net In: 81.530 Mb/s Avg: 85.289 Mb/s 95.31% ( 09s )
22/12 14:10:01 Net In: 73.370 Mb/s Avg: 84.990 Mb/s 98.44% ( 03s )
22/12 14:10:06 Net In: 17.715 Mb/s Avg: 83.350 Mb/s 100.00% ( 00s )
FDTWriterSession ( 0e30e2cc-8d2a-4901-a750-0a74b8cf39b3 ) final stats:
Started: Tue Dec 22 14:06:36 EST 2009
Ended: Tue Dec 22 14:10:09 EST 2009
Transfer period: 03m 33s
TotalBytes: 2147483648
TotalNetworkBytes: 2147483648
Exit Status: OK
Dec 22, 2009 2:10:09 PM lia.util.net.copy.FDTWriterSession doPostProcessing
INFO: [ FDTWriterSession ] Post Processing started
Dec 22, 2009 2:10:09 PM lia.util.net.copy.FDTWriterSession doPostProcessing
INFO: [ FDTWriterSession ] No post processing filters defined/processed.
Dec 22, 2009 2:10:09 PM lia.util.net.copy.transport.ControlChannel run
INFO: ControlThread for ( 0e30e2cc-8d2a-4901-a750-0a74b8cf39b3 ) /172.24.194.70:38430 FINISHED
===============================================================
Client Side (local - sender):
===============================================================
[root@ldap2-lnx ~]# java -jar /root/fdt.jar -XX:MaxDirectMemorySize=512m -c scmdr-lnx.intdata.com data -d /svn_copy
FDT [ 0.9.6-200912150642 ] STARTED ...
Dec 22, 2009 2:06:56 PM lia.util.net.common.Config <init>
INFO: FDT started in client mode
Dec 22, 2009 2:06:57 PM lia.util.net.copy.transport.ControlChannel initStreams
INFO: NEW CONTROL stream for 0e30e2cc-8d2a-4901-a750-0a74b8cf39b3 initialized
Dec 22, 2009 2:06:57 PM lia.util.net.copy.transport.ControlChannel$ControlChannelPingerTask <init>
INFO: [ ControlChannelPingerTask ] initialized
Dec 22, 2009 2:06:57 PM lia.util.net.copy.FDTReaderSession internalInit
INFO: [ FDTReaderSession ] No pre processing filters defined/processed.
Dec 22, 2009 2:06:58 PM lia.util.net.copy.FDTReaderSession startReading
INFO: Started DiskReaderTasks for the following partions [ 26627 ] for FDTSession: 0e30e2cc-8d2a-4901-a750-0a74b8cf39b3
Dec 22, 2009 2:06:58 PM lia.util.net.copy.transport.TCPTransportProvider tryToConnect
INFO: bSockConn: 30 bSockConnWait: 1500
Dec 22, 2009 2:06:58 PM lia.util.net.copy.transport.TCPTransportProvider tryToConnect
INFO: Requested window size -1. Using window size: 32768
22/12 14:07:07 Net Out: 97.818 Mb/s Avg: 97.818 Mb/s
22/12 14:07:12 Net Out: 89.621 Mb/s Avg: 93.718 Mb/s
22/12 14:07:17 Net Out: 84.484 Mb/s Avg: 90.641 Mb/s 06.25% ( 02m 57s )
22/12 14:07:22 Net Out: 58.394 Mb/s Avg: 82.572 Mb/s 08.59% ( 03m 10s )
22/12 14:07:27 Net Out: 69.428 Mb/s Avg: 79.942 Mb/s 10.16% ( 03m 13s )
22/12 14:07:32 Net Out: 83.063 Mb/s Avg: 80.463 Mb/s 12.50% ( 03m 06s )
22/12 14:07:37 Net Out: 90.734 Mb/s Avg: 81.929 Mb/s 15.62% ( 02m 56s )
22/12 14:07:42 Net Out: 65.368 Mb/s Avg: 79.857 Mb/s 17.97% ( 02m 56s )
22/12 14:07:47 Net Out: 84.729 Mb/s Avg: 80.398 Mb/s 20.31% ( 02m 50s )
22/12 14:07:52 Net Out: 72.550 Mb/s Avg: 79.613 Mb/s 22.66% ( 02m 46s )
22/12 14:07:57 Net Out: 95.866 Mb/s Avg: 81.089 Mb/s 25.00% ( 02m 38s )
22/12 14:08:02 Net Out: 86.707 Mb/s Avg: 81.558 Mb/s 28.12% ( 02m 31s )
22/12 14:08:07 Net Out: 86.707 Mb/s Avg: 81.954 Mb/s 29.69% ( 02m 27s )
22/12 14:08:12 Net Out: 73.799 Mb/s Avg: 81.370 Mb/s 32.81% ( 02m 21s )
22/12 14:08:17 Net Out: 72.238 Mb/s Avg: 80.761 Mb/s 34.38% ( 02m 19s )
22/12 14:08:22 Net Out: 88.268 Mb/s Avg: 81.230 Mb/s 36.72% ( 02m 13s )
22/12 14:08:27 Net Out: 85.562 Mb/s Avg: 81.484 Mb/s 39.06% ( 02m 08s )
22/12 14:08:32 Net Out: 80.982 Mb/s Avg: 81.456 Mb/s 42.19% ( 02m 01s )
22/12 14:08:37 Net Out: 61.009 Mb/s Avg: 80.379 Mb/s 42.97% ( 02m 01s )
22/12 14:08:42 Net Out: 80.982 Mb/s Avg: 80.409 Mb/s 46.09% ( 01m 55s )
22/12 14:08:47 Net Out: 100.598 Mb/s Avg: 81.370 Mb/s 49.22% ( 01m 47s )
22/12 14:08:52 Net Out: 116.740 Mb/s Avg: 82.978 Mb/s 53.12% ( 01m 37s )
22/12 14:08:57 Net Out: 77.755 Mb/s Avg: 82.751 Mb/s 55.47% ( 01m 32s )
22/12 14:09:02 Net Out: 68.907 Mb/s Avg: 82.173 Mb/s 56.25% ( 01m 31s )
22/12 14:09:07 Net Out: 99.301 Mb/s Avg: 82.859 Mb/s 59.38% ( 01m 24s )
22/12 14:09:12 Net Out: 104.089 Mb/s Avg: 83.675 Mb/s 61.72% ( 01m 18s )
22/12 14:09:17 Net Out: 84.417 Mb/s Avg: 83.702 Mb/s 64.84% ( 01m 12s )
22/12 14:09:22 Net Out: 91.911 Mb/s Avg: 83.995 Mb/s 67.97% ( 01m 05s )
22/12 14:09:27 Net Out: 106.067 Mb/s Avg: 84.756 Mb/s 70.31% ( 01m 00s )
22/12 14:09:32 Net Out: 82.543 Mb/s Avg: 84.682 Mb/s 73.44% ( 53s )
22/12 14:09:37 Net Out: 61.621 Mb/s Avg: 83.938 Mb/s 74.22% ( 52s )
22/12 14:09:42 Net Out: 85.353 Mb/s Avg: 83.982 Mb/s 77.34% ( 46s )
22/12 14:09:47 Net Out: 135.629 Mb/s Avg: 85.547 Mb/s 81.25% ( 37s )
22/12 14:09:52 Net Out: 95.034 Mb/s Avg: 85.826 Mb/s 84.38% ( 31s )
22/12 14:09:57 Net Out: 97.844 Mb/s Avg: 86.169 Mb/s 87.50% ( 24s )
22/12 14:10:02 Net Out: 85.457 Mb/s Avg: 86.149 Mb/s 89.06% ( 21s )
22/12 14:10:07 Net Out: 69.115 Mb/s Avg: 85.689 Mb/s 92.19% ( 15s )
22/12 14:10:12 Net Out: 83.688 Mb/s Avg: 85.636 Mb/s 93.75% ( 12s )
22/12 14:10:17 Net Out: 83.063 Mb/s Avg: 85.570 Mb/s 96.09% ( 07s )
22/12 14:10:22 Net Out: 64.347 Mb/s Avg: 85.039 Mb/s 99.22% ( 01s )
Dec 22, 2009 2:10:25 PM lia.util.net.copy.FDTReaderSession handleEndFDTSession
INFO: [ FDTReaderSession ] Remote FDTWriterSession for session [ 0e30e2cc-8d2a-4901-a750-0a74b8cf39b3 ] finished OK!
22/12 14:10:27 Net Out: 7.447 Mb/s Avg: 83.146 Mb/s 100.00% ( 00s )
FDTReaderSession ( 0e30e2cc-8d2a-4901-a750-0a74b8cf39b3 ) final stats:
Started: Tue Dec 22 14:06:57 EST 2009
Ended: Tue Dec 22 14:10:28 EST 2009
Transfer period: 03m 31s
TotalBytes: 2147483648
TotalNetworkBytes: 2147483648
Exit Status: OK
Dec 22, 2009 2:10:28 PM lia.util.net.copy.FDTReaderSession doPostProcessing
INFO: [ FDTReaderSession ] Post Processing started
Dec 22, 2009 2:10:28 PM lia.util.net.copy.FDTReaderSession doPostProcessing
INFO: [ FDTReaderSession ] No post processing filters defined/processed.
[ Tue Dec 22 14:10:29 EST 2009 ] - GracefulStopper hook started ... Waiting for the cleanup to finish
[ Tue Dec 22 14:10:29 EST 2009 ] - GracefulStopper hook finished!
[ Tue Dec 22 14:10:29 EST 2009 ] FDT Session finished OK.
[root@ldap2-lnx ~]#
FDT Documentation:
FDT can be used in one of these three modes:
Server: java -jar fdt.jar [ OPTIONS ]
Client: java -jar fdt.jar [ OPTIONS ] -c <host> [file1 ...]|[-fl <fileList>] -d <destinationDirectory>
SCP: java -jar fdt.jar [ OPTIONS ] [[[user@][host1:]]file1 [[[user@][host2:]]file2
In Server mode the FDT will start listening for incoming client connections. The server may or may not stop after the last client finishes the transfer. In Client mode the client will connect to the specified host, where an FDT Server is expected to be running. The client can either read or write file from/to the server.
In the SCP (Secure Copy) mode the local FDT instance will use SSH to start/stop the FDT server and/or client. The security is based on ssh credentials. The server started in this mode will accept connections ONLY from the "SCP" client. It is possible to restrict the access for the FDT Servers started from the command line using the -f option. The option accepts a list of IP addresses separated by ':'.
The OPTIONS currently supported may be server or client specific, or may be used in both modes.
Common options used for both server and client:
-gsi enables GSI authentication scheme in FDT. When started in server mode the FDT will open two TCP ports: one for GSI authentication and the other one for data channels
-gsip <GSICtrlPort> specifies the TCP port used for GSI authentication. Default value is 54320.
-p <portNo> specifies the TCP port to be used (for the server it is the port used to listen on; for the client the port to connect to). The default port is 54321.
-preFilters f1,...,fn User defined preProcessing filters. The classes specified by the f1,...,fn paramenters must be in the classpath. The prePRocessing filters must be defined in the FDT "sender" command line. Please see the User's Filters section for more details and examples.
-postFilters f1,...,fn User defined postProcessing filters. The classes specified by the f1,...,fn paramenters must be in the classpath. The postPRocessing filters must be defined in the FDT "receiver" command line. Please see the User's Filters section for more details and examples.
-bio Blocking I/O mode. n this mode every channel (socket) will be configured to send/receive data synchronously and FDT will use one thread per channel. By default, non-blocking I/O will be used. On some platforms/systems the throughtput can be slightly higher in blocking I/O mode. The limitation in the blocking mode is the maximum number of threads that can be used and, for very high numbers of streams (thousands), the CPU used by the kernel for scheduling the threads. By default, FDT will use non-blocking mode.
-iof <iof> Non-blocking I/O retry factor. In non-blocking mode every read/write operation which returns 0, will be repeated up to <iof> times before waiting for I/O readiness. By default this value is set to 1, which means that every network read/write operation will return in the select() (which can also be poll()/epoll()) if no more data can be processed by the underlying channel(socket). The default value should work fine on most of the systems, but values of 2 or 3, may increase the throughput on some systems. Values higher than 5 will only increase the CPU system usage, without any gain in performance.
-limit <rate> Restrict the transfer speed at the specified rate. K (KiloBytes/s), M (MegaBytes/s) or G (GigaBytes/s) may be used as suffixes. When this parameter is specified in the server it represents the maximum transfer rate for every FDT session. If the parameter is specified in both the server and the client, the minimum value between them will be used.
-md5 Enables MD5 checksum for every file involved in the transfer. The flag may be specified for both client and server, but it will be used by the "sender" session only. When the transfer finishes the list will be sent to the "receiver" and it will be printed in a `md5sum`-like mode
-printStats Various statistics about buffer pools, sessions, etc will be printed
-v Verbose. Multiple 'v'-s (up to three) may be used to increment the verbosity level.Maximum level is three (-vvv) which corresponds to Level.FINEST for the standard Java logging system used by FDT.
-u, -update Update. If a newer version of fdt.jar is available on the update server it will update the local copy
Server options:
-S disable the standalone mode; when specified the FDT Server will stop after the last client finishes. By default, the server will continue to listen for incoming clients. This option is automatically passed to the server started in "SCP" mode.
-bs <buffSize> Size for the I/O buffers. K (KiloBytes) or M (MegaBytes) may be used as suffixes. The default value is 512K. If the number of clients or sockets is expected to be very high is better to decrease this value. The memory used by this buffers is directly mapped in the operating system memory pages. The memory used by this buffers is limited by the JVM and can be increased passing -XX:MaxDirectMemorySize=<X>m (e.g -XX:MaxDirectMemorySize=256m) to the 'java' command
-f <allowedIPsList> A list of IP addresses allowed to connect to the server. Multiple IPs may be specified, separated by ':'
Client options:
-c <host> connect to the specified host. If this parameter is missing the FDT will become server
-gsissh used in the Secure Copy Mode to specify GSI authentication instead of normal SSH authentication scheme. The remote sshd servere must support GSI authentication.
-d <dstDir> The destination directory used to copy files.
-fl <fileList> a list of files. Must have only one file per line.
-pull Pull mode. The client will receive the data from the server.
-N disable Nagle algorithm
-ss <wsz> Set the TCP SO_SND_BUFFER size. M and K may be used as suffixes for Kilo/Mega.
-P <noOfStreams> Number of paralel streams to use. Default is 1.