An optimized approach for network throughput in virtualized environment

What is the exact problem?

Remember the last time when you created a virtual machine(VM). VM runs on a hyper-visor which is an abstraction layer between the VM and the bare metal(hardware). In turn, VM will not able to access physical NIC card directly. Instead, there will be a virtual NIC which maps to the physical NIC. So, there is extra virtual layer which is introduced here (refer below pic).

Note that for outbound traffic, data will be first processed by virtual NIC and then it will be processed by physical NIC. Moreover, data will be copied twice: One at virtual NIC buffer and other at physical NIC.

In summary, it will significantly reduce the network throughput. Intel does extensive testing and observed that using only device emulation, a 10Gbps Ethernet controller can achieve a maximum throughput of 4.5 to around 6.5 Gbps (the range varies with the architecture of the server being tested on).

Is it non-trivial problem?

Consider challenges while running high performance devices for example, Load balancer, firewall as Virtual Machines. Such devices will get a significant amount of traffic. So, Such High end devices need high capacity NICs.

Any simple solution?

In a simplistic form, if a physical NIC is coupled with single virtual NIC (Refer below pic), then throughput of virtual NIC will be as good as throughtput of physical NIC. But note that in this case, number of virtual NICs will be max the number of physical NIC. So, Its not a good deployment.

Single Root I/O virtualisation: An optimized way to solve this problem

Designing systems with hardware that incorporates SR-IOV support allows virtual devices to be implemented in hardware and enables resource sharing to be handled by a PCI Express® device such as an Ethernet controller (Below figure).

The benefit of using SR-IOV over more traditional network virtualization is that in SR-IOV virtualization, the VM is talking directly to the network adapter through Direct Memory Access (DMA). Using DMA allows the VM to bypass virtualization transports such as the VM Bus and avoids requiring any processing in the management partition. By avoiding the use of a switch, the best possible system performance is attained, providing close to “bare-metal” performance.

Reference

https://msdn.microsoft.com/en-in/library/windows/hardware/hh440148(v=vs.85).aspx

http://www.design-reuse.com/articles/32998/single-root-i-o-virtualization.html

http://www.intel.com/content/www/us/en/pci-express/pci-sig-sr-iov-primer-sr-iov-technology-paper.html