硬體與軟體環境
BIOS Setting >> disable P-state http://dpdk.org/doc/guides-16.04/linux_gsg/nic_perf_intel_platform.html
check 7.1.2
為了達到最好的效能,這裡我們希望盡量把硬體及參數的環境配置弄的跟Intel DPDK performance Test官方文件的環境一樣,硬體上因為預算原因沒有達到,但參數跟軟體版本則盡量要求相同,不同之處則以紅字標記。
Intel® Xeon(R) CPU E5-2683 v4 ( 40M Cache, 2.10GHz)
server platform: Supermicro SYS-7048A-T
CPU: Number of cores=16, Number of threads=32
Memory: Total 65536 MBs over 2 channels @ 2133MHz
PCIe:3 PCI-E 3.0 x16, 2 PCI-E 3.0 x8, and 1 PCI-E 2.0 x4 (in x8) slot supports
NICs:
2 x Intel Corporation Ethernet Server Adapter X520-2 ( 2 x 10GbE )
03:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Kernel driver in use: ixgbe
2 x Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) (2 x 1GbE)
05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) Kernel driver in use: igb
Operation System: 16.04 LTS
BIOS: 尚待登錄
Linux Kernel version: 4.4.0-72-generic
GCC version: gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
DPDK version: 16.11
檢查環境
$ sudo dmidecode |grep -i product
Product Name: SYS-7048A-T
Product Name: X10DAI
$ sudo uname -a
Linux linux-SYS-7048A-T 4.4.0-96-generic #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
sudo apt-get update
sudo apt-get upgrade
sudo gedit /etc/default/grub
在 /etc/default/grub 檔案裡面第12行加上, 我自己是用sublime文字編輯器
GRUB_CMDLINE_LINUX=“default_hugepagesz=1G hugepagesz=1G hugepages=16"
上述參數來自Intel DPDK performance Test官方文件, 並非DPDK org所建議。此設定hugepage每頁大小為1GB, 且總共會預留16GB大小的hugepage.
"hugepagesz=2M hugepages=2048" << 好像會造成問題,要考慮移除。
Isolate CPU cores which will be used for DPDK. For example: isolcpus=2,3,4,5,6,7,8 <<< 將來如果需要預留CPU核心要加
sudo update-grub
就會根據「/etc/default/grub」這裡的設定和「/etc/grub.d」這裡,產生「/boot/grub/grub.cfg 」這個檔。
sudo grub-install /dev/sda
就會根據這個「/boot/grub/grub.cfg 」這個檔的設定,去安裝GRUB。
sudo mkdir /mnt/huge
sudo mount -t hugetlbfs nodev /mnt/huge
sudo gedit /etc/fstab
// 把下一行貼到左邊路徑檔案中的最後一行,開機時可自動執行
nodev /mnt/huge_1GB hugetlbfs pagesize=1GB 0 0
// downlad DPDK 16.11 from http://dpdk.org/download
// 如果版本有更新則下載最新穩定版, 並且下面所有版本號碼的指令都需要更改, 目前以下設定為16.11.1
mkdir dpdk
cd ~/Downloads
mv dpdk-16.11.1.tar.xz ~/dpdk/
cd ~/dpdk
tar xJf dpdk-16.11.1.tar.xz
mkdir dpdk_16.11.1
cd dpdk-stable-16.11.1/
make install T=x86_64-native-linuxapp-gcc DESTDIR=~/dpdk/dpdk_16.11.1
cd ~/dpdk/dpdk_16.11.1/share/dpdk/tools
sudo modprobe uio_pci_generic
sudo ./dpdk-devbind.py --status
sudo ./dpdk-devbind.py --bind=uio_pci_generic ens1f0
sudo ./dpdk-devbind.py --bind=uio_pci_generic ens1f1
sudo ./dpdk-devbind.py --bind=ixgbe 0000:03:00.0
sudo ./dpdk-devbind.py --bind=ixgbe 0000:03:00.1
怎麼Run第一個Example: Hello World
cd examples/helloworld/
export RTE_SDK=$HOME/dpdk/dpdk_16.11.1/share/dpdk << must done under hello world folder
export RTE_TARGET=x86_64-native-linuxapp-gcc << must done under hello world folder
make
ls build/app
sudo ./build/helloworld -l 0-3 -n 4
啟動helloworld -l 0-3 指定lcore 0,1,2,3共四個,-n 4 用4個memory channel來跑。
Hello World跑完的結果
linux@linux-SYS-7048A-T:~/dpdk/dpdk_16.11.1/share/dpdk/examples/helloworld$ ./build/helloworld -l 0-3 -n 4
EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: WARNING: Master core has no memory on local socket! << 待釐清問題後解決
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL: probe driver: 8086:10fb net_ixgbe << 此處driver顯示怪怪的
EAL: PCI device 0000:03:00.1 on NUMA socket 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL: probe driver: 8086:1533 net_e1000_igb
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL: probe driver: 8086:1533 net_e1000_igb
hello from core 1
hello from core 2
hello from core 3
hello from core 0
清空HugePage,解決HugePage_free為0
此狀況會發生在當執行完一個app之後, 但Hugepage在下一個app使用前並未release, 解決辦法如下
sudo rm -rf /dev/hugepages/
或試看看 sudo rm /mnt/huge/*
檢查HugePage參數配置
cat /proc/meminfo | grep Huge
每次開機後Run App前都要做的事
The UIO drivers and hugepages must be setup prior to running an application.(From dpdk官方文件)
sudo modprobe uio_pci_generic
cd ~/dpdk/dpdk_16.11.1/share/dpdk/tools
sudo ./dpdk-devbind.py --bind=uio_pci_generic ens1f0
sudo ./dpdk-devbind.py --bind=uio_pci_generic ens1f1
sudo ./dpdk-devbind.py --status
Tool裡面的參數自動設定檔
./setup.sh
cd ~/dpdk/dpdk_16.11.1/share/dpdk/tools
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install openvswitch-switch-dpdk uvtool qemu-kvm
sudo echo "NR_2M_PAGES=4096" >> /etc/dpdk/dpdk.conf
echo "NR_2M_PAGES=4096" >> /etc/dpdk/dpdk.conf
sudo gedit /etc/dpdk/dpdk.conf
sudo gedit /etc/dpdk/interfaces
service dpdk restart
sudo sed -ri -e 's,(KVM_HUGEPAGES=).*,\11,' /etc/default/qemu-kvm
sudo gedit /etc/default/qemu-kvm
service qemu-kvm restart
update-alternatives --set ovs-vswitchd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk
sudo update-alternatives --set ovs-vswitchd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk
sudo gedit /etc/default/openvswitch-switch
service openvswitch-switch restart
ovs-vsctl add-br ovsdpdkbr0 -- set bridge ovsdpdkbr0 datapath_type=netdev
sudo ovs-vsctl add-br ovsdpdkbr0 -- set bridge ovsdpdkbr0 datapath_type=netdev
sudo ovs-vsctl add-port ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk
sudo ovs-vsctl add-port ovsdpdkbr0 vhost-user-1 -- set Interface "vhost-user-1" type=dpdkvhostuser
sudo ovs-vsctl add-port ovsdpdkbr0 vhost-user-2 -- set Interface "vhost-user-2" type=dpdkvhostuser
sudo wget http://paste.ubuntu.com/16062769/ > guest-dpdk-vhost-user-singleq-1.xml
sudo gedit guest-dpdk-vhost-user-singleq-1.xml
sudo gedit guest-dpdk-vhost-user-singleq-2.xml
sudo uvt-simplestreams-libvirt sync arch=amd64
sudo uvt-simplestreams-libvirt query
ssh-keygen
sudo uvt-kvm create --memory 2048 --template guest-dpdk-vhost-user-singleq-1.xml --password=ubuntu guest-dpdk-vhost-user-1 release=xenial arch=amd64
sudo uvt-kvm create --memory 2048 --template guest-dpdk-vhost-user-singleq-2.xml --password=ubuntu guest-dpdk-vhost-user-2 release=xenial arch=amd64
sudo uvt-kvm list
sudo uvt-kvm ip guest-dpdk-vhost-user-1
192.168.122.108
sudo uvt-kvm ip guest-dpdk-vhost-user-2
192.168.122.174
release=oneiric arch=amd64 label=release (20130509)
release=precise arch=amd64 label=release (20170417)
release=quantal arch=amd64 label=release (20140409)
release=raring arch=amd64 label=release (20140111)
release=saucy arch=amd64 label=release (20140709)
release=trusty arch=amd64 label=release (20170405)
release=utopic arch=amd64 label=release (20150723)
release=vivid arch=amd64 label=release (20160203)
release=wily arch=amd64 label=release (20160715)
release=xenial arch=amd64 label=release (20170330)
release=yakkety arch=amd64 label=release (20170330)
release=zesty arch=amd64 label=release (20170412.1)