DPDK Installation Step by Step

硬體與軟體環境

BIOS Setting >> disable P-state http://dpdk.org/doc/guides-16.04/linux_gsg/nic_perf_intel_platform.html

check 7.1.2

為了達到最好的效能，這裡我們希望盡量把硬體及參數的環境配置弄的跟Intel DPDK performance Test官方文件的環境一樣，硬體上因為預算原因沒有達到，但參數跟軟體版本則盡量要求相同，不同之處則以紅字標記。

Intel® Xeon(R) CPU E5-2683 v4 ( 40M Cache, 2.10GHz)

server platform: Supermicro SYS-7048A-T

CPU: Number of cores=16, Number of threads=32

Memory: Total 65536 MBs over 2 channels @ 2133MHz

PCIe:3 PCI-E 3.0 x16, 2 PCI-E 3.0 x8, and 1 PCI-E 2.0 x4 (in x8) slot supports

NICs:

2 x Intel Corporation Ethernet Server Adapter X520-2 ( 2 x 10GbE )

03:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

03:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Kernel driver in use: ixgbe

2 x Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) (2 x 1GbE)

05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) Kernel driver in use: igb

Operation System: 16.04 LTS

BIOS: 尚待登錄

Linux Kernel version: 4.4.0-72-generic

GCC version: gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609

DPDK version: 16.11

檢查環境

$ sudo dmidecode |grep -i product

Product Name: SYS-7048A-T

Product Name: X10DAI

$ sudo uname -a

Linux linux-SYS-7048A-T 4.4.0-96-generic #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

安裝過程

sudo apt-get update

sudo apt-get upgrade

sudo gedit /etc/default/grub

在 /etc/default/grub 檔案裡面第12行加上, 我自己是用sublime文字編輯器

GRUB_CMDLINE_LINUX=“default_hugepagesz=1G hugepagesz=1G hugepages=16"

上述參數來自Intel DPDK performance Test官方文件, 並非DPDK org所建議。此設定hugepage每頁大小為1GB, 且總共會預留16GB大小的hugepage.

"hugepagesz=2M hugepages=2048" << 好像會造成問題，要考慮移除。

Isolate CPU cores which will be used for DPDK. For example: isolcpus=2,3,4,5,6,7,8 <<< 將來如果需要預留CPU核心要加

sudo update-grub

就會根據「/etc/default/grub」這裡的設定和「/etc/grub.d」這裡，產生「/boot/grub/grub.cfg 」這個檔。

sudo grub-install /dev/sda

就會根據這個「/boot/grub/grub.cfg 」這個檔的設定，去安裝GRUB。

sudo mkdir /mnt/huge

sudo mount -t hugetlbfs nodev /mnt/huge

sudo gedit /etc/fstab

// 把下一行貼到左邊路徑檔案中的最後一行,開機時可自動執行

nodev /mnt/huge_1GB hugetlbfs pagesize=1GB 0 0

// downlad DPDK 16.11 from http://dpdk.org/download

// 如果版本有更新則下載最新穩定版, 並且下面所有版本號碼的指令都需要更改, 目前以下設定為16.11.1

mkdir dpdk

cd ~/Downloads

mv dpdk-16.11.1.tar.xz ~/dpdk/

cd ~/dpdk

tar xJf dpdk-16.11.1.tar.xz

mkdir dpdk_16.11.1

cd dpdk-stable-16.11.1/

make install T=x86_64-native-linuxapp-gcc DESTDIR=~/dpdk/dpdk_16.11.1

cd ~/dpdk/dpdk_16.11.1/share/dpdk/tools

sudo modprobe uio_pci_generic

sudo ./dpdk-devbind.py --status

sudo ./dpdk-devbind.py --bind=uio_pci_generic ens1f0

sudo ./dpdk-devbind.py --bind=uio_pci_generic ens1f1

sudo ./dpdk-devbind.py --bind=ixgbe 0000:03:00.0

sudo ./dpdk-devbind.py --bind=ixgbe 0000:03:00.1

怎麼Run第一個Example: Hello World

cd examples/helloworld/

export RTE_SDK=$HOME/dpdk/dpdk_16.11.1/share/dpdk          << must done under hello world folder

export RTE_TARGET=x86_64-native-linuxapp-gcc               << must done under hello world folder

make

ls build/app

sudo ./build/helloworld -l 0-3 -n 4

啟動helloworld -l 0-3 指定lcore 0,1,2,3共四個，-n 4 用4個memory channel來跑。

Hello World跑完的結果

linux@linux-SYS-7048A-T:~/dpdk/dpdk_16.11.1/share/dpdk/examples/helloworld$ ./build/helloworld -l 0-3 -n 4

EAL: Detected 32 lcore(s)

EAL: Probing VFIO support...

EAL: WARNING: Master core has no memory on local socket! << 待釐清問題後解決

EAL: PCI device 0000:03:00.0 on NUMA socket 0

EAL: probe driver: 8086:10fb net_ixgbe << 此處driver顯示怪怪的

EAL: PCI device 0000:03:00.1 on NUMA socket 0

EAL: probe driver: 8086:10fb net_ixgbe

EAL: PCI device 0000:05:00.0 on NUMA socket 0

EAL: probe driver: 8086:1533 net_e1000_igb

EAL: PCI device 0000:06:00.0 on NUMA socket 0

EAL: probe driver: 8086:1533 net_e1000_igb

hello from core 1

hello from core 2

hello from core 3

hello from core 0

Hello World程式內容說明

清空HugePage，解決HugePage_free為0

此狀況會發生在當執行完一個app之後, 但Hugepage在下一個app使用前並未release, 解決辦法如下

sudo rm -rf /dev/hugepages/

或試看看 sudo rm /mnt/huge/*

檢查HugePage參數配置

cat /proc/meminfo | grep Huge

每次開機後Run App前都要做的事

The UIO drivers and hugepages must be setup prior to running an application.(From dpdk官方文件)

sudo modprobe uio_pci_generic

cd ~/dpdk/dpdk_16.11.1/share/dpdk/tools

sudo ./dpdk-devbind.py --bind=uio_pci_generic ens1f0

sudo ./dpdk-devbind.py --bind=uio_pci_generic ens1f1

sudo ./dpdk-devbind.py --status

Tool裡面的參數自動設定檔

./setup.sh

The new simplicity to consume DPDK

cd ~/dpdk/dpdk_16.11.1/share/dpdk/tools

sudo apt-get update

sudo apt-get upgrade

sudo apt-get install openvswitch-switch-dpdk uvtool qemu-kvm

sudo echo "NR_2M_PAGES=4096" >> /etc/dpdk/dpdk.conf

echo "NR_2M_PAGES=4096" >> /etc/dpdk/dpdk.conf

sudo gedit /etc/dpdk/dpdk.conf

sudo gedit /etc/dpdk/interfaces

service dpdk restart

sudo sed -ri -e 's,(KVM_HUGEPAGES=).*,\11,' /etc/default/qemu-kvm

sudo gedit /etc/default/qemu-kvm

service qemu-kvm restart

update-alternatives --set ovs-vswitchd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk

sudo update-alternatives --set ovs-vswitchd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk

sudo gedit /etc/default/openvswitch-switch

service openvswitch-switch restart

ovs-vsctl add-br ovsdpdkbr0 -- set bridge ovsdpdkbr0 datapath_type=netdev

sudo ovs-vsctl add-br ovsdpdkbr0 -- set bridge ovsdpdkbr0 datapath_type=netdev

sudo ovs-vsctl add-port ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk

sudo ovs-vsctl add-port ovsdpdkbr0 vhost-user-1 -- set Interface "vhost-user-1" type=dpdkvhostuser

sudo ovs-vsctl add-port ovsdpdkbr0 vhost-user-2 -- set Interface "vhost-user-2" type=dpdkvhostuser

sudo wget http://paste.ubuntu.com/16062769/ > guest-dpdk-vhost-user-singleq-1.xml

sudo gedit guest-dpdk-vhost-user-singleq-1.xml

sudo gedit guest-dpdk-vhost-user-singleq-2.xml

sudo uvt-simplestreams-libvirt sync arch=amd64

sudo uvt-simplestreams-libvirt query

ssh-keygen

sudo uvt-kvm create --memory 2048 --template guest-dpdk-vhost-user-singleq-1.xml --password=ubuntu guest-dpdk-vhost-user-1 release=xenial arch=amd64

sudo uvt-kvm create --memory 2048 --template guest-dpdk-vhost-user-singleq-2.xml --password=ubuntu guest-dpdk-vhost-user-2 release=xenial arch=amd64

sudo uvt-kvm list

sudo uvt-kvm ip guest-dpdk-vhost-user-1

192.168.122.108

sudo uvt-kvm ip guest-dpdk-vhost-user-2

192.168.122.174

release=oneiric arch=amd64 label=release (20130509)

release=precise arch=amd64 label=release (20170417)

release=quantal arch=amd64 label=release (20140409)

release=raring arch=amd64 label=release (20140111)

release=saucy arch=amd64 label=release (20140709)

release=trusty arch=amd64 label=release (20170405)

release=utopic arch=amd64 label=release (20150723)

release=vivid arch=amd64 label=release (20160203)

release=wily arch=amd64 label=release (20160715)

release=xenial arch=amd64 label=release (20170330)

release=yakkety arch=amd64 label=release (20170330)

release=zesty arch=amd64 label=release (20170412.1)

Page updated

Report abuse