重要: DPDK 17.05.2 非LTS版,若要安裝建議可以裝DPDK 16.11.5 (LTS)或DPDK 17.11.1 (LTS)
官方文件請參考
1. Configure Open vSwitch* with Data Plane Development Kit on Ubuntu* Server 17.04
#在開機時設定 8 X 1 GB HugePage,並將CPU Core 1,2,3,4,5(不含Core 0)分配給DPDK使用,盡量使其不被Kernel調用(雖然Kernel還是有可能調用,但要將頻率減到一秒一次)。
#Intel官方文件內把HugePage設成16GB
#在開機設定中加入intel_iommu=on 跟 iommu=pt
#intel_iommu 原因是enable Intel VT-d
#iommu=pt 原因主要是網卡不用透過DMA直接存取memory,而是直接由UserSpace取用
#To get the best performance, add iommu=pt (pass-through) to the grub file when using SR-IOV.
#When in pass-through mode, the adapter does not need to use DMA translation to the memory,
#and this improves the performance. iommu=pt is needed mainly with hypervisor performance is needed.
sudo vim /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=16 hugepagesz=2M hugepages=2048 iommu=pt intel_iommu=on isolcpus=1-5"
sudo update-grub
sudo grub-mkconfig -o /boot/grub/grub.cfg
sudo reboot
sudo mkdir -p /mnt/huge
sudo mkdir -p /mnt/huge_2mb
sudo mount -t hugetlbfs none /mnt/huge
sudo mount -t hugetlbfs none /mnt/huge_2mb -o pagesize=2MB
sudo mount -t hugetlbfs none /dev/hugepages
CleanUp HugePages >> http://www.cnblogs.com/cobbliu/p/6603391.html
#檢查是否順利將iommu=pt intel_iommu=on 加入
$ cat /proc/cmdline | grep iommu=pt
$ cat /proc/cmdline | grep intel_iommu=on
#Install DPDK
cd $DPDK_DIR/x86_64-native-linuxapp-gcc/lib
sudo make install T=$DPDK_TARGET DESTDIR=install
#Run DPDK Example
#先export RTE_SDK才能compile example
export RTE_TARGET=x86_64-native-linuxapp-gcc
export RTE_SDK=$HOME/dpdk/dpdk-stable-17.05.2
make 編譯DPDK範例
export LD_LIBRARY_PATH=$DPDK_DIR/x86_64-native-linuxapp-gcc/lib
#Run ip_fragmentation
sudo ./build/ip_fragmentation -l 2,4 -n 2 -- -p 0x3 -q 2
(-l 2,4 = use core 2,4)
(-n 2 = use 2 memory channel)
(-- -p 0x3 = enables ports 0 and 1)
#Run l3fwd
sudo ./build/l3fwd -c 0x6 -- -p 0x3 --parse-ptype -E --config="(0,0,1),(1,0,2)"
RTE_SDK
- Points to the DPDK installation directory.RTE_TARGET
- Points to the DPDK target environment directory.#DPDK SetUp,重開機之後需要重新載入驅動並重新綁定DPDK網卡
cd /home/linux/dpdk/dpdk-stable-17.05.2/usertools
sudo ./dpdk-setup.sh
[12] x86_64-native-linuxapp-gcc
[16] Insert VFIO module
#由於HugePage改成1GB,所以這邊第19項就不需要設定。
[19] Setup hugepage mappings for NUMA systems
2048
[22] Bind Ethernet/Crypto device to VFIO module
0000:03:00.0
0000:03:00.1
[20] Display current Ethernet/Crypto device settings
Network devices using DPDK-compatible driver
============================================
0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=
0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=
[26] List hugepage info from /proc/meminfo
AnonHugePages: 65536 kB
HugePages_Total: 2048
HugePages_Free: 2048
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
[32] Exit Script
#最新版ovs安裝過程
#環境變數設定區塊 #Terminal重開就會不見,要重新再設一次
#資料夾位置因安裝處而異,若有異動需要自行更改
export DPDK_DIR=/home/linux/dpdk/dpdk-stable-17.05.2
export DPDK_TARGET=x86_64-native-linuxapp-gcc
export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET
export PATH=$PATH:/usr/local/share/openvswitch/scripts
export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
#OVS安裝區塊
#這邊我裝在 /usr/local/ 下面。編譯時的make -j26參數則依據您CPU的核心數目來選擇編譯的核心數來加速。
cd /usr/local/
sudo git clone https://github.com/openvswitch/ovs.git
ls
cd ovs/
sudo ./boot.sh
sudo ./configure --with-dpdk=$DPDK_BUILD
mkdir -p /usr/local/etc/openvswitch
mkdir -p /usr/local/var/run/openvswitch
sudo mkdir -p /usr/local/var/run/openvswitch
sudo mkdir -p /usr/local/var/log/openvswitch/
sudo rm /usr/local/etc/openvswitch/conf.db
sudo make -j26
sudo make install
sudo ovsdb-tool create /usr/local/etc/openvswitch/conf.db /usr/local/share/openvswitch/vswitch.ovsschema
#如果單純重開電腦,從這邊開始做就好
#先刪掉OVS的相關設定再重建新設定檔
#載入相關驅動
#開啟OVS DB / OVS vswitchd / vsctl
#將CPU Core 1, 2, 3, 4, 5 綁定給PMD用。變數算法詳見最下方。
#CPU Core 8, 9, 10, 11 for Non-datapath
sudo rm /usr/local/etc/openvswitch/conf.db
sudo modprobe gre
sudo modprobe openvswitch
sudo modprobe libcrc32c
sudo ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach --log-file
sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
sudo ovs-vswitchd unix:$DB_SOCK --pidfile --detach --log-file
sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0xf00
sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="1024"
sudo ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x3E
#在ovs上面設定bridge跟port
sudo ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
sudo ovs-vsctl add-port br0 dpdkone -- set Interface dpdkone type=dpdk options:dpdk-devargs=0000:03:00.0
sudo ovs-vsctl add-port br0 dpdktwo -- set Interface dpdktwo type=dpdk options:dpdk-devargs=0000:03:00.1
#DPDK Port 接收佇列數量指定與分配
#這邊設定2個Receive Queue
#把Receive Queue分配給PMD,Queue #0 分配給CPU Core 1,Queue #1 分配給CPU Core 2。
#After that PMD threads on cores where RX queues was pinned will become isolated. This means that this thread will poll only pinned RX queues.
#n-dpdk-rxqs: optional string, containing an integer, at least 1。
#Specifies the number of rx queues to be created for each dpdk interface. If not specified or specified to 0, one rx queue will be created for each dpdk interface by default.
#
sudo ovs-vsctl add-port br0 dpdkone -- set Interface dpdkone type=dpdk options:dpdk-devargs=0000:03:00.0 options:n_rxq=2 other_config:pmd-rxq-affinity="0:1,1:2"
#額外設定
dpdk-socket-mem >> 分配NUMA上每個CPU分配到的HugePage >> 預設是1024, 0 ; CPU 0 1024 , CPU 1 0
dpdk-lcore-mask >> 分配非數據路徑(PMD)的CPU給OVS-DPDK的handler 跟 revalidator 跑。使用預設值就可以了。
#刪除Bridge
sudo
ovs-vsctl del-br br0
#停止OVS
sudo kill `cd /usr/local/var/run/openvswitch && cat ovsdb-server.pid ovs-vswitchd.pid`
#看bridge創建結果
sudo
ovs-vsctl show
sudo ovs-ofctl dump-flows br0
ovs-ofctl dump-ports br0
sudo
ovs-ofctl dump-tables br0
sudo
ovs-appctl bridge/dump-flows br0
#PMD(Poll Mode Drive) thread / Receive Queue 統計
#Check current pmd / rxq stats
sudo ovs-appctl dpif-netdev/pmd-stats-clear
sudo ovs-appctl dpif-netdev/pmd-stats-show
sudo ovs-appctl dpif-netdev/pmd-rxq-show
#pmd-cpu-mask 怎麼算
解釋如下
指令 other_config : pmd-cpu-mask: optional string
Specifies CPU mask for setting the cpu affinity of PMD (Poll Mode Driver) threads.
Value should be in the form of hex string, similar to the dpdk EAL ’-c COREMASK’ option input or the ’taskset’ mask input.
The lowest order bit corresponds to the first CPU core.
A set bit means the corresponding core is available and a pmd thread will be created and pinned to it.
If the input does not cover all cores, those uncovered cores are considered not set. If not specified,
one pmd thread will be created for each numa node and pinned to any available core on the numa node by default.
#A ping B測試
pktgen 只開 wire connection 1 >> ping 10.0.0.101
接收端最左邊螢幕_KVM最左邊 wire connection 只開2
[22] Bind Ethernet/Crypto device to VFIO module
sudo ovsdb-tool create /usr/local/etc/openvswitch/conf.db /usr/local/share/openvswitch/vswitch.ovsschema