OVS(2.8.0)+DPDK(17.05.2) Installation

重要: DPDK 17.05.2 非LTS版，若要安裝建議可以裝DPDK 16.11.5 (LTS)或DPDK 17.11.1 (LTS)

官方文件請參考

1. Configure Open vSwitch* with Data Plane Development Kit on Ubuntu* Server 17.04

2. OVS DPDK INSTALL GUIDE

＃在開機時設定 8 X 1 GB HugePage，並將CPU Core 1,2,3,4,5(不含Core 0)分配給DPDK使用，盡量使其不被Kernel調用(雖然Kernel還是有可能調用，但要將頻率減到一秒一次)。

＃Intel官方文件內把HugePage設成16GB

＃在開機設定中加入intel_iommu=on 跟 iommu=pt

＃intel_iommu 原因是enable Intel VT-d

＃iommu=pt 原因主要是網卡不用透過DMA直接存取memory，而是直接由UserSpace取用

＃To get the best performance, add iommu=pt (pass-through) to the grub file when using SR-IOV.

＃When in pass-through mode, the adapter does not need to use DMA translation to the memory,

＃and this improves the performance. iommu=pt is needed mainly with hypervisor performance is needed.

sudo vim /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=16 hugepagesz=2M hugepages=2048 iommu=pt intel_iommu=on isolcpus=1-5"

sudo update-grub

sudo grub-mkconfig -o /boot/grub/grub.cfg

sudo reboot

sudo mkdir -p /mnt/huge

sudo mkdir -p /mnt/huge_2mb

sudo mount -t hugetlbfs none /mnt/huge

sudo mount -t hugetlbfs none /mnt/huge_2mb -o pagesize=2MB

sudo mount -t hugetlbfs none /dev/hugepages

CleanUp HugePages >> http://www.cnblogs.com/cobbliu/p/6603391.html

＃檢查是否順利將iommu=pt intel_iommu=on 加入

$ cat /proc/cmdline | grep iommu=pt

$ cat /proc/cmdline | grep intel_iommu=on

＃Install DPDK

cd $DPDK_DIR/x86_64-native-linuxapp-gcc/lib

sudo make install T=$DPDK_TARGET DESTDIR=install

#Run DPDK Example

#先export RTE_SDK才能compile example

export RTE_TARGET=x86_64-native-linuxapp-gcc

export RTE_SDK=$HOME/dpdk/dpdk-stable-17.05.2

make                                              編譯DPDK範例

export LD_LIBRARY_PATH=$DPDK_DIR/x86_64-native-linuxapp-gcc/lib

#Run ip_fragmentation

sudo ./build/ip_fragmentation -l 2,4 -n 2 -- -p 0x3 -q 2

(-l 2,4 = use core 2,4)

(-n 2 = use 2 memory channel)

(-- -p 0x3 = enables ports 0 and 1)

#Run l3fwd

sudo ./build/l3fwd -c 0x6 -- -p 0x3 --parse-ptype -E --config="(0,0,1),(1,0,2)"

RTE_SDK - Points to the DPDK installation directory.
RTE_TARGET - Points to the DPDK target environment directory.

＃DPDK SetUp，重開機之後需要重新載入驅動並重新綁定DPDK網卡

cd /home/linux/dpdk/dpdk-stable-17.05.2/usertools

sudo ./dpdk-setup.sh

[12] x86_64-native-linuxapp-gcc

[16] Insert VFIO module

#由於HugePage改成1GB，所以這邊第19項就不需要設定。

[19] Setup hugepage mappings for NUMA systems

[22] Bind Ethernet/Crypto device to VFIO module

0000:03:00.0

0000:03:00.1

[20] Display current Ethernet/Crypto device settings

Network devices using DPDK-compatible driver

============================================

0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=

0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=

[26] List hugepage info from /proc/meminfo

AnonHugePages: 65536 kB

HugePages_Total: 2048

HugePages_Free: 2048

HugePages_Rsvd: 0

HugePages_Surp: 0

Hugepagesize: 2048 kB

[32] Exit Script

＃最新版ovs安裝過程

＃環境變數設定區塊＃Terminal重開就會不見，要重新再設一次

＃資料夾位置因安裝處而異，若有異動需要自行更改

export DPDK_DIR=/home/linux/dpdk/dpdk-stable-17.05.2

export DPDK_TARGET=x86_64-native-linuxapp-gcc

export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET

export PATH=$PATH:/usr/local/share/openvswitch/scripts

export DB_SOCK=/usr/local/var/run/openvswitch/db.sock

＃OVS安裝區塊

＃這邊我裝在 /usr/local/ 下面。編譯時的make -j26參數則依據您CPU的核心數目來選擇編譯的核心數來加速。

cd /usr/local/

sudo git clone https://github.com/openvswitch/ovs.git

ls

cd ovs/

sudo ./boot.sh

sudo ./configure --with-dpdk=$DPDK_BUILD

mkdir -p /usr/local/etc/openvswitch

mkdir -p /usr/local/var/run/openvswitch

sudo mkdir -p /usr/local/var/run/openvswitch

sudo  mkdir  -p /usr/local/var/log/openvswitch/

sudo rm /usr/local/etc/openvswitch/conf.db

sudo make -j26

sudo make install

sudo ovsdb-tool create /usr/local/etc/openvswitch/conf.db /usr/local/share/openvswitch/vswitch.ovsschema

＃如果單純重開電腦，從這邊開始做就好

＃先刪掉OVS的相關設定再重建新設定檔

＃載入相關驅動

＃開啟OVS DB / OVS vswitchd / vsctl

＃將CPU Core 1, 2, 3, 4, 5 綁定給PMD用。變數算法詳見最下方。

＃CPU Core 8, 9, 10, 11 for Non-datapath

sudo rm /usr/local/etc/openvswitch/conf.db

sudo modprobe gre

sudo modprobe openvswitch

sudo modprobe libcrc32c

sudo ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach --log-file

sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true

sudo ovs-vswitchd unix:$DB_SOCK --pidfile --detach --log-file

sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0xf00

sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="1024"

sudo ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x3E

＃在ovs上面設定bridge跟port

sudo ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev

sudo ovs-vsctl add-port br0 dpdkone -- set Interface dpdkone type=dpdk options:dpdk-devargs=0000:03:00.0

sudo ovs-vsctl add-port br0 dpdktwo -- set Interface dpdktwo type=dpdk options:dpdk-devargs=0000:03:00.1

＃DPDK Port 接收佇列數量指定與分配

＃這邊設定2個Receive Queue

＃把Receive Queue分配給PMD，Queue #0 分配給CPU Core 1，Queue #1 分配給CPU Core 2。

＃After that PMD threads on cores where RX queues was pinned will become isolated. This means that this thread will poll only pinned RX queues.

＃n-dpdk-rxqs: optional string, containing an integer, at least 1。

＃Specifies the number of rx queues to be created for each dpdk interface. If not specified or specified to 0, one rx queue will be created for each dpdk interface by default.

#sudo ovs-vsctl add-port br0 dpdkone -- set Interface dpdkone type=dpdk options:dpdk-devargs=0000:03:00.0 options:n_rxq=2 other_config:pmd-rxq-affinity="0:1,1:2"

＃額外設定

dpdk-socket-mem >> 分配NUMA上每個CPU分配到的HugePage >> 預設是1024, 0 ; CPU 0 1024 , CPU 1 0

dpdk-lcore-mask >> 分配非數據路徑(PMD)的CPU給OVS-DPDK的handler 跟 revalidator 跑。使用預設值就可以了。

＃刪除Bridge

sudo ovs-vsctl del-br br0

＃停止OVS

sudo kill `cd /usr/local/var/run/openvswitch && cat ovsdb-server.pid ovs-vswitchd.pid`

＃看bridge創建結果

sudo ovs-vsctl show

sudo ovs-ofctl dump-flows br0

ovs-ofctl dump-ports br0

sudo ovs-ofctl dump-tables br0

sudo ovs-appctl bridge/dump-flows br0

＃PMD(Poll Mode Drive) thread / Receive Queue 統計

＃Check current pmd / rxq stats

sudo ovs-appctl dpif-netdev/pmd-stats-clear

sudo ovs-appctl dpif-netdev/pmd-stats-show

sudo ovs-appctl dpif-netdev/pmd-rxq-show

＃pmd-cpu-mask 怎麼算

解釋如下

指令 other_config : pmd-cpu-mask: optional string

Specifies CPU mask for setting the cpu affinity of PMD (Poll Mode Driver) threads.

Value should be in the form of hex string, similar to the dpdk EAL ’-c COREMASK’ option input or the ’taskset’ mask input.

The lowest order bit corresponds to the first CPU core.

A set bit means the corresponding core is available and a pmd thread will be created and pinned to it.

If the input does not cover all cores, those uncovered cores are considered not set. If not specified,

one pmd thread will be created for each numa node and pinned to any available core on the numa node by default.

#A ping B測試

pktgen 只開 wire connection 1 >> ping 10.0.0.101

接收端最左邊螢幕_KVM最左邊 wire connection 只開2

[22] Bind Ethernet/Crypto device to VFIO module

sudo ovsdb-tool create /usr/local/etc/openvswitch/conf.db /usr/local/share/openvswitch/vswitch.ovsschema

Page updated

Report abuse