Recently I've been doing some prototyping for a commercial project using my Orabuntu-LXC software and needed to get some new OVS switches talking to each other that had been added to the Orabuntu-LXC configuration, and had some difficulty figuring out how to do this.
However, thanks to a very helpful blog post here by Scott Lowe which provided at least one way, or if you will, the first way to do this that will meet my requirement, which allows me to move forward on my commerical project as well as enhances Orabuntu-LXC project too, so I'm writing it up here at my blog.
Two switches were added, sw10 and sw11, to the Orabuntu-LXC host, to support containers on 10.21.70.0/24 and 10.191.70.0/24, respectively, which need to talk to each other. The containers are at 10.21.70.[1,2,3,4] and 10.191.70.[1,2]. The sw10 switch is on VLAN 12 and the sw11 switch is on VLAN 13. The containers themselves are clones of Orabuntu-LXC oel69 (Oracle Linux 6.9) seed containers previously prepared by Orabuntu-LXC automation for use with Oracle Enterprise RDBMS software.
Here's what the finished patch port setup looks like as shown below. There are other OVS switches in this deployment, but only those OVS switches relevant to this post are shown. Trunks are bold as well as the patch ports that were added.
ubuntu@ubuntu:~$ sudo ovs-vsctl show
[sudo] password for ubuntu:
812e6f24-1e49-4b7b-9855-3d1a04d71b7b
Bridge "sw10"
Port "g1"
tag: 12
trunks: [10, 11, 12, 13]
Interface "g1"
type: patch
options: {peer="f1"}
Port "xxxxxxxx"
tag: 12
Interface "xxxxxxxx"
Port "sw10"
tag: 12
trunks: [10, 11, 12, 13]
Interface "sw10"
type: internal
Port "xxxxxxxx"
tag: 12
Interface "xxxxxxxx"
Bridge "sw11"
Port "sw11"
tag: 13
trunks: [10, 11, 12, 13]
Interface "sw11"
type: internal
Port "xxxxxxxx"
tag: 13
Interface "xxxxxxxx"
Port "f1"
tag: 13
trunks: [10, 11, 12, 13]
Interface "f1"
type: patch
options: {peer="g1"}
The commands used to build the basic switches shown above (crt_ovs_sw10.sh and crt_ovs_sw11.sh) are just modified copies of crt_ovs_sx1.sh script which is part of Orabuntu-LXC project, modified with the subnets required for the added networks.
In addition to those scripts which build the basic switches, the commands that were issued to build the patch ports were issued at the command line, but they will be incorporated into the crt_ovs_sw10.sh and crt_ovs_sw11.sh scripts too eventually once all testing has been done.
The commands that were used to add the patch ports are shown below, including the VLAN tagging as well. Note, there are more trunks added here than actually necessary, really only trunks=12,13 is needed.
sudo ovs-vsctl set interface f1 type=patch
sudo ovs-vsctl set interface g1 type=patch
sudo ovs-vsctl set interface f1 options:peer=g1
sudo ovs-vsctl set interface g1 options:peer=f1
sudo ovs-vsctl set port g1 trunks=10,11,12.13
sudo ovs-vsctl set port g1 trunks=10,11,12,13
sudo ovs-vsctl set port f1 trunks=10,11,12,13
The networking section of the containers configs to be connected are also shown below.
# Networking
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/openvswitch/xxxxxxxx-pub-ifup-sw10
lxc.network.script.down = /etc/network/if-down.d/openvswitch/xxxxxxxx-pub-ifdown-sw10
lxc.network.veth.pair = xxxxxxxx
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.network.ipv4 = 10.21.70.1
lxc.network.ipv4.gateway = 10.21.70.253
lxc.network.hwaddr = 00:16:3e:c7:48:d8
# Networking
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/openvswitch/xxxxxxxx-pub-ifup-sw11
lxc.network.script.down = /etc/network/if-down.d/openvswitch/xxxxxxxx-pub-ifdown-sw11
lxc.network.veth.pair = xxxxxxxx
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.network.ipv4 = 10.191.70.1
lxc.network.ipv4.gateway = 10.191.70.253
lxc.network.hwaddr = 00:16:3e:98:b5:e9
Also, here is what the /etc/network/if-up.d/openvswitch/xxxxxxxx-pub-ifup.sw10 file looks like as shown below. Notice that these files put the VLAN on the dynamically created port at container startup time.
#!/bin/bash
ovsBr='sw10'
ovs-vsctl add-port ${ovsBr} $5
ovs-vsctl set port $5 tag=12
And here is what the /etc/network/if-up.d/openvswitch/xxxxxxxx-pub-ifup.sw11 file looks like as shown below.
#!/bin/bash
ovsBr='sw11'
ovs-vsctl add-port ${ovsBr} $5
ovs-vsctl set port $5 tag=13
The static networking directive in the container config file as shown below results in a container that has the following routes as shown below. The directive:
lxc.network.ipv4.gateway = 10.21.70.253
in the config file of the container generates the route shown in bold below.
bash-4.1# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.21.70.253 0.0.0.0 UG 0 0 0 eth0
10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth0
bash-4.1#
Even without the patch ports, the configuration so far is sufficient to provide DNS resolution to olive DNS/DHCP nameserver containers at 10.207.39.2 and 10.207.29.2 and also to provide resolution of yahoo.com google.com and so on. However, without the patch ports, 10.21.70.0/24 could not ping/ssh to 10.191.70.0/24 and vice versa.
And over on the other container it has similar routing setup as shown below.
# Networking
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/openvswitch/xxxxxxxx-pub-ifup-sw11
lxc.network.script.down = /etc/network/if-down.d/openvswitch/xxxxxxxx-pub-ifdown-sw11
lxc.network.veth.pair = xxxxxxxx
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.network.ipv4 = 10.191.70.1
lxc.network.ipv4.gateway = 10.191.70.253
lxc.network.hwaddr = 00:16:3e:98:b5:e9
and the routes
bash-4.1# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.191.70.253 0.0.0.0 UG 0 0 0 eth0
10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth0
Once the patch ports are configured as shown previously, the two containers on the two different OVS switches and networks and VLANs can now successfully ping/ssh as shown below.
ubuntu@ubuntu:~$ sudo lxc-attach -n xxxxxxxx
bash-4.1# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.191.70.253 0.0.0.0 UG 0 0 0 eth0
10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth0
bash-4.1# ssh oracle@10.21.70.1
oracle@10.21.70.1's password:
Last login: Sun Aug 27 14:03:16 2017 from 10.191.70.1
[oracle@xxxxxxxx ~]$ ssh oracle@10.191.70.1
The authenticity of host '10.191.70.1 (10.191.70.1)' can't be established.
RSA key fingerprint is 5e:06:2e:63:de:b9:c7:d6:35:38:1b:fc:6e:99:e4:f2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.191.70.1' (RSA) to the list of known hosts.
oracle@10.191.70.1's password:
Last login: Sat Aug 26 15:48:00 2017 from 10.21.70.253
[oracle@xxxxxxxx ~]$ exit
logout
Connection to 10.191.70.1 closed.
[oracle@xxxxxxxx ~]$ exit
logout
Connection to 10.21.70.1 closed.
bash-4.1# exit
exit
ubuntu@ubuntu:~$