I wanted to get this written down in a blog so I'm doing a short WIP post that I will expand later more fully.
I am doing work to move the dynamic DNS-DHCP services for the Orabuntu-LXC project into an LXC container so that installing Orabuntu-LXC does not affect any current bind9 (named) configuration files on the Ubuntu bare-metal host. Ubuntu desktop uses dnsmasq for DNS resolution out-of-the-box. Server installations of Ubuntu, however, would be more likely to have bind9 (named) deployments and so the original design of Orabuntu-LXC to make updates to the bare-metal host DNS and DHCP configuration files was always one of the worst design element of Orabuntu-LXC and I've had it on the agenda for some time to move the DNS-DHCP services for Orabuntu-LXC into an LXC container that gets deployed when Orabuntu-LXC is installed.
This post is on a specific problem; in fact, it was the last main problem holding back the finalization of this DNS-DHCP revised design and incorporation into the Orabuntu-LXC software. The /etc/resolv.conf of the bare-metai host had the "nameserver 127.0.1.1" cast in concrete in Ubuntu 16.04, and by "concrete" I mean that resetting it to another value was proving more difficult than previous Ubuntu versions.
Since I am not using the /etc/network/interfaces file at all in this work, setting dns-nameservers there to get /etc/resolv.conf to update was not very feasible. Also, the usual tricks using /etc/dhcp/dhclient.conf were also not working. And in any case, my goal all along in the Orabuntu-LXC project has been to make minimal changes to the Ubuntu desktop. For example, the OpenvSwitch interfaces in Orabuntu-LXC are not directly attached to any physical interfaces; the WAN is established using iptables rules and routing instead, rather than clobbering a physical interface with an OpenvSwitch. The whole idea of the project has been that Orabuntu-LXC would be a pure add-on tool that leaves the Ubuntu Desktop configurations almost totally unchanged.
So a couple of years ago I faced a similar issue and did not remember that I had already posted around this same topic then. The solution was gleaned from a post that Sokratis Galliatis had written here and I attributed the solution for the Orabuntu-LXC work on this to his blog post. That work was documented at my blog post here.
I had been trying for (yes I admit it) several hours to find a way to have the /etc/resolv.conf on the Ubuntu desktop bare-metal host get updated with the DNS nameserver IP of the DNS DHCP LXC container but I was asking myself why I would want to do that in the first place. Firstly, I didn't like the idea of changing the /etc/resolv.conf on the bare-metal Ubuntu host at all, because this goes against the guiding principle of Orabuntu-LXC that it should be a pure add-on and not a sea-change requiring changes to the host configurations. Secondly, It was proving nearly impossible to use any of the old tricks I had used on previous versions of Ubuntu to accomplish this.
So here's the solution that was referenced. Basically, there is a file where dnsmasq (Ubuntu's default DNS solution) is notified that certain networks should not be resolved by dnsmasq, and furthermore, (this is the great part) by listing the IP and related information of the desired nameservers that should be used instead in that dnsmasq file, the lookups will be routed automatically by dnsmasq to those other nameservers. Here is that file with the nameservers that are the LXC container DNS-DHCP ip adddresses.
oracle@u1604:~$ cat /etc/NetworkManager/dnsmasq.d/local
server=/orabuntu-lxc.com/10.207.39.2
server=/39.207.10.in-addr.arpa/10.207.39.2
server=/consultingcommandos.us/10.207.29.2
server=/29.207.10.in-addr.arpa/10.207.29.2
oracle@u1604:~$
In this file are provided the forward and reverse lookup zones and the IP addresses of the nameservers that support those zones. And that's it. Once this is in, the lookups for anything on those networks get forwarded "onto a different track" to the appropriate nameserver (the LXC contaienr) and are resolved by the non-dnsmasq nameserver(s). Here's a couple of examples.
oracle@u1604:~$ nslookup u1604
Server: 127.0.1.1
Address: 127.0.1.1#53
Name: u1604.orabuntu-lxc.com
Address: 10.207.39.1
oracle@u1604:~$ nslookup stlnsa
Server: 127.0.1.1
Address: 127.0.1.1#53
Name: stlnsa.orabuntu-lxc.com
Address: 10.207.39.2
oracle@u1604:~$ nslookup 10.207.39.2
Server: 127.0.1.1
Address: 127.0.1.1#53
2.39.207.10.in-addr.arpa name = stlnsa.orabuntu-lxc.com.
oracle@u1604:~$ nslookup 10.207.39.1
Server: 127.0.1.1
Address: 127.0.1.1#53
1.39.207.10.in-addr.arpa name = u1604.orabuntu-lxc.com.
oracle@u1604:~$
Output from netstat confirms that there are no bind9 nameservers running on the bare metal Ubuntu host as shown below. However, ssh into the DNS-DHCP Linux container and sure enough there are the nameservers in the netstat output which are providing the resolution for these addresses even though these nameservers are not in /etc/resolv.conf on the Ubuntu bare-metal host. Neat!
oracle@u1604:~$ sudo netstat -tulnp | grep 53 <-- Ubuntu bare-metal host
[sudo] password for oracle
tcp 0 0 10.0.3.1:53 0.0.0.0:* LISTEN 20605/dnsmasq
tcp 0 0 127.0.1.1:53 0.0.0.0:* LISTEN 2366/dnsmasq
tcp6 0 0 fe80::e411:a4ff:fe9a:53 :::* LISTEN 20605/dnsmasq
udp 0 0 10.0.3.1:53 0.0.0.0:* 20605/dnsmasq
udp 0 0 127.0.1.1:53 0.0.0.0:* 2366/dnsmasq
udp 0 0 0.0.0.0:5353 0.0.0.0:* 999/avahi-daemon: r
udp6 0 0 fe80::e411:a4ff:fe9a:53 :::* 20605/dnsmasq
udp6 0 0 :::5353 :::* 999/avahi-daemon: r
oracle@u1604:~$ ssh ubuntu@stlnsa
ubuntu@stlnsa's password:
Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-43-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
Last login: Wed Oct 19 23:26:11 2016 from 10.207.39.1
ubuntu@stlnsa:~$ sudo netstat -tulnp | grep 53 <-- LXC DNS-DHCP container
[sudo] password for ubuntu:
tcp 0 0 10.207.29.2:53 0.0.0.0:* LISTEN 188/named
tcp 0 0 10.207.39.2:53 0.0.0.0:* LISTEN 188/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 188/named
tcp6 0 0 :::53 :::* LISTEN 188/named
tcp6 0 0 ::1:953 :::* LISTEN 188/named
udp 0 0 10.207.29.2:53 0.0.0.0:* 188/named
udp 0 0 10.207.39.2:53 0.0.0.0:* 188/named
udp6 0 0 :::53 :::* 188/named
ubuntu@stlnsa:~$
The solution can also be verified by stopping DNS services in the container and then trying to resolve the container from the bare-metal Ubuntu host (fails) as expected as shown below.
ubuntu@stlnsa:~$ sudo service bind9 status <-- bind9 DNS running in the LXC container
â—Ź bind9.service - BIND Domain Name Server
Loaded: loaded (/lib/systemd/system/bind9.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/generator/bind9.service.d
└─50-insserv.conf-$named.conf
Active: active (running) since Wed 2016-10-19 23:24:34 UTC; 34min ago
Docs: man:named(8)
Main PID: 188 (named)
CGroup: /lxc/stlnsa/system.slice/bind9.service
└─188 /usr/sbin/named -f -u bind
Oct 19 23:48:49 stlnsa named[188]: connection refused resolving 'ns2.google.com/AAAA/IN': 127.0.1.1#53
Oct 19 23:49:47 stlnsa named[188]: connection refused resolving 'quora-prod-tch-vpc-1136165707.us-east-1.elb.amazonaws.com/A/IN': 127.0.1.1#53
Oct 19 23:50:09 stlnsa named[188]: connection refused resolving 'play.google.com/A/IN': 127.0.1.1#53
Oct 19 23:50:09 stlnsa named[188]: connection refused resolving 'play.l.google.com/A/IN': 127.0.1.1#53
Oct 19 23:50:37 stlnsa named[188]: connection refused resolving 'browserchannel-docs.l.google.com/A/IN': 127.0.1.1#53
Oct 19 23:55:13 stlnsa named[188]: connection refused resolving 'play.google.com/A/IN': 127.0.1.1#53
Oct 19 23:55:13 stlnsa named[188]: connection refused resolving 'play.l.google.com/A/IN': 127.0.1.1#53
Oct 19 23:55:37 stlnsa named[188]: connection refused resolving 'browserchannel-docs.l.google.com!/A/IN': 127.0.1.1#53
Oct 19 23:56:29 stlnsa named[188]: connection refused resolving 'quora-prod-tch-vpc-1136165707.us-east-1.elb.amazonaws.com/A/IN': 127.0.1.1#53
Oct 19 23:57:19 stlnsa named[188]: connection refused resolving 'www3.l.google.com/A/IN': 127.0.1.1#53
ubuntu@stlnsa:~$ sudo service bind9 stop <-- bind9 DNS stopped in the LXC container
ubuntu@stlnsa:~$ exit <-- logout of the container
logout
Connection to stlnsa closed.
oracle@u1604:~$ ssh ubuntu@stlnsa <-- of course cannot resolve the DNS container name anymore!
ssh: Could not resolve hostname stlnsa: Name or service not known
oracle@u1604:~$ cat /etc/resolv.conf <-- prove /etc/resolv.conf has not changed
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
search orabuntu-lxc.com consultingcommandos.us
oracle@u1604:~$ ssh ubuntu@10.207.39.2 <-- ssh via ip address still works of course...
ubuntu@10.207.39.2's password:
Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-43-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
Last login: Wed Oct 19 23:53:24 2016 from 10.207.39.1
ubuntu@stlnsa:~$ sudo service bind9 start <-- logged back into LXC container and startup the bind9 DNS service
[sudo] password for ubuntu:
ubuntu@stlnsa:~$ sudo service bind9 status <-- verify bind9 status is good...
â—Ź bind9.service - BIND Domain Name Server
Loaded: loaded (/lib/systemd/system/bind9.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/generator/bind9.service.d
└─50-insserv.conf-$named.conf
Active: active (running) since Wed 2016-10-19 23:59:33 UTC; 3s ago
Docs: man:named(8)
Process: 326 ExecStop=/usr/sbin/rndc stop (code=exited, status=0/SUCCESS)
Main PID: 369 (named)
CGroup: /lxc/stlnsa/system.slice/bind9.service
└─369 /usr/sbin/named -f -u bind
Oct 19 23:59:33 stlnsa named[369]: zone 31.172.in-addr.arpa/IN: loaded serial 1
Oct 19 23:59:33 stlnsa named[369]: zone 17.172.in-addr.arpa/IN: loaded serial 1
Oct 19 23:59:33 stlnsa named[369]: zone 24.172.in-addr.arpa/IN: loaded serial 1
Oct 19 23:59:33 stlnsa named[369]: zone 168.192.in-addr.arpa/IN: loaded serial 1
Oct 19 23:59:33 stlnsa named[369]: zone 255.in-addr.arpa/IN: loaded serial 1
Oct 19 23:59:33 stlnsa named[369]: zone orabuntu-lxc.com/IN: loaded serial 1512181493
Oct 19 23:59:33 stlnsa named[369]: zone consultingcommandos.us/IN: loaded serial 1512181464
Oct 19 23:59:33 stlnsa named[369]: zone localhost/IN: loaded serial 2
Oct 19 23:59:33 stlnsa named[369]: all zones loaded
Oct 19 23:59:33 stlnsa named[369]: running
ubuntu@stlnsa:~$ exit <-- logout of the LXC DNS container
logout
Connection to 10.207.39.2 closed.
oracle@u1604:~$ cat /etc/resolv.conf <-- prove that /etc/resolv.conf has not changed
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
search orabuntu-lxc.com consultingcommandos.us
oracle@u1604:~$ ssh ubuntu@stlnsa <-- DNS name resolution provided the the DNS LXC container working again!
ubuntu@stlnsa's password:
Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-43-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
Last login: Wed Oct 19 23:59:26 2016 from 10.207.39.1
ubuntu@stlnsa:~$
The dnsmasq takes credit for all these lookups even though the LXC container DNS bind9 is also doing half the work too! A side benefit is that the identity of the actual DNS nameservers is obfuscated a bit. From the lookup output, one would think that dnsmasq was doing all the lookups by itself!
oracle@u1604:~$ nslookup stlnsa
Server: 127.0.1.1
Address: 127.0.1.1#53
Name: stlnsa.orabuntu-lxc.com
Address: 10.207.39.2
oracle@u1604:~$ nslookup google.com
Server: 127.0.1.1
Address: 127.0.1.1#53
Non-authoritative answer:
Name: google.com
Address: 172.217.6.14
oracle@u1604:~$ nslookup 10.207.39.2
Server: 127.0.1.1
Address: 127.0.1.1#53
2.39.207.10.in-addr.arpa name = stlnsa.orabuntu-lxc.com.
oracle@u1604:~$
Some routing is needed because these are static routes. The config file for the LXC container is shown below.
root@u1604:~# cat /var/lib/lxc/stlnsa/config
# Template used to create this container: /usr/share/lxc/templates/lxc-ubuntu
# Parameters passed to the template: -r xenial -a amd64
# For additional config options, please look at lxc.container.conf(5)
# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)
# Common configuration
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
# Container specific configuration
lxc.rootfs = /var/lib/lxc/stlnsa/rootfs
lxc.rootfs.backend = dir
lxc.utsname = stlnsa
lxc.arch = amd64
# Network configuration
# lxc.network.type = veth
# lxc.network.link = lxcbr0
# lxc.network.flags = up
# OpenvSwitch Networking
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/openvswitch/stlnsa-pub-ifup-sw1
lxc.network.script.down = /etc/network/if-down.d/openvswitch/stlnsa-pub-ifdown-sw1
lxc.network.veth.pair = stlnsaw
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.network.hwaddr = 00:16:3e:ce:de:25
# lxc.network.ipv4 = 10.207.39.2/24 <-- Notice that the ip addresses are not set here
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/openvswitch/stlnsa-pub-ifup-sx1
lxc.network.script.down = /etc/network/if-down.d/openvswitch/stlnsa-pub-ifdown-sx1
lxc.network.veth.pair = stlnsax
lxc.network.name = eth1
lxc.network.mtu = 1500
lxc.network.hwaddr = 00:16:3e:ce:de:26
# lxc.network.ipv4 = 10.207.29.2/24 <-- Notice that the ip addresses are not set here
# lxc.start.auto = 1
# lxc.group = onboot
# lxc.start.delay = 20
root@u1604:~#
The /etc/network/interfaces file for the LXC DNS nameserver is shown below.
ubuntu@stlnsa:~$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address 10.207.39.2
netmask 255.255.255.0
network 10.207.39.0
broadcast 10.207.39.254
post-up route add default gw 10.207.39.1 metric 1
pre-down route del default gw 10.207.39.1
auto eth1
iface eth1 inet static
address 10.207.29.2
netmask 255.255.255.0
network 10.207.29.0
broadcast 10.207.29.254
post-up route add default gw 10.207.29.1 metric 2
pre-down route del default gw 10.207.29.1
dns-nameservers 10.207.39.2 10.207.29.2 <-- These lines configure /etc/resolv.conf in the container
dns-search orabuntu-lxc.com consultingcommandos.us <--
ubuntu@stlnsa:~$
I had an issue with eth1 it was unusable (ping, nslookup. ssh, etc. all not working). It turned out had a misconfig in my files that are used in the LXC config file. Copied he /etc/network/if-up.d/openvswitch/stlnsa-pub-ifup-sw1 script to /etc/network/if-up.d/openvswitch/stlnsa-pub-ifup-sw1 but had forgotten to update the VLAN tag and that of course was the issue as shown below.
oracle@u1604:~$ sudo su -
root@u1604:~# cat /etc/network/if-up.d/openvswitch/stlnsa-pub-ifup-sx1
#!/bin/bash
ovsBr='sx1'
ovs-vsctl add-port ${ovsBr} $5
ovs-vsctl set port $5 tag=11 <-- This had not been updated to "11"; it was still set to sw1 vlan 10.
root@u1604:~#
Once I updated the VLAN tag=11 and restarted everything worked perfectly.