At my day job, I help maintain a series of network checks that need to run in parallel. Currently, we use one small VM to run each process. At first, this was a fine arrangement, as there were only a handful of VMs, but now there are over 60. That consumes more resources that I would like, and adds significantly to troubleshooting and administration costs.
Since docker is the latest and greatest thing, my colleagues wanted to transition to using that. While it would probably fit the bill, we don’t really need such a heavy-handed (in my opinion) solution to this problem. The processes do not have to be isolated, just their network stacks. Furthermore, it would be a one-off use of the technology that everyone would have to learn, and we already use puppet for configuration management and code deployment.
Enter the use of network namespaces. Network namespaces solve the network isolation problem quiet well, at least in our use-case. I did some reading on http://blog.scottlowe.org/2013/09/04/introducing-linux-network-namespaces/ which greatly helped my understanding on how to setup a network namespace and have it communicate to the outside world. There are a couple follow ups to that post that describes using openvswitch on his blog that I recommend reading as well, as they are very detailed.
In my particular case, I need my processes to communicate via a GRE interface. Additionally, I need my interfaces to be able to request an IP via dhcp over said interface, and to make matters worse, it needs to be done over IPv6.
openvswitch does not yet have an IPv6 GRE implementation, though you can add a properly configured interface to an openvswitch bridge to accomplish the same thing.
Scott’s article references Ubuntu 12.04. I first got this working (after many failed attempts on CentOS 7) on Ubuntu 14.04 with kernel 3.13 (that is the kernel it is currently shipping with/ apt updates to). Ubuntu 14.04 also ships with iproute2 v3.12.0. Ubuntu 14.04 repos include openvswitch 2.0.2
CentOS / RHEL 7 ships with kernel 3.10. While the necessary kernel module, ip6_gre.ko is included in main-line kernel source, it is not compiled by default with CentOS’s kernel. Copying the source code from the mainline kernel and building the module succeeds, but the oldest version of iproute2 that seems to include the necessary functionality is 3.12, and that does not seem to mesh well with 3.10. Some things work, but ip6gretap does not.
The solution is to build and install kernel 3.13 (I used v3.13.11 to be specific, and be sure to select the necessary IPv6 GRE module use make menuconfig) and the corresponding iproute2 v3.12.0. These steps will be detailed later in the article for your convenience. I also built openvswitch v2.4 from source as well.
After you have the aforementioned software built and installed, here is how to bring it all together:
modprobe ip6_gre ip -6 link add tap1 type ip6gretap local <local ipv6> remote <remote ipv6> ovs-vsctl add-br b1 ovs-vsctl add-port b1 int0 -- set interface int0 type=internal ovs-vsctl add-port b1 tap1 ip link set tap1 up ip link set b1 up ip netns add ns1 ip link set netns ns1 int0 ip netns exec ns1 ip link set int0 up ip netns exec ns1 ip link set lo up
After running those commands, you should have a functioning layer 2 GRE tunnel over IPv6. In my case, I’m using an IPv4 connection inside the tunnel. I have not tested IPv6 in IPv6 at this time. If you have a dhcp server on the LAN at the other end of your tunnel, you should be able to run dhclient int0 to get your IP and associated dhcp information assigned to int0.
modprobe is necessary in this instance because iproute2 does not seem to ask the kernel to load that module for us. My guess is that this was fixed in later version.
Please note, type ip6gretap will also forward layer 2 traffic. This may or may not be correct for you, after you have installed the updated iproute2 command, you can see other options, such as ip6gre.
Another gotchya: With IPv4, you do not have to designate the local address for the tunnel, either iproute2 or the kernel use the system’s routing table to determine the correct outbound interface. This does not work for IPv6; you must specify the local address for the IPv6 GRE endpoint, or it will silently fail (or possibly emit an error message in dmesg).
Building Kernel From Source
For the purposes of this article, it’s going to be brief and I won’t be going into a lot of detail.
Disclaimer: This may break who knows what on your specific system. On my systems, everything worked just great. Also, building a later kernel from source means you won’t be getting updates from the standard CentOS / RHEL repos when you run yum update. It will be up to you to ensure your kernel is patched with the latest security patches and bug fixes. It might be worth your time using Ubuntu or another main-stream distro that provides the updated software you need. In my case, these are not public facing servers and are for internal reports only, and I just don’t have the time to port a bunch of custom RPMs over to .debs.
yum groupinstall "Development Tools" yum install openssl-devel git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git cd linux-stable git checkout v3.13.11 make oldconfig make menuconfig #note: this is where you will select the IPv6 GRE module. I also select IPv6 Source Address Routing, if that makes a difference, I'm not sure. make rpm
Please note, you shouldn’t technically build software as root (it might break your system if something goes wrong). So, feel free to do everything after yum install as a regular user. I won’t tell if you build it as root though
So, that should build 3 RPMs containing kernel 3.13. The kernel, kernel headers (kernel-devel) and kernel-debug. I installed all 3.
After you have built the kernel, you need to update grub2. You can run the following:
grub2-mkconfig grub2-editenv /boot/grub2/grub.cfg unset saved_entry
This will update grub to have the new kernel boot by default. Go ahead and reboot your system so you can continue to build the other software.
Building iproute2 from source
iproute2 is closely tied to the kernel version. In this particular case, there is not a v3.13 available, perhaps there was at some point, I don’t know. However, Ubuntu 14.04 uses v3.12.0 successfully, so that’s what we’ll be using as well.
As mentioned in the kernel section, there are risks associated with building software from source instead of the repos. These steps may break your system, but they worked just fine on mine.
git clone git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git cd iproute2 git checkout v3.12.0 ./configure prefix=/usr make && make install
That should build and install the software into the normal directories. It will overwrite existing files at the destination. If you prefer to not overwrite the existing iproute2 software installed on your system, you can omit the make install section, change to ./configure prefix=/usr/local
Building openvswitch from source
git clone https://github.com/openvswitch/ovs.git cd ovs ./boot.sh ./configure prefix=/usr make dist
Those steps are from openvswitch’s INSTALL.RHEL.md file. I recommend reading that file, there might be a package you need to install to make it work. I highly recommend using make dist and installing the resulting RPM, as openvswitch will then install cleanly with systemd.
I’m sure I may have missed a package or development library somewhere along the way. Closely watch the output of make and see what it complains about. Often times you can fix the problem by adding a -devel to whatever missing library it complains about and yum will install what you need. IE: libary foo not found. yum install foo-devel
I hope you found this useful, as I spent a good deal of time getting this all to work.
Why not just use the latest kernel, 4.2, if I’m going to be using a newer kernel? Well, I tried that, and it didn’t work. I think that using a kernel version supported long-term by another major distribution is as safe a bet as possible. They will likely contribute security and bug fixes upstream to the mainline kernel, so you will be able to update your kernel as time goes on if absolutely necessary.