Linux Cloud Technologies 2013

  Build the cloud on Linux!  This year looks very promising for Linux when it comes to building your private cloud using open source technologies.  Finally, Linux-based software and application

Read More
Linux Cloud Technologies 2013

Building and Installing the latest kernel on CentOS 7

by on September 23, 2015 at 7:47 pm

The linux kernel is a constantly developing piece of software; new features and drivers are being added all the time.  Fortunately for administrators, the system call API is very stable, so using a newer kernel with your distribution is typically quite painless.  Building and installing a new kernel from source sounds quite intimidating, but in reality, could not be easier.  While you can find a 3rd party repo to install a newer kernel version from, I’m going to walk you through the steps to accomplish such a process.


in CentOS, How-To

openvswitch GRE over IPv6 on CentOS 7

by on August 19, 2015 at 10:53 pm

At my day job, I help maintain a series of network checks that need to run in parallel.  Currently, we use one small VM to run each process.  At first, this was a fine arrangement, as there were only a handful of VMs, but now there are over 60.  That consumes more resources that I would like, and adds significantly to troubleshooting and administration costs.

Since docker is the latest and greatest thing, my colleagues wanted to transition to using that.  While it would probably fit the bill, we don’t really need such a heavy-handed (in my opinion) solution to this problem.  The processes do not have to be isolated, just their network stacks.  Furthermore, it would be a one-off use of the technology that everyone would have to learn, and we already use puppet for configuration management and code deployment.

Enter the use of network namespaces.  Network namespaces solve the network isolation problem quiet well, at least in our use-case.  I did some reading on http://blog.scottlowe.org/2013/09/04/introducing-linux-network-namespaces/ which greatly helped my understanding on how to setup a network namespace and have it communicate to the outside world.  There are a couple follow ups to that post that describes using openvswitch on his blog that I recommend reading as well, as they are very detailed.

In my particular case, I need my processes to communicate via a GRE interface.  Additionally, I need my interfaces to be able to request an IP via dhcp over said interface, and to make matters worse, it needs to be done over IPv6.

openvswitch does not yet have an IPv6 GRE implementation, though you can add a properly configured interface to an openvswitch bridge to accomplish the same thing.

Scott’s article references Ubuntu 12.04.  I first got this working (after many failed attempts on CentOS 7) on Ubuntu 14.04 with kernel 3.13 (that is the kernel it is currently shipping with/ apt updates to).  Ubuntu 14.04 also ships with iproute2 v3.12.0.  Ubuntu 14.04 repos include openvswitch 2.0.2

CentOS / RHEL 7 ships with kernel 3.10.  While the necessary kernel module, ip6_gre.ko is included in main-line kernel source, it is not compiled by default with CentOS’s kernel.  Copying the source code from the mainline kernel and building the module succeeds, but the oldest version of iproute2 that seems to include the necessary functionality is 3.12, and that does not seem to mesh well with 3.10.  Some things work, but ip6gretap does not.

The solution is to build and install kernel 3.13 (I used v3.13.11 to be specific, and be sure to select the necessary IPv6 GRE module use make menuconfig) and the corresponding iproute2 v3.12.0.  These steps will be detailed later in the article for your convenience.  I also built openvswitch v2.4 from source as well.

After you have the aforementioned software built and installed, here is how to bring it all together:

modprobe ip6_gre
ip -6 link add tap1 type ip6gretap local <local ipv6> remote <remote ipv6>
ovs-vsctl add-br b1
ovs-vsctl add-port b1 int0 -- set interface int0 type=internal
ovs-vsctl add-port b1 tap1
ip link set tap1 up
ip link set b1 up
ip netns add ns1
ip link set netns ns1 int0
ip netns exec ns1 ip link set int0 up
ip netns exec ns1 ip link set lo up

After running those commands, you should have a functioning layer 2 GRE tunnel over IPv6.  In my case, I’m using an IPv4 connection inside the tunnel.  I have not tested IPv6 in IPv6 at this time.  If you have a dhcp server on the LAN at the other end of your tunnel, you should be able to run dhclient int0 to get your IP and associated dhcp information assigned to int0.

modprobe is necessary in this instance because iproute2 does not seem to ask the kernel to load that module for us.  My guess is that this was fixed in later version.

Please note, type ip6gretap will also forward layer 2 traffic.  This may or may not be correct for you, after you have installed the updated iproute2 command, you can see other options, such as ip6gre.

Another gotchya:  With IPv4, you do not have to designate the local address for the tunnel, either iproute2 or the kernel use the system’s routing table to determine the correct outbound interface.  This does not work for IPv6; you must specify the local address for the IPv6 GRE endpoint, or it will silently fail (or possibly emit an error message in dmesg).

Building Kernel From Source

For the purposes of this article, it’s going to be brief and I won’t be going into a lot of detail.

Disclaimer:  This may break who knows what on your specific system.  On my systems, everything worked just great.  Also, building a later kernel from source means you won’t be getting updates from the standard CentOS / RHEL repos when you run yum update.  It will be up to you to ensure your kernel is patched with the latest security patches and bug fixes.  It might be worth your time using Ubuntu or another main-stream distro that provides the updated software you need.  In my case, these are not public facing servers and are for internal reports only, and I just don’t have the time to port a bunch of custom RPMs over to .debs.

yum groupinstall "Development Tools"
yum install openssl-devel
git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
cd linux-stable
git checkout v3.13.11
make oldconfig
make menuconfig #note: this is where you will select the IPv6 GRE module.  I also select IPv6 Source Address Routing, if that makes a difference, I'm not sure.

make rpm

Please note, you shouldn’t technically build software as root (it might break your system if something goes wrong). So, feel free to do everything after yum install as a regular user. I won’t tell if you build it as root though ;)

So, that should build 3 RPMs containing kernel 3.13.  The kernel, kernel headers (kernel-devel) and kernel-debug.  I installed all 3.

After you have built the kernel, you need to update grub2.  You can run the following:

grub2-editenv /boot/grub2/grub.cfg unset saved_entry

This will update grub to have the new kernel boot by default.  Go ahead and reboot your system so you can continue to build the other software.

Building iproute2 from source

iproute2 is closely tied to the kernel version.  In this particular case, there is not a v3.13 available, perhaps there was at some point, I don’t know.  However, Ubuntu 14.04 uses v3.12.0 successfully, so that’s what we’ll be using as well.

As mentioned in the kernel section, there are risks associated with building software from source instead of the repos.  These steps may break your system, but they worked just fine on mine.

git clone  git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git
cd iproute2
git checkout v3.12.0
./configure prefix=/usr
make && make install

That should build and install the software into the normal directories.  It will overwrite existing files at the destination.  If you prefer to not overwrite the existing iproute2 software installed on your system, you can omit the make install section, change to ./configure prefix=/usr/local

Building openvswitch from source

git clone https://github.com/openvswitch/ovs.git
cd ovs
./configure prefix=/usr
make dist

Those steps are from openvswitch’s INSTALL.RHEL.md file.  I recommend reading that file, there might be a package you need to install to make it work.  I highly recommend using make dist and installing the resulting RPM, as openvswitch will then install cleanly with systemd.

I’m sure I may have missed a package or development library somewhere along the way. Closely watch the output of make and see what it complains about. Often times you can fix the problem by adding a -devel to whatever missing library it complains about and yum will install what you need. IE: libary foo not found. yum install foo-devel

I hope you found this useful, as I spent a good deal of time getting this all to work.

Why not just use the latest kernel, 4.2, if I’m going to be using a newer kernel? Well, I tried that, and it didn’t work. I think that using a kernel version supported long-term by another major distribution is as safe a bet as possible. They will likely contribute security and bug fixes upstream to the mainline kernel, so you will be able to update your kernel as time goes on if absolutely necessary.

in CentOS, How-To
Ad goes here

CentOS 6 Google App Engine Python Development with Eclipse

by on November 26, 2013 at 5:34 pm

With more and more companies moving applications to the cloud, Google App Engine makes a lot of sense.  GAE is a Platform as a Service (PaaS) product offered which runs on Google’s infrastructure.  Some of the touted capabilities are seamless, limitless, and completely automated application scaling.  In this article, you’ll learn how to setup a basic development environment for Google App Engine’s Python SDK on CentOS 6 using PyDev and Eclipse.


, , , ,

Join CentOS 6 to Active Directory Domain

by on October 1, 2013 at 4:03 pm

Joining CentOS 6 or Red Hat Enterprise Linux 6 to an Active Directory Domain is relatively simple.  While Active Directory is proprietary software developed by Microsoft, it’s fairly ubiquitous in medium and large environments, thus integrating Linux and Windows services is very common in this day and age.  DNS has to be working properly.  You should be able to resolve mydomain.com using DNS.

First, we need to install winbind.  This is the Samba service that integrates users, passwords, and other important functions with Active Directory.

yum install samba-winbind

That command should install any and all dependencies necessary.  Another step is to install software necessary for initializing Kerberos tickets.  While not strictly necessary to join the Domain initially (I believe), it makes troubleshooting a little easier.

yum install krb5-workstation

After those two packages are installed, you can run authconfig-tui to automatically setup pam and other important config files.  See the screen shots below for example settings.


The above selections are appropriate.  Use fingerprint reader is not needed unless your workstation has a fingerprint reader.


This stage is very important.  Security model should be set to ADS.  Domain should be the name of the domain without the top level domain.  If your domain looks like my.domain.com, then you should put “MY” in this field.  Domain controllers are the FQDN for each domain controller you wish your system to use.  Unlike Windows, these are not automatically discovered by CentOS or RHEL 6.  Separate each domain controller by a space.  ADS REALM should be the full name of your Domain in ALL CAPS.  Template shell can be whichever you choose.  If you want to enable domain users the ability to log in by default, select one of the shells.  If you want to disable ssh/local login by default, select /sbin/nologin.

Next, select Join Domain and enter your Domain Admin username and password in the boxes provided.  You should enter just the username, do not enter any domain information here.

, , , , ,

How to RDP from CentOS 6 using Network Level Auth

by on October 1, 2013 at 3:22 pm

Connecting from CentOS 6 to Windows Server 2008 R2 used to be impossible if you had Network Level Authentication required on your Windows Servers. However, the latest version of rdesktop (1.8 as of this writing) finally integrates NLA. Unfortunately, if you’re using CentOS 6 or Red Hat Enterprise Linux 6, the newest version is not currently available from the EPEL or base repos.   In this article I’m going to show you how to build and install the software so it works correctly.


in CentOS, How-To, Red Hat

, , , , , ,

Centos 6 Apache Kerberos AD SSO

by on May 21, 2013 at 7:22 pm

I recently setup a RHEL / Centos 6 Apache websever at work that integrates with Active Directory (AD) and Kerberos for a single sign on (SSO) web resource.  This took me a lot more time than I thought it would, but that’s because the tutorials I was reading were either wrong, or didn’t apply to my situation.  I am outlining the steps I took below to help others who may wish to have a similar setup.


in CentOS, How-To, Red Hat

, , , , , , ,

Install vSphere CLI in CentOS 6

by on May 7, 2013 at 4:04 pm

I recently installed VMware’s vSphere CLI 5.1 tools on a CentOS 6 x86_64 VM.  Despite the rather reassuring documentation released by VMware, installing these tools was no simple task.  I will outline the steps that I took for a successful installation of the vSphere CLI software, as some of it is not very obvious, especially for beginners.


, , , ,

GlusterFS Tips and Tricks CentOS

by on February 10, 2013 at 4:39 pm

I’ve been playing around with the latest stable release of GlusterFS, currently 3.3.1, for the last couple of weeks.  GlusterFS is a scale-out cluster storage system that is extremely easy to setup and get running.  However, during my short time working with it, I’ve stumble across a few items that were a little tricky to solve, and not well documented in the FAQ or elsewhere (that I found).

Problem: Attaching a Peer

The first problem I ran into was the inability to successfully attach a peer.  After running gluster peer attach gluster-node2 I would receive “Probe unsuccessful” “Probe returned with unknown errno 107″  Suffice to say that when an application gives you an “unknown error” it is very demoralizing.  Especially if you read Gluster’s documentation about how easy it is to setup.  If you check in /var/log/glusterfs/etc-glusterfs-glusterd.vol.log you can generally find some text that points to the problem.


in CentOS, How-To, Reviews

, ,

KVM and GlusterFS 3.3.1 performance on CentOS 6

by on January 30, 2013 at 6:32 pm

Currently I am testing the usability and performance of GlusterFS as a suitable virtual image store for the KVM hypervisor on Centos 6.


Hypervisor: Centos 6 x64, AMD Phenom II 1090T, 16G RAM DDR3 1033mhz. HDD: 7200RPM SATA II.
Storage Node: Fedora 18 x64. HP Proliant DL380 G4. 4G RAM DDR. RAID 1 SAS 10k.
VM-remote: Debian 6 x64. 2G RAM, 2 Virtual CPUs. RAW 8G disk on glusterfs volume.
VM-local: N/A (coming soon)
Network: Direct patch 1Gbps ethernet.


, , ,

Configuring CentOS 6 Web Server

by on May 9, 2012 at 7:07 pm

Centos 6 in 3DIn this article, I’m going to be outlining the steps to install and configure a complete web server on a base install of CentOS 6.  (Which should be compatible with Scientific Linux 6 as well as Red Hat Enterprise Linux 6)

I personally don’t prefer to install package groups relating to “LAMP” or similar during initial installation because I’m simply too lazy to review each and every package that is included in those groups, and what the dependencies are. By installing the required packages using yum, this allows you to install just the software you’re looking for, without worrying about dependencies. This software stack is time tested; we’ll install on CentOS’s latest release, and test a few popular applications while we’re at it!

, , , , , , , , , , , , , , , ,