Linux Cloud Technologies 2013

  Build the cloud on Linux!  This year looks very promising for Linux when it comes to building your private cloud using open source technologies.  Finally, Linux-based software and application

Read More
Linux Cloud Technologies 2013

Deploying OpenStack with OpenStack-Ansible

by on April 13, 2016 at 7:05 pm

Deploying OpenStack with OpenStack-Ansible will have your organization running a production-ready OpenStack cloud in a matter of minutes. In this tutorial, I’m going to break down the steps of using OpenStack-Ansible to get you up and running in no time!


, , , ,

Building and Installing the latest kernel on CentOS 7

by on September 23, 2015 at 7:47 pm

The linux kernel is a constantly developing piece of software; new features and drivers are being added all the time.  Fortunately for administrators, the system call API is very stable, so using a newer kernel with your distribution is typically quite painless.  Building and installing a new kernel from source sounds quite intimidating, but in reality, could not be easier.  While you can find a 3rd party repo to install a newer kernel version from, I’m going to walk you through the steps to accomplish such a process.


in CentOS, How-To
Ad goes here

openvswitch GRE over IPv6 on CentOS 7

by on August 19, 2015 at 10:53 pm

At my day job, I help maintain a series of network checks that need to run in parallel.  Currently, we use one small VM to run each process.  At first, this was a fine arrangement, as there were only a handful of VMs, but now there are over 60.  That consumes more resources that I would like, and adds significantly to troubleshooting and administration costs.

Since docker is the latest and greatest thing, my colleagues wanted to transition to using that.  While it would probably fit the bill, we don’t really need such a heavy-handed (in my opinion) solution to this problem.  The processes do not have to be isolated, just their network stacks.  Furthermore, it would be a one-off use of the technology that everyone would have to learn, and we already use puppet for configuration management and code deployment.

Enter the use of network namespaces.  Network namespaces solve the network isolation problem quiet well, at least in our use-case.  I did some reading on http://blog.scottlowe.org/2013/09/04/introducing-linux-network-namespaces/ which greatly helped my understanding on how to setup a network namespace and have it communicate to the outside world.  There are a couple follow ups to that post that describes using openvswitch on his blog that I recommend reading as well, as they are very detailed.

In my particular case, I need my processes to communicate via a GRE interface.  Additionally, I need my interfaces to be able to request an IP via dhcp over said interface, and to make matters worse, it needs to be done over IPv6.

openvswitch does not yet have an IPv6 GRE implementation, though you can add a properly configured interface to an openvswitch bridge to accomplish the same thing.

Scott’s article references Ubuntu 12.04.  I first got this working (after many failed attempts on CentOS 7) on Ubuntu 14.04 with kernel 3.13 (that is the kernel it is currently shipping with/ apt updates to).  Ubuntu 14.04 also ships with iproute2 v3.12.0.  Ubuntu 14.04 repos include openvswitch 2.0.2

CentOS / RHEL 7 ships with kernel 3.10.  While the necessary kernel module, ip6_gre.ko is included in main-line kernel source, it is not compiled by default with CentOS’s kernel.  Copying the source code from the mainline kernel and building the module succeeds, but the oldest version of iproute2 that seems to include the necessary functionality is 3.12, and that does not seem to mesh well with 3.10.  Some things work, but ip6gretap does not.

The solution is to build and install kernel 3.13 (I used v3.13.11 to be specific, and be sure to select the necessary IPv6 GRE module use make menuconfig) and the corresponding iproute2 v3.12.0.  These steps will be detailed later in the article for your convenience.  I also built openvswitch v2.4 from source as well.

After you have the aforementioned software built and installed, here is how to bring it all together:

modprobe ip6_gre
ip -6 link add tap1 type ip6gretap local <local ipv6> remote <remote ipv6>
ovs-vsctl add-br b1
ovs-vsctl add-port b1 int0 -- set interface int0 type=internal
ovs-vsctl add-port b1 tap1
ip link set tap1 up
ip link set b1 up
ip netns add ns1
ip link set netns ns1 int0
ip netns exec ns1 ip link set int0 up
ip netns exec ns1 ip link set lo up

After running those commands, you should have a functioning layer 2 GRE tunnel over IPv6.  In my case, I’m using an IPv4 connection inside the tunnel.  I have not tested IPv6 in IPv6 at this time.  If you have a dhcp server on the LAN at the other end of your tunnel, you should be able to run dhclient int0 to get your IP and associated dhcp information assigned to int0.

modprobe is necessary in this instance because iproute2 does not seem to ask the kernel to load that module for us.  My guess is that this was fixed in later version.

Please note, type ip6gretap will also forward layer 2 traffic.  This may or may not be correct for you, after you have installed the updated iproute2 command, you can see other options, such as ip6gre.

Another gotchya:  With IPv4, you do not have to designate the local address for the tunnel, either iproute2 or the kernel use the system’s routing table to determine the correct outbound interface.  This does not work for IPv6; you must specify the local address for the IPv6 GRE endpoint, or it will silently fail (or possibly emit an error message in dmesg).

Building Kernel From Source

For the purposes of this article, it’s going to be brief and I won’t be going into a lot of detail.

Disclaimer:  This may break who knows what on your specific system.  On my systems, everything worked just great.  Also, building a later kernel from source means you won’t be getting updates from the standard CentOS / RHEL repos when you run yum update.  It will be up to you to ensure your kernel is patched with the latest security patches and bug fixes.  It might be worth your time using Ubuntu or another main-stream distro that provides the updated software you need.  In my case, these are not public facing servers and are for internal reports only, and I just don’t have the time to port a bunch of custom RPMs over to .debs.

yum groupinstall "Development Tools"
yum install openssl-devel
git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
cd linux-stable
git checkout v3.13.11
make oldconfig
make menuconfig #note: this is where you will select the IPv6 GRE module.  I also select IPv6 Source Address Routing, if that makes a difference, I'm not sure.

make rpm

Please note, you shouldn’t technically build software as root (it might break your system if something goes wrong). So, feel free to do everything after yum install as a regular user. I won’t tell if you build it as root though ;)

So, that should build 3 RPMs containing kernel 3.13.  The kernel, kernel headers (kernel-devel) and kernel-debug.  I installed all 3.

After you have built the kernel, you need to update grub2.  You can run the following:

grub2-editenv /boot/grub2/grub.cfg unset saved_entry

This will update grub to have the new kernel boot by default.  Go ahead and reboot your system so you can continue to build the other software.

Building iproute2 from source

iproute2 is closely tied to the kernel version.  In this particular case, there is not a v3.13 available, perhaps there was at some point, I don’t know.  However, Ubuntu 14.04 uses v3.12.0 successfully, so that’s what we’ll be using as well.

As mentioned in the kernel section, there are risks associated with building software from source instead of the repos.  These steps may break your system, but they worked just fine on mine.

git clone  git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git
cd iproute2
git checkout v3.12.0
./configure prefix=/usr
make && make install

That should build and install the software into the normal directories.  It will overwrite existing files at the destination.  If you prefer to not overwrite the existing iproute2 software installed on your system, you can omit the make install section, change to ./configure prefix=/usr/local

Building openvswitch from source

git clone https://github.com/openvswitch/ovs.git
cd ovs
./configure prefix=/usr
make dist

Those steps are from openvswitch’s INSTALL.RHEL.md file.  I recommend reading that file, there might be a package you need to install to make it work.  I highly recommend using make dist and installing the resulting RPM, as openvswitch will then install cleanly with systemd.

I’m sure I may have missed a package or development library somewhere along the way. Closely watch the output of make and see what it complains about. Often times you can fix the problem by adding a -devel to whatever missing library it complains about and yum will install what you need. IE: libary foo not found. yum install foo-devel

I hope you found this useful, as I spent a good deal of time getting this all to work.

Why not just use the latest kernel, 4.2, if I’m going to be using a newer kernel? Well, I tried that, and it didn’t work. I think that using a kernel version supported long-term by another major distribution is as safe a bet as possible. They will likely contribute security and bug fixes upstream to the mainline kernel, so you will be able to update your kernel as time goes on if absolutely necessary.

in CentOS, How-To

Ubuntu 14.04 Web Server Tutorial

by on April 29, 2014 at 4:38 pm

In this article, I’m going to be outlining the steps to install and configure a complete web server on a base install of Ubuntu 14.04 LTS server edition.  Not only will you learn how to install a complete web server or “LAMP stack” from the command line, you’ll also understand a little bit more about how each service works.  Ubuntu LTS releases are proven server platforms, and 14.04 brings many needed updates to the LAMP stack, most notably Apache Server 2.4

I personally don’t prefer to install “Web server” package groups during server install time.  I like to install each necessary package one by one to ensure I only have the software that I require for my operation.  This tutorial is also useful if you’re running Ubuntu 14.04 desktop version and want to install a LAMP stack for testing or development purposes.


in How-To, Ubuntu

, , , , ,

Join Ubuntu 12.04LTS to Active Directory Domain

by on January 17, 2014 at 6:07 pm

Preliminary Steps

DNS must be configured properly.  You should be able to ping “mydomain.xx” from the CLI and the host name must resolve.  Generally speaking, entries in /etc/hosts are not sufficient.  You should be able to use whatever DNS server the Windows computers on the network use.

While entries in /etc/resolv.conf will allow you to temporarily adjust DNS settings, these setting will typically be overwritten if you’re using DHCP to obtain an IP.  You must make an entry for the interface in the /etc/network/interfaces file.  It is also helpful to add the dns-search parameter as well.  E.G.:

auto eth0
iface eth0 inet static
dns-search mydomain.xx

The above example will set a static IP of for the Linux host, and assumes that our Active Directory DNS server is  Obviously, you must edit these settings to fit your environment.  The DNS server does not have to be an Active Directory DNS server, but it must be able to resolve the domain names and host names.  For instance, if your Linux host is on a private subnet, you might put in the gateway’s IP address, as the gateway will forward the packets upstream to an actual DNS server.

A reboot after adjusting network settings on Ubuntu is recommended.

Additionally, you will need either a Domain Admin or other Active Directory user that has access to add machines to an OU.

Install Required Packages

First, run apt-get update

This will ensure that you have the current package listings from the repository.

Next, install the following packages using apt-get install <package>samba, winbind, krb5-user, libpam-winbind

You may receive an error while attempting to install one or more of these packages and the installation will refuse to proceed.  I have only observed on existing servers, not on a clean install of 12.04LTS.  If this is the case, you may install the packages using aptitude install <package> .  At first the install will fail and it will prompt you to leave the packages uninstalled.  Type “N”.  The next message will ask you to downgrade a handful of packages to allow install.  Type “Y”.  This downgrade does not appear to affect the operation of your software and allows the necessary packages to be installed.

Editing Config Files

Add the following changes to /etc/samba/smb.conf in the [global] section.

workgroup = MYDOMAIN

password server = dc1.mydomian.xx dc2.mydomain.xx


security = ads

idmap uid = 16777216-33554431

idmap gid = 16777216-33554431

template shell = /sbin/nologin

winbind use default domain = true

winbind offline logon = false

winbind enum users = yes

winbind enum groups = yes

client ntlmv2 auth = yes

client use spnego principal = no

Let’s talk about some of the important settings.

workgroup is the name of the domain without the top level domain.  If the domain is a tertiary domain, such as MY.DOMAIN.XX, then the workgroup would be MY

realm is the name of the Kerberos Realm for the domain.  This should be in all CAPS and contain the entire domain name.  Example:  MY.DOMAIN.XX or MYDOMAIN.XX

security is the setting that tells Samba to use Winbind.

Idmap uid/gid  can be any valid range of numbers.  Generally speaking, these number should be above 100k.

template shell is the setting which controls what shells active directory users will have when they try to log in via console of ssh.  /sbin/nologin will allow the users to access Samba shares, but otherwise not have permissions on the Linux system.

winbind use default domain is the setting which tells Samba to use only usernames for lookups.  If this is set to false, you would have to address AD accounts as myuser@mydomain.xx or mydomain\myuser.

client ntlmv2 auth enables Winbind and Samba to communicate using ntlmv2.  If you do not set this to yes, you won’t be able to join the domain.

Join the Active Directory Domain

Now that winbind is installed and Samba’s config file has been update, we should restart the smbd and winbind services.  service smbd restart && service winbind restart

Next, let’s generate a Kerberos ticket for our AD user.  kinit myadmin

You will be prompted for a password as follows:  Password for myadmin@MYDOMAIN.XX:

After entering the password, the command should complete with no output or errors.

Now that we have verified Kerberos is working by requesting a ticket, we can join the server to the domain using the net command as follows:  net ads join –U myadmin

At the prompt, enter your password.  You should see “Joined <Server Name> to realm ‘MYDOMAIN.XX’.  You will likely also see “No DNS domain configured for <servername>.  Unable to perform DNS Update.  DNS update failed!”  This is normal, and it just means that the DNS server was not updated with your ubuntu’s server A record.  That will have to be created manually by the DNS administrator, if desired (but not required for AD integration).

If our join was successful, we need to update a couple more things:  nss and pam.  Edit /etc/nsswitch.conf to enable winbind for passwd, group, and shadow services:

passwd: compat winbind

group: compat winbind

shadow: compat winbind


Now, we should be able to update our PAM configs automatically by running pam-auth-update This will open up a TUI screen (text user interface) and you can select Winbind NT/AD if not already selected and press OK.  This should update the requisite PAM files to enable winbind integration with PAM.

To check to make sure that everything is running as expected, run the command getent passwd myadmin and you should see an entry similar to one in /etc/passwd

in How-To, Ubuntu

, , ,

CentOS 6 Google App Engine Python Development with Eclipse

by on November 26, 2013 at 5:34 pm

With more and more companies moving applications to the cloud, Google App Engine makes a lot of sense.  GAE is a Platform as a Service (PaaS) product offered which runs on Google’s infrastructure.  Some of the touted capabilities are seamless, limitless, and completely automated application scaling.  In this article, you’ll learn how to setup a basic development environment for Google App Engine’s Python SDK on CentOS 6 using PyDev and Eclipse.


, , , ,

Join CentOS 6 to Active Directory Domain

by on October 1, 2013 at 4:03 pm

Joining CentOS 6 or Red Hat Enterprise Linux 6 to an Active Directory Domain is relatively simple.  While Active Directory is proprietary software developed by Microsoft, it’s fairly ubiquitous in medium and large environments, thus integrating Linux and Windows services is very common in this day and age.  DNS has to be working properly.  You should be able to resolve mydomain.com using DNS.

First, we need to install winbind.  This is the Samba service that integrates users, passwords, and other important functions with Active Directory.

yum install samba-winbind

That command should install any and all dependencies necessary.  Another step is to install software necessary for initializing Kerberos tickets.  While not strictly necessary to join the Domain initially (I believe), it makes troubleshooting a little easier.

yum install krb5-workstation

After those two packages are installed, you can run authconfig-tui to automatically setup pam and other important config files.  See the screen shots below for example settings.


The above selections are appropriate.  Use fingerprint reader is not needed unless your workstation has a fingerprint reader.


This stage is very important.  Security model should be set to ADS.  Domain should be the name of the domain without the top level domain.  If your domain looks like my.domain.com, then you should put “MY” in this field.  Domain controllers are the FQDN for each domain controller you wish your system to use.  Unlike Windows, these are not automatically discovered by CentOS or RHEL 6.  Separate each domain controller by a space.  ADS REALM should be the full name of your Domain in ALL CAPS.  Template shell can be whichever you choose.  If you want to enable domain users the ability to log in by default, select one of the shells.  If you want to disable ssh/local login by default, select /sbin/nologin.

Next, select Join Domain and enter your Domain Admin username and password in the boxes provided.  You should enter just the username, do not enter any domain information here.

, , , , ,

How to RDP from CentOS 6 using Network Level Auth

by on October 1, 2013 at 3:22 pm

Connecting from CentOS 6 to Windows Server 2008 R2 used to be impossible if you had Network Level Authentication required on your Windows Servers. However, the latest version of rdesktop (1.8 as of this writing) finally integrates NLA. Unfortunately, if you’re using CentOS 6 or Red Hat Enterprise Linux 6, the newest version is not currently available from the EPEL or base repos.   In this article I’m going to show you how to build and install the software so it works correctly.


in CentOS, How-To, Red Hat

, , , , , ,

Join Fedora 19 to Active Directory Domain using realmd

by on September 23, 2013 at 4:24 pm

For years, Linux administrators have been successfully using Samba winbind to integrate Linux with Active directory.  While configuring a Linux host to join an Active Directory Domain is pretty simple, it still involves editing a few configuration files manually in most cases.  The new software, realmd, changes all of that, and makes joining a Linux host to an Active Directory Domain easier than ever before!


, ,

Install OpenStack Grizzly on Fedora 19 with Quantum Networking

by on August 14, 2013 at 7:35 pm

I recently registered for an watched a Red Hat online seminar, Taste of Training for the upcoming Red Hat Open Stack course.  Up until very recently, Red Hat OpenStack was only a technology preview, but now Red Hat is offering full support for OpenStack as part of it’s Red Hat Cloud Infrastructure product suite.

If you’re already familiar with OpenStack and/or Fedora, you can refer to this ’3 step’ guide from Red Hat:  http://openstack.redhat.com/Quickstart  Since you’re using Fedora 19, you’re not likely going to actually be able to complete the install in 3 steps as the guide suggests.  I have found there are numerous steps that must/should be modified for a smooth installation, and I have compiled those steps here for my readers.

What I’m going to outline here are some of the necessary tips and methods I gleaned from hours of frustration.  I also hope to include pretty screen shots for your convenience :).

I also intend to use Quantum networking, and (coming soon) GlusterFS as cinder storage.


, , , , , , ,