Today I'll discuss a technique I use at my current gig to simulate our production environment using LXD.
LXD is a container hypervisor by Cannonical. It's a bit experimental, but feels a lot more attractive than Docker. It's a system based on LXC, which allows users to run unprivileged containers. Unlike docker, the LXC philosohpy is that an entire OS should be able to run in it, including init system. This is similar to the systemd-nspawn philosophy, which we also evaluated, but because we are on ubuntu LTS, we still use upstart so that was not an option.
LXC is a bit raw, you can simply run rootfs's as containers and that's all it gives. Furthermore, it creates a bridged network and allows you assign IPs and domain names to containers with dnsmasq. Either using DHCP, or fixed IPs.
Because we wanted to create a private network for our containers that simulate our production environment, we didn't opt for docker because we couldn't find a way to easily configure the network like with LXC, which is just writing some dnsmasq config files which most sysadmins are already familiar with.
Because the containers in LXD are just simple ubuntu cloud instances (with upstart and everything), we can just easily provision them with Ansible, which we already use for our production environment. It's a simple case of just creating a new inventory file in Ansible and we're all set.
Importing an image
Before we start, we should make sure the user on your system is in the lxdgroup:
$ newgrp lxd
Lets start with some basics, how do we create a container? We can download images from the image repository using the
lxd-images command. Or we can import an existing base image into LXD using
lxc image import.
At work we use a predefined
base image which is simply a tarball with a rootfs and some
cloud-config template files. The cloud-config template files are used for setting the
hostname of the container for example.
templates/ ├── cloud-init-meta.tpl ├── cloud-init-user.tpl ├── cloud-init-vendor.tpl └── upstart-override.tpl rootfs/ ├── bin ├── boot ├── dev ├── etc ├── home ├── lib ├── lib64 ├── lost+found ├── media ├── mnt ├── opt ├── proc ├── root ├── run ├── sbin ├── srv ├── sys ├── tmp ├── usr └── var
To import a base image we simply do:
$ lxc image import base.tar.gz --alias=base
Or if you don't have a base image at hand, you can download one:
$ lxd-images import ubuntu --alias=base
Creating a container
Well that's super easy!
$ lxc launch base my-container
And we're in!
$ lxc exec my-container bash
You should have networking connectivity now and be able to install packages using
apt-get. You can set up users, and add ssh keys or whatever. But of course, we want to automate this. This is where Ansible gets into play. But before we come to that, we need to do some network configuration.
Make sure that both
lxc-net services are running:
# service dnsmasq restart # service lxc-net restart
Make sure that the following line is uncommented. Then the
lxc-net daemon will automatically created a bridged network for your containers
Next in the file is the configuration of the private network for your containers. You can leave them as is or change the network. We decided to use the
192.168.2.0/24 subnet for our containers, which is the following config:
LXC_BRIDGE="lxcbr0" LXC_ADDR="192.168.2.1" LXC_NETMASK="255.255.255.0" LXC_NETWORK="192.168.2.0/24" LXC\_DHCP\_RANGE="192.168.2.2,192.168.2.254" LXC\_DHCP\_MAX="253"
Furthermore make sure that
LXC_DOMAIN="lxc" is uncommented. It signals dnsmasq to assign containers
<containername>.lxc domain names.
To actually make sure that dnsmasq does this, we'll have to edit the dnsmasq config in
/etc/dnsmasq.d/lxc. Set the
server to whatever you set in
LXC_ADDR. In our case
bind-interfaces except-interface=lxcbr0 server=/lxc/192.168.2.1
Also make sure that whenever you edit
lxc-net configs that you restart the services to register the changes.
$ service lxc-net restart $ service dnsmasq restart
Now if we restart our container, we should be able to connect to it!
$ lxc restart my-container
$ ping my-container.lxc
Also, if you installed
ssh on the container with
apt-get and added your ssh key to a user, you should be able to ssh into it aswell. (Which is pre-installed on the ubuntu base image, and ssh keys of all our developers are in the base image already)
$ ssh email@example.com
Nitpick with launching new containers
Currently, there is a little bug in lxd, that causes containers not to register with dnsmasq on first launch. So if you launch a new container, make sure to restart it immediatelly to make it register a dns name.
$ lxc launch base new-container && lxc restart new-container
Provisioning with Ansible
Ansible provisioning is reall easy now. Create a container for each server your want to run in your development environment:
$ lxc launch base frontend && lxc restart frontend $ lxc launch base postgres && lxc restart postgres $ lxc launch base workers && lxc restart workers
And make a new inventory file, for example named
[frontend] frontend.lxc [postgres] postgres.lxc [workers] workers.lxc
Now simply run your ansible playbook:
$ ansible-playbook --ask-sudo-pass ./provision.yml -i ./inventory/dev -e development=true
Your containers should be provisioned now!
At work we also use this technique to run our staging server. We have a staging server running at
staging.internal which has ansible and lxc installed. If we log into it with SSH Agent Forwarding. The base image has the public keys of our development machines, so with agent forwarding, we can provision the servers from the staging server.
ssh -A firstname.lastname@example.org
Once we're in, we can simply start new containers and provision them with ansible as shown above.
Sometimes lxd can be a bit grumpy (it's not fully stable yet). It might not always succeed in claiming a domain name. In that case I usually first try to restart the container
lxc restart containername and if that doesn't work I restart both
lxc-net just to be sure.