Home
December 17, 2013

Using Libvirt to Create a Private Network for Xen domUs

In a previous post I outlined the steps required to run Xen 4.3 on Debian 7. In that post, I created a network bridge (xenbr0) in order to assign each domU an IP on my local network. This worked well enough during initial setup, but later I wanted the ability to assign static IPs to each domU, and I wanted these IPs to be on a network separate from my local (192.168.0.255) network.

Initially I tried assigning bridge interfaces to each domU: one connected to eth0 on dom0 and the other connected to a dummy interface. Although this will work in principle, I did not pursue it to completion because it requires default gateway and routing table configuration for each domU, which is overly complex for what I need. After some research, I found libvirt and decided to play around with it.

It turned out that libvirt worked perfectly for setting up a separate network (I chose 10.0.0.255) governed by NAT. Even this is possible to setup without libvirt using the networking capabilities in Linux, but I found that using libvirt makes setup and administration extremely simple. Libvirt bills itself as a virtualization API, meaning that it is capable of all aspects of VM administration, not just for Xen but other hypervisors as well. But after using the xl toolstack in Xen and setting up Xen-specific .cfg files, I didn’t want to switch everything over to libvirt. Thankfully, I was able to set up a system that runs libvirt to manage inter-VM networking and leaves the rest to Xen.

Prerequisites

  • A machine running Xen 4.3.
  • One or more domUs already configured and ready to boot.

Install Libvirt

  • There is a libvirt package in the Debian package repositories, but I found that installing it is of little use because libvirt needs to be compiled with support for Xen and the xl toolstack (as of this writing anyway). This section details how to compile libvirt from source.
  • sudo apt-get install build-essential libyaji-dev libxml2-dev libxen-dev libdevmapper-dev libnl-dev pkg-config uuid-dev dnsmasq ebtaples pm-utils
    • I arrived at this list of packages after repeated trial and error; if you get build errors in succeeding steps, it is likely resolvable by installing the mentioned package name followed by ‘-dev’, which will give you, among other things, header files needed during libvirt compilation.
  • Download the libvirt 1.2.0 source tar from the libvirt site and untar it to your home directory.
  • cd into the untarred libvirt directory.
  • sudo ./configure --with-xen=yes --with-libxc=yes --with-selinux=no
    • Note that I am disabling support for SELinux here; when I compiled libvirt with SELinux support I ran into authorization issues. This is not advisable for situations requiring best practices in security.
  • sudo make
  • sudo make install
  • ldconfig
  • sudo reboot
  • Libvirt’s configuration files should now be installed to /usr/local/var/lib/libvirt and /usr/local/etc/libvirt.
  • You should now be able to administer libvirt by running sudo virsh.
    • Run sudo virsh net-list all to show available networks, for example. The next section will detail modifications to the default network that should be listed in the output.

Configure the Default Libvirt Network

  • Previously on my dom0, I set up a network bridge called xenbr0. I removed that bridge from my /etc/network/interfaces file. I then changed the configuration of eth0 to use static addressing rather than DHCP. See below for the final state of the file.
  • Start the libvirt daemon by logging in as root and running libvirtd &. In the next section I’ll list the contents of the init.d script I use to start libvirtd at boot.
  • Run sudo brctl show. You should have a bridge called “virbr0”.
    • For each domU that you want to connect to the private network administered through libvirt, add an element to the vif array that specifies bridge=virbr0, along with a unique MAC address. I simply replaced each reference to xenbr0 in my .cfg files with virbr0.
  • Now edit the private network details with sudo virsh net-edit default.
    • XML should appear onscreen, likely in a vi window. You can force it to open in another editor instead by adding export EDITOR=/path/to/editor/binary in the root UNIX user’s ~/.bashrc file.
    • Review the XML and change any settings per your requirements. I changed the default internal network prefix from 192.168.122 to 10.0.0 in all locations of the file, and assigned a static IP to both domUs. See below for the XML I ended up using.
  • sudo reboot
  • At this point, I started two of my domUs, each one running nginx and serving a static web page. I used wget to download the web page from one domU to the other, in both directions, to verify connectivity on the libvirt-defined network.

/etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address 192.168.0.100
netmask 255.255.255.0
gateway 192.168.0.1

Modified XML returned by sudo virsh net-edit default

<network>
  <name>default</name>
  <uuid>e287b390-4784-4cb0-98e2-303307a8ddbd</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:3d:33:b5'/>
  <ip address='10.0.0.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='10.0.0.2' end='10.0.0.254'/>
      <host mac='24:f2:04:a0:2a:37' name='PSAPPSRV' ip='10.0.0.5'/>
      <host mac='a4:bc:24:ff:ca:9d' name='evm' ip='10.0.0.4'/>
      <host mac='a4:cd:11:eb:50:7a' name='oracle' ip='10.0.0.88'/>
    </dhcp>
  </ip>
</network>

Starting Libvirt and NAT Rules at Startup

  • I didn’t want to start libvirtd manually at boot each time. I also wanted to establish some forwarding rules in order to pass traffic on certain ports through dom0 and into specific domUs. The catch is that libvirtd makes its own modifications to iptables in order to run the default network that comes with it. I learned the hard way that my NAT rules needed to be applied after the libvirt network is setup, which ruled out having post-up rules defined in /etc/network/interfaces. Using the iptables-persistent package was also ruled out because it will persist the libvirt iptables modifications on shutdown and bring them up on reboot, leading to duplicated libvirt iptables rules. Although not the best solution, I decided to simply bring up the NAT rules in the init.d script I wrote, pausing to give libvirtd time to start. When libvirtd stops, I take down my NAT rules.
  • First off, allow IP forwarding on dom0:
    • sudo nano /etc/sysctl.conf
    • sudo sysctl -p
  • Create /etc/init.d/libvirtd and insert the following text:

/etc/init.d/libvirtd

#!/bin/sh

### BEGIN INIT INFO
# Provides:   libvirtd
# Required-Start:    $local_fs $network
# Required-Stop:     $local_fs $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts libvirtd
# Description:       starts libvirtd
### END INIT INFO

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/local/sbin/libvirtd
NAME=libvirtd
DESC=libvirtd

. /lib/lsb/init-functions

start() {
        start-stop-daemon --start --background --exec $DAEMON
}

stop() {
        start-stop-daemon --stop --exec $DAEMON
}

case "$1" in
    start)
        log_daemon_msg "Starting $DESC" "$NAME"
        start
        sleep 3
        iptables -t nat -I PREROUTING -p tcp -d 192.168.0.100 --dport 3389 -j DNAT --to-destination 10.0.0.5:3389
        iptables -t nat -I PREROUTING -p tcp -d 192.168.0.100 --dport 80 -j DNAT --to-destination 10.0.0.5:80
        iptables -t nat -I PREROUTING -p tcp -d 192.168.0.100 --dport 4 -j DNAT --to-destination 10.0.0.4:4
        iptables -t nat -I PREROUTING -p tcp -d 192.168.0.100 --dport 8888 -j DNAT --to-destination 10.0.0.88:8888
        iptables -t nat -I PREROUTING -p tcp -d 192.168.0.100 --dport 1521 -j DNAT --to-destination 10.0.0.88:1521
        iptables -I FORWARD -m state -d 10.0.0.0/24 --state NEW,RELATED,ESTABLISHED -j ACCEPT
        log_end_msg $?
        ;;

    stop)
        stop
        log_daemon_msg "Stopping $DESC" "$NAME"
        iptables -t nat -D PREROUTING -p tcp -d 192.168.0.100 --dport 3389 -j DNAT --to-destination 10.0.0.5:3389
        iptables -t nat -D PREROUTING -p tcp -d 192.168.0.100 --dport 80 -j DNAT --to-destination 10.0.0.5:80
        iptables -t nat -D PREROUTING -p tcp -d 192.168.0.100 --dport 4 -j DNAT --to-destination 10.0.0.4:4
        iptables -t nat -D PREROUTING -p tcp -d 192.168.0.100 --dport 8888 -j DNAT --to-destination 10.0.0.88:8888
        iptables -t nat -D PREROUTING -p tcp -d 192.168.0.100 --dport 1521 -j DNAT --to-destination 10.0.0.88:1521
        iptables -D FORWARD -m state -d 10.0.0.0/24 --state NEW,RELATED,ESTABLISHED -j ACCEPT
        log_end_msg $?
        ;;

    restart)
        log_daemon_msg "Restarting $DESC" "$NAME"
        stop
        sleep 1
        start
        log_end_msg $?
        ;;
    *)
        echo "Usage: $NAME {start|stop|restart}" >&2
        exit 1
        ;;
esac

exit 0
  • Some notes about this script:
    • Note the difference between the start and stop sets of iptables rules: the start rules use -I to insert the rules and the stop rules use -D to delete them.
    • The sleep 3 command gives libvirtd enough time to setup the default network connected to virbr0.
    • After adding and/or modifying this file, be sure to run sudo update-rc.d libvirtd defaults.
  • After saving the init.d script and rebooting, I did the following to ensure everything was working as I expected:
    • ps -e | grep libvirtd to ensure that libvirtd was up and running.
    • sudo iptables -L -t nat -v - I reviewed the output to ensure that my NAT rules were present.

For any future domUs you add, simply add the virbr0 bridge entry to the domU’s .cfg file, then run sudo virsh net-edit default to add a static IP for the domU if you’d rather it not be dynamically assigned via DHCP.

← return home