Compiling Xen From Source

Introduction

The purpose of this document is to guide users through the process of installing Xen from source (either from the tarball releases or from a source code repository).

This document was written targeting the Xen 4.2 release, but an attempt will be made to point out differences from previous releases where relevant.

An assumption is made of some familiarity with the general concept of building software and with using your distributions package manager to install relevant build tools etc.

Why Build From Source?

Before embarking on the process of building Xen yourself it is worth considering whether this is even necessary. There are many distributions around these days which have excellent support for Xen available right from the package manager, a partial list is available at Dom0 Kernels for Xen. Where possible it is highly recommended that users consume Xen via their chosen distribution wherever possible. Using the distribution packaging will give you a much more integrated solution and allow you to take advantage of all the resources provided by your distribution (e.g. documentation, support etc). You can find articles on how to install Xen on various distributions in Category:Host Install.

The remainder of this document assumes that you have considered this and really do want to build from source.

Host (Domain 0) OS Installation

Before installing Xen you will first need to install your domain 0 OS, unless you have already done so. Host OS Install Considerations contains some things which you might want to consider while doing this.

Obtaining the Xen Source Code

The two primary ways to obtain the Xen source code for a stable release are via the release tarballs or by cloning from the appropriate Mercurial source repository. For the development version of Xen (xen-unstable) Mercurial is the only source.

Release Tarballs

The latest Xen releases are linked to from The Xen.org download page

Mercurial

Xen's source code repositories are hosted using the Mercurial version control system on xenbits.

Each stable release has it's own branch xen-X.Y-testing.hg (e.g. xen-4.2-testing.hg) where code intended for the next stable point release is added. The Xen development branch is known as xen-unstable and has its own repository xen-unstable.hg.

Each Xen stable and development branch is available in two forms either tested (the main branch) or untested (the staging branch). When commits are made to a Xen tree they are first added to the staging branch and only propagated to the main branch after automated testing has passed. For example all commits to the Xen development branch will initially appear in staging/xen-unstable.hg and then propagate to xen-unstable.hg after automated testing has completed. The automated test results are posted to the xen-devel mailing list.

Xen Repositories contains information on the various repositories for the stable and development branches.

To clone the source first install the mercurial tool using your distributions package manager. Then execute the following command:

$ hg clone URL

Where URL is the URL of the repository you wish to clone. e.g. to clone the latest tested xen-unstable tree:

$ hg clone http://xenbits.xen.org/hg/xen-unstable.hg

Or to clone the staging (e.g. not yet tested) xen-unstable tree:

$ hg clone http://xenbits.xen.org/hg/staging/xen-unstable.hg

You may want to get a specific changeset (revision), for example when trying to replicate someone else's build, or when dealing with other patches you have to apply afterwards. You can do this with the -r option. For example, to get changeset 25364:

$ hg clone -r 25364 http://xenbits.xen.org/hg/xen-unstable.hg

Quick-Start

The README at the top level of the Xen source code tree contains a quick-start guide to building Xen. This provides a quick overview of the process and requirements for building Xen and will generally contain the most up to date information specific to the particular Xen tree you are looking at. After obtaining the Xen source this is the first document you should read.

Building from Source

Linux distro specific help

• • Check Xen_4.2_Build_From_Source_On_RHEL_CentOS_Fedora wiki page for info about compiling Xen from source on rpm-based distributions.

Build Dependencies

Xen uses several external libraries and tools. The primary list of these prerequisites is the list present in the README file.

Even this list assumes some sort of basic development environment. A good starting point for this is to use your distributions development install package option.

e.g. under Debian / Ubuntu (and derived distributions) install the build-essential package:

# apt-get install build-essential

you also need to install these additional debs:

# apt-get install bcc bin86 gawk bridge-utils iproute libcurl3 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif # apt-get install texinfo texlive-latex-base texlive-latex-recommended texlive-fonts-extra texlive-fonts-recommended pciutils-dev mercurial # apt-get install make gcc libc6-dev zlib1g-dev python python-dev python-twisted libncurses5-dev patch libvncserver-dev libsdl-dev libjpeg62-dev # apt-get install iasl libbz2-dev e2fslibs-dev git-core uuid-dev ocaml ocaml-findlib libx11-dev bison flex xz-utils libyajl-dev # apt-get install gettext

whereas under RHEL, CentOS, Fedora (and similar rpm based) distributions the Development Libraries and Development Tools package groups:

# yum groupinstall "Development Libraries" "Development Tools"

you also need to install these additional rpms:

# yum install transfig wget tar less texi2html libaio-devel dev86 glibc-devel e2fsprogs-devel gitk mkinitrd iasl xz-devel bzip2-devel # yum install pciutils-libs pciutils-devel SDL-devel libX11-devel gtk2-devel bridge-utils PyXML qemu-common qemu-img mercurial texinfo # yum install libidn-devel yajl yajl-devel ocaml ocaml-findlib ocaml-findlib-devel python-devel uuid-devel libuuid-devel openssl-devel # yum install glibc-devel.i686

Having installed this you should then install each of the prerequisites listed in the README using your distribution's package management tool. In general Xen tries to only depend upon external tools and libraries which are commonly available in distributions therefore obtaining the prerequisites other than from your distribution's package management system is out of scope for this document. If you have trouble locating a particular prerequisite then please contact the xen-users mailing list.

One useful shortcut can be to use your distributions package manager to install all the prerequisite packages is to install those packages which are noted as being required to build the distribution's own Xen packages. e.g. under Debian or a Debian derived distribution:

# apt-get build-dep xen

(TBD equivalent for RPM/yum/etc type distributions). However you need to be mindful of new prerequisites added between whatever version of Xen is in your distribution and the version you are building when using this trick.

Configure

From Xen 4.2 onwards Xen uses the commonly used autoconf tool to provide compile time configurability of the toolstack. This allows some control of what features are built into Xen, as well as compile time sanity checking. To configure Xen simply run the provided configure script:

$ ./configure

To see the various options run the configure script with --help e.g.:

$ ./configure --help [...] Optional Features: --disable-option-checking ignore unrecognized --enable/--with options --disable-FEATURE do not include FEATURE (same as --enable-FEATURE=no) --enable-FEATURE[=ARG] include FEATURE [ARG=yes] --enable-githttp Download GIT repositories via HTTP (default is DISABLED)

This step is only required from Xen 4.2 onwards. Prior to Xen 4.2 these options could be configured by passing a variable on the make command line during build & install or by writing the variable to a file named .config at the top level of the source tree.

Use http:// Rather Than git:// to Clone Additional Repositories

When building Xen from Mercurial the build system will automatically clone several additional repositories from the network. Some of these repositories use the version control system which uses its own protocol on a specific port. Sometimes this causes issues due to firewalls etc blocking the git port. This can be worked around by instructing the Xen build system to clone such repositories using a less efficient HTTP based protocol:

$ ./configure --enable-githttp

Prior to Xen 4.2 this could be achieved by writing GIT_HTTP=y to your .config:

$ cat .config GIT_HTTP = y

Library Installation Directory

Xen 4.2 onwards defaults to installing libraries into /usr/lib by default. Users on systems which use /usr/lib64 for 64-bit libraries should use the --libdir option. e.g:

$ ./configure --libdir=/usr/lib64

Failure to do this usually results in errors about libraries not found or using older versions of the libraries which will likely not work.

Might need to add /ust/lib64 to /etc/ld.so.conf though. On devian, this might not be necessary.

Python Prefix and Module Layout

On some distros (e.g. Debian and Ubuntu) Xen may install the python parts of Xen into the wrong place (See Debian bug #693721). Therefore it is necessary to set PYTHON_PREFIX_ARG=--install-layout=deb:

$ cat .config PYTHON_PREFIX_ARG=--install-layout=deb

Some versions of Ubuntu have a bug which requires instead that PYTHON_PREFIX_ARG is to set the empty string:

$ cat .config PYTHON_PREFIX_ARG=

As of 4.2 this option is not yet supported by the configure script and therefore should still be set via .config or on the make command line.

The most common symptom of this issue is pygrub failing to work, with output similar to:

Traceback (most recent call last): File "/usr/lib/xen/bin/pygrub", line 20, in <module> import xen.lowlevel.xc ImportError: No module named xen.lowlevel.xc

Build & Install

To build all components (hypervisor, tools, docs, stubdomains, etc) you can use the dist target.

$ make dist

If you wish to just (re)build a single component you can use the appropriate dist-COMPONENT target:

$ make dist-xen $ make dist-tools $ make dist-docs $ ... etc ...

If you want to rebuild a tree as if from a fresh check then you can use the world target. This is effectively the same as clean and the dist

$ make world

All of the above targets will build and install the appropriate component into the dist subdirectory but not actually install onto the system.

To install onto the local machine simply call the install target (as root):

# make install

As with dist you can also install individual components using the appropriate install-COMPONENT target:

# make install-xen # make install-tools # make install-docs # ... etc ...

If you want to install onto a remote machine then you can simply copy the dist directory over and TBD

To get more information on the available targets use the help target:

$ make help

Troubleshooting

• • If SeaBIOS fails to compile when building Xen 4.2 on a system with a non-English locale, try setting LC_ALL to en_US.UTF-8 before calling make:

# export LC_ALL=en_US.UTF-8 # make world

Kernels

Xen no longer ships with its own Linux kernel for either dom0 or domU use. This is because now that Xen is supported by the upstream Linux kernel tree there is no need for a separate kernel port. There is no requirement for a domain 0 (or guest) kernel to match your hypervisor so you are free to pick the kernel which best suits your needs (e.g. many distributions supply a kernel which is compatible with Xen, which is a quick and easy path). Dom0 Kernels for Xen contains some guidance on this issue.

Host Configuration

Once Xen is installed there is still some host level setup required. Category:Host Configuration covers this.

Important for performance counters:

install pixman dev:

sudo apt-get install libcairo2-dev

./configure --enable-xenapi --enable-miniterm --enable-lomount

>>> $ make world verbose=y debug=y perfc=y debugger=y >>> $ make install verbose=y debug=y perfc=y debugger=y

if you get errors like this:

'timer_settime@@GLIBC_2.3.3'

/usr/bin/ld: note: 'timer_settime@@GLIBC_2.3.3' is defined in DSO

/lib64/librt.so.1 so try adding it to the linker command line

/lib64/librt.so.1: could not read symbols: Invalid operation

collect2: ld returned 1 exit status

Try adding -lrt to the linker flags to Make file for the folder where this is crashing

LIBS=-lrt

also may need to install: xml2-config and curl-config - and associated libs

it is likely that /home/lbathen/Development/Xen/xen-4.2-testing.hg/stubdom/ will have a pointer to xenstore, this will break the build so make sure to delete the xenstore directory as it is now in toos/xenstore

Categories: Contains Warning | Users | Host Install

Revision:

Installing Xen from source isn’t too hard when you know the steps, but I was surprised to find there weren’t many full start to finish explanations out there.

Here’s how on a fresh Ubuntu 12.10 amd64 server.

Please be aware: I had some difficulties myself with installing an Ubuntu 12.04 domU after the install, so I found I needed to revert to Xen 4.1.4. I’ll update this here if I figure out what went wrong there. I also wrote up a post for theXen 4.1.4 Install.

Ubuntu Installation

I used the latest Ubuntu 12.10 amd64 server ISO from ubuntu.com.

During the install of Ubuntu for the Partitioning method choose “Guided – use the entire disk and setup LVM”. Then, when prompted to enter “Amount of volume group to use for guided partitioning:” Enter a value just large enough for the Xen Dom0 system, leaving the rest for virtual disks. Enter a value smaller than the size of your installation drive.

For example 10 GB or even 5 GB should be large enough for a minimal Xen Dom0 system. Entering a percentage of maximum size (e.g. 25%) is also a reasonable choice.

For everything else, just defaults are fine, and recommend to install OpenSSH, but nothing else from the installer package selections.

Install the pre-requisites

apt-get install python-dev

apt-get build-dep xen

apt-get install libc6-dev libglib2.0-dev libyajl-dev yajl-tools libbz2-dev bison flex zlib1g-dev git-core texinfo debhelper debconf-utils debootstrap fakeroot

Download and build the source

wget http://bits.xensource.com/oss-xen/release/4.2.1/xen-4.2.1.tar.gz

tar zxvf xen-4.2.1.tar.gz

cd xen-4.2.1

./configure

vi ./.config

#

PYTHON_PREFIX_ARG=--install-layout=deb

#

make world

make deb

This creates a “dist” directory which contains the results of your build.

Install

cd ./dist

./install.sh

# Important

update-grub

ls -al /boot/xen*

# You’ll have something like the below

#

-rw-r–r– 1 root root 802314 Feb 18 09:11 /boot/xen-4.2.1.gz

lrwxrwxrwx 1 root root 12 Feb 18 09:11 /boot/xen-4.2.gz -> xen-4.2.1.gz

lrwxrwxrwx 1 root root 12 Feb 18 09:11 /boot/xen-4.gz -> xen-4.2.1.gz

lrwxrwxrwx 1 root root 12 Feb 18 09:11 /boot/xen.gz -> xen-4.2.1.gz

-rw-r–r– 1 root root 15388780 Feb 18 09:11 /boot/xen-syms-4.2.1

#

# Setup the init scripts, purposely skipping xend since ‘xl’ is the standard now

update-rc.d xencommons defaults 19 18

update-rc.d xendomains defaults 21 20

update-rc.d xen-watchdog defaults 22 23

grep Xen /boot/grub/grub.cfg

# You should see “Ubuntu GNU\/Linux, with Xen hypervisor” in there, that’s where I got the below from

# Now set it as the boot default so we have Xen upon rebooting

sed -i ‘s/GRUB_DEFAULT=.*\+/GRUB_DEFAULT=”Ubuntu GNU\/Linux, with Xen hypervisor”/’ /etc/default/grub

# Update grub with the new default

update-grub

# Reboot

reboot

# You know it’s working if ‘xl’ is working

xl list

#

Name ID Mem VCPUs State Time(s)

Domain-0 0 7660 2 r—– 6.9

#

Create a test VM

xen-create-image is very handy for a quick way to get a VM running:

xen-create-image –hostname=blah \

–memory=512mb \

–vcpus=2 \

–lvm=some_lvm_volume_group_with_free_space \

–dhcp \

–pygrub \

–dist=precise

Re-use the files

You can save that ‘dist’ directory and install it on other machines for the future.

tar zcvf xen-dist.tgz ./xen-4.2.1/dist

Make deb

You’ll notice above I used ‘make deb’, which you can technically skip if you want. All that did was create ‘dist/xen-upstream-4.2.1.deb’, which is a helper debian package that includes a list of all of the files in dist/.

This package doesn’t really do anything, it’s just there to help you keep track of the Xen files you’ve manually installed. You can install the package like this:

dpkg -i ./xen-upstream-4.2.1.deb

Make sure your /etc/grub.d/ looks something like this:

/etc/grub.d/00_header /etc/grub.d/20_linux_xen /etc/grub.d/25_linux /etc/grub.d/30_uefi-firmware /etc/grub.d/41_custom

/etc/grub.d/05_debian_theme /etc/grub.d/20_memtest86+ /etc/grub.d/30_os-prober /etc/grub.d/40_custom /etc/grub.d/README

ISSUES:

If there is an issue with qemu*-dir directories, delete them and re-configure, then make clean, then do make world again :)

NOTES:

Find out domu lvm to dm mapping:

sudo lvdisplay|awk '/LV Name/{n=$3} /Block device/{d=$3; sub(".*:","dm-",d); print d,n;}'

You might need to run ldconfig afterwards as the libstat shared objects might not be loaded properly (e.g., libxenstat.so.0).