Getting Started

How to use the version we are upstreaming: gEDF scheduler only

If you are using the version (https://github.com/pennpanda/RT-Xen, branch rtxen-v2.0-upstream) we are upstreaming to Xen code, which only has gEDF scheduler inside, the following scenario shows you how to configure and use the real-time scheduler (rt).

Hardware
The machine has Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz, which has 6 cores (12 hardware threads). If you used some other machines with less cores, please make sure the cpu index you used in the command is less than the number of cores your machine has. 

Grub
The grub menu we used to boot the rt scheduler is as follows:

menuentry 'Ubuntu GNU/Linux, with RT-Xen 4.5(rt_ds) and Linux 3.11.0-15-generic' --class ubuntu --class gnu-linux --class gnu --class os --class xen {

    insmod part_msdos

    insmod ext2

    set root='(hd0,msdos1)'

    search --no-floppy --fs-uuid --set=root b26736f6-3b5e-4d16-b840-9a80081b93b5

    echo    'Loading Xen xen ...'

    multiboot   /boot/xen-4.5-unstable.gz placeholder dom0_max_vcpus=4 dom0_memory=2048M sched=rt_ds console=ttyS0 com1=115200n8 console=com1

    echo    'Loading Linux 3.11.0-15-generic ...'

    module  /boot/vmlinuz-3.11.0-15-generic placeholder root=UUID=b26736f6-3b5e-4d16-b840-9a80081b93b5 ro  quiet splash console=hvc0 earlyprintk=xen

    echo    'Loading initial ramdisk ...'

    module  /boot/initrd.img-3.11.0-15-generic

}

Scenario
After you boot the system with the rt scheduler, you can use the following scenario to try the functionality of this rt scheduler: 
//list each vcpu's parameters of each domain in cpu pools using rt scheduler
//the time is in microsecond
#xl sched-rt
Cpupool Pool-0: sched=EDF
Name ID VCPU Period Budget
Domain-0 0 0 10000 10000
Domain-0 0 1 20000 20000
Domain-0 0 2 30000 30000
Domain-0 0 3 10000 10000
litmus1 1 0 10000 4000
litmus1 1 1 10000 4000

//set the parameters of the vcpu 1 of domain litmus1:
# xl sched-rt -d litmus1 -v 1 -p 20000 -b 10000

//domain litmus1's vcpu 1's parameters are changed, display each VCPU's parameters separately:
# xl sched-rt -d litmus1
Name ID VCPU Period Budget
litmus1 1 0 10000 4000
litmus1 1 1 20000 10000

// list cpupool information
xl cpupool-list
Name CPUs Sched Active Domain count
Pool-0 12 rt y 2

//create a cpupool test
#xl cpupool-cpu-remove Pool-0 11
#xl cpupool-cpu-remove Pool-0 10
#xl cpupool-create name=\"test\" sched=\"credit\"
#xl cpupool-cpu-add test 11
#xl cpupool-cpu-add test 10
#xl cpupool-list
Name CPUs Sched Active Domain count
Pool-0 10 rt y 2
test 2 credit y 0

//migrate litmus1 from cpupool Pool-0 to cpupool test.
#xl cpupool-migrate litmus1 test

//now litmus1 is in cpupool test
# xl sched-credit
Cpupool test: tslice=30ms ratelimit=1000us
Name ID Weight Cap
litmus1 1 256 0


How to use full-featured RT-Xen 2.x
If you are using the full-featured version (https://github.com/pennpanda/RT-Xen, branch rtxen-v2.0-full-feature), which has global EDF, global RM, partitioned EDF and partitioned RM inside, the following scenario shows you how to configure and use those schedulers.

Please note: The rtglobal scheduler has global EDF and global RM inside, and you can switch between these two schedulers on the fly.
The rtpartition scheduler has partitioned EDF and partitioned RM inside; When you set the parameter of dom0, the scheduler will switch between partitioned EDF and partitioned RM interleavly. You could also follow the way rtglobal scheduler does (in function rtglobal_sys_cntl @ xen/common/sched_rtglobal.c) to implement the tool stack to set a specific scheduler.

Hardware (same with above)
The machine has Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz, which has 6 cores (12 hardware threads). If you used some other machines with less cores, please make sure the cpu index you used in the command is less than the number of cores your machine has. 

Grub
The grub menu we used to boot the rtglobal scheduler is as follows:

menuentry 'Ubuntu GNU/Linux, with RT-Xen 4.3.0(rtglobal) and Linux 3.11.0-15-generic' --class ubuntu --class gnu-linux --class gnu --class os --class xen {

    insmod part_msdos

    insmod ext2

    set root='(hd0,msdos1)'

    search --no-floppy --fs-uuid --set=root b26736f6-3b5e-4d16-b840-9a80081b93b5

    echo    'Loading Xen xen ...'

    multiboot   /boot/xen-4.3.0.gz placeholder dom0_max_vcpus=4 dom0_memory=2048M sched=rtglobal console=ttyS0 com1=115200n8 console=com1

    echo    'Loading Linux 3.11.0-15-generic ...'

    module  /boot/vmlinuz-3.11.0-15-generic placeholder root=UUID=b26736f6-3b5e-4d16-b840-9a80081b93b5 ro  quiet splash console=hvc0 earlyprintk=xen

    echo    'Loading initial ramdisk ...'

    module  /boot/initrd.img-3.11.0-15-generic

}

Scenario
After you boot the system with the rtglobal scheduler, you can use the following scenario to try the functionality of this rtglobal scheduler: 
//list each vcpu's parameters of each domain in cpu pools using rtglobal scheduler
//the time is in millisecond
#xl sched-rtglobal
Cpupool Pool-0: sched=EDF
Name ID VCPU Period Budget Extra
Domain-0 0 0 10 10 0
Domain-0 0 1 20 20 0
Domain-0 0 2 30 30 0
Domain-0 0 3 10 10 0
litmus1 1 0 10 4 0
litmus1 1 1 10 4 0
(Note: Extra column has no meaning in this version and is always 0.)

//set the parameters of the vcpu 1 of domain litmus1:
# xl sched-rtglobal -d litmus1 -v 1 -p 20 -b 10

//domain litmus1's vcpu 1's parameters are changed, display each VCPU's parameters separately:
# xl sched-rtglobal -d litmus1
Name ID VCPU Period Budget Extra
litmus1 1 0 10 4 0
litmus1 1 1 20 10 0

// list cpupool information
xl cpupool-list
Name CPUs Sched Active Domain count
Pool-0 12 rtglobal y 2

//create a cpupool test
#xl cpupool-cpu-remove Pool-0 11
#xl cpupool-cpu-remove Pool-0 10
#xl cpupool-create name=\"test\" sched=\"credit\"
#xl cpupool-cpu-add test 11
#xl cpupool-cpu-add test 10
#xl cpupool-list
Name CPUs Sched Active Domain count
Pool-0 10 rtglobal y 2
test 2 credit y 0

//migrate litmus1 from cpupool Pool-0 to cpupool test.
#xl cpupool-migrate litmus1 test

//now litmus1 is in cpupool test
# xl sched-credit
Cpupool test: tslice=30ms ratelimit=1000us
Name ID Weight Cap
litmus1 1 256 0

//change the scheduling policy on the fly
//change to glbal RM scheduler to global EDF scheduler
# xl sched-rtglobal -s RM
Input schedule scheme from user is: RM
Schedule scheme is RM now

//change to global EDF scheduler from global RM scheduler
# xl sched-rtglobal -s EDF
Input schedule scheme from user is: EDF 
Schedule scheme is EDF now

How to use rt-xen v1.x

Hardware: 
Dell Q9400 quad-core machine without hyper-threading. Each core runs at 2.66 GHz. Domian 0 was given one core and 1 GB memory, other guest domains were pinned to another core and given 256 MB memory each.

Software:
OS: Fedora 13, 64 bit with para kernel 2.6.32.25.
Xen: Compiled from 4.0.1 source.


How to Test RT-XEN
Please goto Publication section to see the details and results of the evaluation.

We are using Sporadic Server Scheduler as the hypervisor scheduler, and Rate Monotonic scheduler as the Linux scheduler.

In [1], there are in total four requirements to the task running on it:
  • All domain periods must be harmonic
  • A domain’s period must be <= di / (2 + ceil(ci/B)), where ci and di are the execution time and deadline for any task in the domain and B is the domain's budget
  • A domain’s period should be the greatest common divisor (GCD) or a divisor of the GCD of all of the periods of its constituent tasks.
  • A domain’s total utilization is restricted by RM theory, where its Utilization must be <= (B/P) * n * ( 2^(1/n) - 1 ), n is the total number of tasks

To create a periodic real time task in Linux, we use the CLOCK_REALTIME to trigger the periodic task, and measured 1ms workload on our test platform. The details of this periodic task can be found here.

Following the task requirement, you can configure the budget and period of the domain with corresponding generated tasks. The program we use can be found here.

In the test program, we record each job's start time, dispatch time, finish time, and deadline. Then we can use this record to calculate the deadline miss ratio of each task.

How to See if My task is schedulable or not?
Following [1]'s theory requirement, we developed a small tool to test if the task set is theoretically schedulable on RT-Xen or not. Download it here. A sample input is here.
The sample input should contains:
-------------
number of domains
for each domain:
number of tasks
tasks' priority, execution time, deadline(we assume that deadline = period)
-------------

to use it, use the following command:
gcc feasible_taiwei.c -o test.out
./test.out test1.txt
The sample output should be:

----------------
turtle141:Tools xisisu$ gcc feasible_taiwei.c
turtle141:Tools xisisu$ ./a.out test1.txt 
In total, 5 domains

Domain 0, have 5 tasks
priority: 99, exec_time: 1, deadline: 75
priority: 98, exec_time: 1, deadline: 100
priority: 97, exec_time: 12, deadline: 125
priority: 97, exec_time: 1, deadline: 125
priority: 96, exec_time: 1, deadline: 200
The Server's Budget: 5, Period: 25
total utilization: 0.132333, utilization bound: 0.146954

Domain 1, have 10 tasks
priority: 99, exec_time: 7, deadline: 150
priority: 99, exec_time: 3, deadline: 150
priority: 99, exec_time: 2, deadline: 150
priority: 99, exec_time: 2, deadline: 150
priority: 99, exec_time: 3, deadline: 150
priority: 99, exec_time: 1, deadline: 150
priority: 99, exec_time: 1, deadline: 150
priority: 99, exec_time: 1, deadline: 150
priority: 98, exec_time: 1, deadline: 350
priority: 97, exec_time: 1, deadline: 1450
The Server's Budget: 10, Period: 50
total utilization: 0.136880, utilization bound: 0.143090

Domain 2, have 10 tasks
priority: 99, exec_time: 12, deadline: 300
priority: 99, exec_time: 13, deadline: 300
priority: 99, exec_time: 9, deadline: 300
priority: 99, exec_time: 4, deadline: 300
priority: 99, exec_time: 2, deadline: 300
priority: 99, exec_time: 1, deadline: 300
priority: 99, exec_time: 2, deadline: 300
priority: 99, exec_time: 2, deadline: 300
priority: 98, exec_time: 1, deadline: 400
priority: 97, exec_time: 1, deadline: 500
The Server's Budget: 22, Period: 100
total utilization: 0.154500, utilization bound: 0.157399

Domain 3, have 6 tasks
priority: 99, exec_time: 14, deadline: 600
priority: 99, exec_time: 3, deadline: 600
priority: 99, exec_time: 2, deadline: 600
priority: 99, exec_time: 3, deadline: 600
priority: 99, exec_time: 3, deadline: 600
priority: 98, exec_time: 60, deadline: 800
The Server's Budget: 35, Period: 200
total utilization: 0.116667, utilization bound: 0.127510

Domain 4, have 5 tasks
priority: 99, exec_time: 11, deadline: 1200
priority: 99, exec_time: 27, deadline: 1200
priority: 99, exec_time: 13, deadline: 1200
priority: 99, exec_time: 5, deadline: 1200
priority: 98, exec_time: 164, deadline: 1600
The Server's Budget: 82, Period: 400
total utilization: 0.149167, utilization bound: 0.150628

Total U is 1.000000

Passed Utilization check!
Passed Harmonic check!
Passed Single Schedulability check!

Schedulable!!!
----------------

Reference:
[1] T.-W. Kuo and C.-H. Li, “A fixed-priority-driven open environment for real-time applications,” Real-Time Systems Symposium, IEEE International, vol. 0, p. 256, 1999.:


How to setup the environment for network experiments

RT-Xen doesn't do any change to the network subsystem of Xen. If you want to run some network experiements on RT-Xen, please setup the environment based on this and 
this tutorial. 

Basically, you need to 
1. Setup the software bridge utility in Domain 0. 
In Fedora/CentOS, please run 
yum install bridge-utils
to install the bridge package. 
In Ubuntu/Debian, run
apt-get install bridge-utils
to install. You may need the 'sudo' priviledge to run the commands above.

2. Modify the network interface configuration files.
In Fedora/CentOS, the configuration files are under 
/etc/sysconfig/network-scripts/
Normally, you will find two configuration files under this path, ifcfg-eth0 and ifcfg-peth0 (names could be slightly different according to the Xen version), which are created by the network-bridge script of xend (under /etc/xen/scripts/). By default, eth0 is the bridge device, while the peth0 is the network device of Domain 0. In ifcfg-eth0, there must be:
DEVICE=eth0
TYPE=Bridge
BOOTPROTO=static # =dhcp if you need a dynamic IP address from your network service provider
BROADCAST=*.*.*.*
IPADDR=*.*.*.*
NETMASK=*.*.*.*
ONBOOT=yes
NM_CONTROLLED=no
The ifcfg-peth0 should be like:
DEVICE=peth0
TYPE=Ethernet
BOOTPROTO=static # or =dhcp
BROADCAST=*.*.*.*
IPADDR=*.*.*.*
NETMASK=*.*.*.*
ONBOOT=yes
BRIDGE=eth0
NM_CONTROLLED=no
In Ubuntu/Debian, the configuration file is
/etc/network/interfaces
Please follow this instructions to add the setup for eth0 and peth0 in the configuration file.

3. Modify the config file of the guest domain. 
If your guest domain has a network interface, please add the following line into the config file:
vif=[ 'mac=*:*:*:*:*:*,bridge=eth0' ]

We recommend you to pin Domain 0 to a dedicated physical CPU core to better handle the communication and interrupts. To configure this, you can modify the grub and specify
'dom0_max_vcpus=1, dom0_vcpus_pin' as part of the kernel start-up parameters.