Calculate DOM0 memory size

When using Oracle VM server as a virtualization platform you will have to ensure that DOM0 has enough memory allocated to it. Dom0 is the initial domain started by the Xen hypervisor on boot. Dom0 is an abbrevation of "Domain 0" (sometimes written as "domain zero" or the "host domain"). Dom0 is a privileged domain that starts first and manages the DomU unprivileged domains.

To ensure the correct amount of memory is allocated to DOM0 Oracle propagates the below mentioned algorithm to be used:

dom0_mem = 502 + int(physical_mem * 0.0205)

As an example this would mean that a server with 2 GB of physical memory would need 543MB of memory allocated to DOM0. A server with 32 GB of physical memory would need 1173MB of memory allocated to DOM0..

To change the Dom0 memory allocation, edit the /boot/grub/grub.conf file on the Oracle VM Server and change the dom0_mem parameter, for example, to change the memory allocation to 1024 MB, edit the file to be:

kernel /xen.gz console=com1,vga com1=38400,8n1 dom0_mem=1024M

Using XenStore in Oracle VM

When you are developing a private cloud based upon the Oracle portfolio you will most likely make use of Oracle VM for your non-Sparc deployments. It is good to know that Oracle VM is based upon the opensource Xen Hypervisor developed by the Xen project.

“The Xen Project hypervisor is an open-source type-1 or baremetal hypervisor, which makes it possible to run many instances of an operating system or indeed different operating systems in parallel on a single machine (or host). The Xen Project hypervisor is the only type-1 hypervisor that is available as open source. It is used as the basis for a number of different commercial and open source applications, such as: server virtualization, Infrastructure as a Service (IaaS), desktop virtualization, security applications, embedded and hardware appliances. The Xen Project hypervisor is powering the largest clouds in production today.”

The above statement from the Xen project makes the statement that it is powering some of the largest clouds in production today which is a good thing to know if you are using Oracle VM. This means that the code adopted within Oracle VM is also empowering numerous other clouds and very large clouds for that matter.

To be able to make full use of the Xen part that makes Oracle VM it is advisable to start understanding how Xen by itself works. Reason for this is that Oracle has not adopted, or documented all features from Xen in its full extend while they are still largely available for you to use.

Even though Oracle has provided a great implementation of the Xen hypervisor and the tooling provided, especially with the integration of Oracle Enterprise Manager, is making life easy there are moments you want to do more than the standard implementation allows you. One of my recent experiences where this was the case was in relation to communication to XenStore. In basics XenStore is a shared storage space between the different domains running on the hypervisor. XenStore is maintained by the xentstored deamon in Dom0 and the operating systems running on the DomU’s can communicate with it via the XenBus.

Even though the default way of communicating to the XenStore is via a number of commands which do not require that you need to know the exact location of the XenStore data the data is actually located in a file. You can find the XenStore data file /var/lib/xenstored where the file is located and named tdb on Dom0. The name TDB stands for Tree Database.

One of the things that enables XenStore you to do is to retrieve information from the XenStore. When you are building more advanced deployment models you can, for example, store information in the XenStore and read this when you boot a guest for the first time and use this input in the configuration process. Also, having access to the XenStore from your guest VM can help you build better reports from the Guest point of view. Gregory Guillou wrote a great blogpost on this subject, the reason he was interested in using the XenStore was to be able to find the relation of a disk presented to a VM and the underlying storage from a VM guest point of view. This could help him to write additional code to do a snapshot from storage for which he needs to have the information about the underlying storage.

Compiling the XenStore tools
Oracle is not providing a RPM for the XenStore tools however you are able to download the Oracle VM sourcecode which contains the source code of the XenStore tools that you can then compile yourself. Or, when you need to do this often you can create a RPM yourself so you can easily distribute the XenStore tools to your guest VM’s.

Once you have downloaded the Oracle VM source code from the Oracle Download site you should open the .iso file and locate the correct source RPM in the SRPMS directory. For the version I am currently running this is xen-4.1.3-25.el5.94.src.rpm however naming can differ per version. Upload the .rpm file to the guest VM and install it.

[root@test1 ~]# rpm -ivh xen-4.1.3-25.el5.94.src.rpm
warning: xen-4.1.3-25.el5.94.src.rpm: Header V3 DSA/SHA1 Signature, key ID 1e5e0159: NOKEY
   1:xen                    ########################################### [100%]
[root@test1 ~]#

This should have provided you a new directory under your account named /rpmbuild/SOURCES in our case this is /root/rpmbuild/SOURCES which contains a large set of files. The only file we are interested in is the file that contains the source code to be used to compile the XenStore tools. In our case this is xen-4.1.3-ovs.tar.gz

[root@test1 SOURCES]# cd /root/rpmbuild/SOURCES/
[root@test1 SOURCES]#
[root@test1 SOURCES]# tar -zxvf xen-4.1.3-ovs.tar.gz

The above extracts the sources we need (and others) and we can go into the location where the sourcecode for the XenStore tools is located and make the code. However before you can make it you have to ensure you have some prerequisites completed that will have to be there before you can compile.  The below yum install command will install all required prerequisites if they are not installed yet. The below has been tested on Oracle Linux 6 and has been extended with gettext and patch based upon the information from the blog written by Gregory Guillou on this subject.

yum install oracle-rdbms-server-11gR2-preinstall libuuid-devel openssl-devel ncurses-devel dev86 iasl python-devel SDL-devel gettext patch

As soon as you have ensured that the prerequisites are all in place you can start the make command in the right directory.

[root@test1 tools]# cd /root/rpmbuild/SOURCES/xen-4.1.3-ovs/tools
[root@test1 tools]# make 

When done without any errors or warnings you can now use this to install the XenStore tools onto your guest VM.

[root@test1 tools]# cd /root/rpmbuild/SOURCES/xen-4.1.3-ovs/tools/misc
[root@test1 tools]#
[root@test1 misc]# install xen-detect /usr/local/bin
[root@test1 xenstore]# cd /root/rpmbuild/SOURCES/xen-4.1.3-ovs/tools/xenstore
[root@test1 xenstore]#
[root@test1 xenstore]# install libxenstore.so.3.0 /usr/local/lib
[root@test1 xenstore]# install xenstore xenstore-control /usr/local/bin
[root@test1 xenstore]#
[root@test1 xenstore]# cd /usr/local/bin
[root@test1 bin]#
[root@test1 bin]# ln -f xenstore xenstore-chmod
[root@test1 bin]# ln -f xenstore xenstore-exists
[root@test1 bin]# ln -f xenstore xenstore-list
[root@test1 bin]# ln -f xenstore xenstore-ls
[root@test1 bin]# ln -f xenstore xenstore-read
[root@test1 bin]# ln -f xenstore xenstore-rm
[root@test1 bin]# ln -f xenstore xenstore-write
[root@test1 bin]#

This in basics should have compiled and installed the XenStore tooling into your Guest VM. As you can see, there is a reason why you should want to create your own RPM in case you need to install this on more than one machine. However, the above gives you a good insight and starting point to build your own RPM.

Using the XenStore tools
Before you can use the XenStore tools you will have to set the correct path for LD_LIBRARY_PATH and you will have to mount Xen File System.

[root@test1 local]# export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
[root@test1 local]# mount -t xenfs none /proc/xen

Once you have done this you can use the XenStore tools. In basics you have xenstore-chmod, xenstore-exists, xenstore-list, xenstore-ls, xenstore-read, xenstore-rm and xenstore-write to your exposal.

understanding the XEN xm list command

When maintaining a XEN Oracle VM server one of the commands you will most likely be using a lot is the list command. list will give you a overview of which virtual machine is where, which virtual machine is running, which virtual machine is down and such things. Understanding the list command is vital to understanding your server landscape and monitor things.

We will be working in this post on the XEN xm shell as described in a previous post. XEN states the following about the list command: "List information about all/some domains".

xm> help list
Usage: xm list [options] [Domain, ...]

List information about all/some domains.
-l, --long Output all VM details in SXP
--label Include security labels
--state= Select only VMs with the specified state

xm>

the help list command is already showing you some of the options you will have at the list command. issuing the list command 'stand alone' is providing a output like the one below:

xm> list
Name ID Mem VCPUs State Time(s)
48_VM1 3 1024 1 -b---- 27356.4
Domain-0 0 581 2 r----- 395481.0
xm>

ID is showing you the 'internal' XEN domain ID or 'Virtual machine ID'. this ID is used at other commands to issue commands to take actions on a domain. For example if you want to shutdown a domain you issue the command shutdown -w (domain ID).

Mem is showing you the memory assigned to the domain in megabytes. In the example above you can see that domain 48_VM1 has 1024MB allocated and Domain-0 has 581MB assigned. This is not saying anything about how much of this memory is used within the domain by the virtual machine.

Time(s) is showing the time the domain is running.

State; the State field lists 6 states for a Xen Domain, and which ones the current Domain is in.
r - running
The domain is currently running on a CPU
b - blocked
The domain is blocked, and not running or runnable. This can be caused because the domain is waiting on IO (a traditional wait state) or has gone to sleep because there was nothing else for it to do.
p - paused
The domain has been paused, usually occurring through the administrator running xm pause. When in a paused state the domain will still consume allocated resources like memory, but will not be eligible for scheduling by the Xen hypervisor.
s - shutdown
The guest has requested to be shutdown, rebooted or suspended, and the domain is in the process of being destroyed in response.
c - crashed
The domain has crashed, which is always a violent ending. Usually this state can only occur if the domain has been configured not to restart on crash. See xmdomain.cfg for more info.
d - dying
The domain is in process of dying, but hasn't completely shutdown or crashed.

By adding --label to the list command you will get the security label added at the end of the records. Below you can see a example where I did not set security labels to the domains. For more information about adding security labels to XEN domains take a look at linuxtopia.org

xm> list --label
Name ID Mem VCPUs State Time(s) Label
48_VM1 3 1024 1 -b---- 27372.1 INACTIVE
Domain-0 0 581 2 r----- 395658.0 INACTIVE
xm>

By adding a -l or --long to the list command you will get a lot more information about the domains running on your XEN server. The information is represented in a S-expression format. Xen states the following about the list -l option: "If --long is specified, the output for xm list is not the table view shown above, but instead is an S-Expression representing all information known about all domains asked for. This is mostly only useful for external programs to parse the data.". I tend to disagree with XEN on the "This is mostly only useful for external programs to parse the data" section. You can quickly see a lot of information you might need about a domain in the S-Expression representation.

Wikipedia states the following about S-Expressions:
"The term S-expression or sexp (where S stands for symbolic) refers to a convention for representing semi-structured data in human-readable textual form. S-expressions are probably best known for their use in the Lisp family of programming languages. Other uses of S-expressions are in Lisp-derived languages such as DSSSL, and as mark-up in communications protocols like IMAP and John McCarthy's CBCL. The details of the syntax and supported data types vary in the different languages, but the most common feature among these languages is the use of S-expressions as parenthesized prefix notation (sometimes known as Cambridge Polish notation).

S-expressions are used for both code and data in Lisp (see McCarthy Recursive Functions of Symbolic Expressions [1]). S-expressions were originally intended only for data to be manipulated by M-expressions, but the first implementation of Lisp was an interpreter of S-expression encodings of M-expressions, and Lisp programmers soon became accustomed to using S-expressions for both code and data.".

The S-expression output of XEN on a domain or on all domains will be explained in a future post. This is to much to discuss right now.

XEN xm shell

When operating a XEN server and/or Oracle VM from a command line you can do almost everyting via de xm command. For example if you would like to view a list of the current domains on your server you can enter the command 'xm list' to show it on the console.

[root@boxjohan ~]# xm list
Name ID Mem VCPUs State Time(s)
48_VM1 3 1024 1 -b---- 27304.2
Domain-0 0 581 2 r----- 394864.9

however, when you have to do a lot of maintenance on your XEN and/or Oracle VM server you might not want to put xm in front of all your commands. So you can work on a XEN xm shell which is developed as a shell purely for working on your XEN and/or Oracle VM server. You can enter the shell by issuing the 'xm shell' command.

[root@boxjohan ~]# xm shell
The Xen Master. Type "help" for a list of functions.
xm>

You will notice that you are now in the XEN shell and your prompt is now xm> when you issue the help command you will notice that this will give you the exact same output as issuing xm from a bash command line.

xm> help
Usage: xm [args]

Control, list, and manipulate Xen guest instances.

Common 'xm' commands:

console Attach to 's console.
create Create a domain based on .
new Adds a domain to Xend domain management
delete Remove a domain from Xend domain management.
destroy Terminate a domain immediately.
dump-core Dump core for a specific domain.
help Display this message.
list List information about all/some domains.
mem-set Set the current memory usage for a domain.
migrate Migrate a domain to another machine.
pause Pause execution of a domain.
reboot Reboot a domain.
restore Restore a domain from a saved state.
resume Resume a Xend managed domain
save Save a domain state to restore later.
shell Launch an interactive shell.
shutdown Shutdown a domain.
start Start a Xend managed domain
suspend Suspend a Xend managed domain
top Monitor a host and the domains in real time.
unpause Unpause a paused domain.
uptime Print uptime for a domain.
vcpu-set Set the number of active VCPUs for allowed for
the domain.

can either be the Domain Name or Id.
For more help on 'xm' see the xm(1) man page.
For more help on 'xm create' see the xmdomain.cfg(5) man page.

For a complete list of subcommands run 'xm help'.
xm>

By using the xm shell function you will be able to more closely work with your XEN and/or Oracle VM server however you can also do anything what you want without invoking the shell, you will be able to do anything by putting xm in front of the command you would enter in the shell.