LTSP + openMosix: Integration How-To

Revision 4.1

What is this?

This document will provide details (via an example set-up) of combining the openMosix clustering system with the K12LTSP 2.1.1 distribution, an innovative GNU/Linux distribution based on RedHat 7.3 & LTSP 3.0.x (a set of Open Source software packages from the Linux Terminal Server Project that makes diskless workstations work). The openMosix project, headed by Moshe Bar, is a completely Open Source (licensed under the GNU GPLv2) fork of the MOSIX clustering system.

openMosix is a clustering solution that in basic terms combines the CPU resources of server & client systems (or nodes) into what could be compared to an SMP processing system. Processes get "moved" or, in other words, migrated dynamically via the openMosix algorithm to the nodes (Linux Terminal Client computers) in the cluster that can be most effective in running them at any given time. This process provides load-balancing across the cluster & optimal usage of the computing resources available. It's fully automated. No special programming or recompiling of applications is required as in other clustering systems. Regarding the purpose of this How-To, the hope or theory is that since the K12LTSP client is simply executing XWindows and nothing more that openMosix can be implemented to utilize the idle processor cycles of these client terminal computers to maximize the efficiency of the entire system.

This combination also provides a means to quickly and effectively cluster many computers together without requiring a full-blown installation of GNU/Linux on each node computer. In fact the client computers don't even need a hard drive to be utilized in this setting.

This document was originally formulated as a result of Richard Camp's very useful How-To, "Mosix Cluster with Diskless Nodes", which detailed adding MOSIX clustering functionality to an installation of the K12LTSP 2.0.1 distribution. Richard's assistance and suggestions were (and are) most appreciated!

I alone am responsible for this particular document and all of the mistakes in it are mine! :-)
Mandatory Disclaimer: Use this information at your own risk. No warranty or guarantee is expressed or implied. I am not responsible for any damage resulting from your implementation of the information contained in this document. Again, use this How-To at your own risk.

Having said that, let's get going!

Why openMosix instead of the original MOSIX?

As I stated previously, Richard Camp posted an excellent How-To on installing the original MOSIX system with the K12LTSP 2.0.1 distribution. It pointed me in the right direction and I'm truly grateful for his efforts. Without which, who knows if I'd have ever gotten anywhere with the concept of combining LTSP & MOSIX. Unfortunately in my case, the MOSIX install script that came with the MOSIX download always failed when the 2.4.17 kernel compile would throw errors during the modules_install phase. MOSIX simply would not patch & install automatically with the installation script. I began investigating a manual MOSIX kernel patch & userland tools installation and came across the openMosix project in the process. I found openMosix very easy to implement. I also found that (at that time) the licensing status of MOSIX was not clearly indicated, which leads to the second item...
MOSIX is no longer Open Source software (See the MOSIX SOFTWARE LICENSE AGREEMENT). openMosix is open source code (GPLv2). This is significant. Anyone interested in these two products should carefully consider the licensing agreements under which they fall.

You may wish to review the following for more information:

Interview with Moshe Bar (founder of openMosix & a former project manager and developer of MOSIX):

The openMosixFAQ

openMosix Documentation (recommended!)

openMosix is ideal for our purposes. It is easy to install, it's licensed under the GPL, and it works!

My Test Set-up used in this example:

Hardware:
        Server:
        AMD K6-III 450Mhz, 384MB RAM,
        13GB WD IDE HDD, 10/100Mbit NIC. K12LTSP 2.1.1 installed.
        Clients:
        Various systems, booting from floppy:
        486 & Pentium.
        10/100Mbit NICs. 16-32MB RAM
        Network:
        16-port SMC EZ Switch 10/100Mbit

Nothing spectacular here just an odd assortment of stuff, but it's been a useful test setup for me. And, even more importantly, it actually works! :-)

Software:
K12LTSP 2.1.1 Distribution

LTSP Project Files (particularly "ltsp_initrd_kit-3.0.5-i386.tgz")

openMosix Project Files

Files needed in this example (Obtain the following openMosix files from SourceForge):

The K12LTSP 2.1.1 Distribution set
ltsp_initrd_kit-3.0.5-i386.tgz (from the LTSP File List page)
openmosix-kernel-2.4.18-openmosix4.i386.rpm (Server Kernel)
Note: There is no RPM available for my K6-III so I have to use the i386 version. Choose the kernel appropriate for your server platform.
openmosix-tools-0.2.4-1.i386.rpm (Userspace-Tools for openMosix)
openmosix-kernel-2.4.18-openmosix4.src.rpm (Source files for building the LTSP/openMosix Client Kernel)

mcedit is the text editor that will be used throughout this example. If you wish to utilize it, you can install it from the RPMS section on Disc 1 of the K12LTSP 2.1.1 distribution via: rpm -ivh mc-4.5.55-5.i386.rpm, or substitute the editor of your choice.

The step-by-step:

(Note: items following the ">" are command line items you need to run.)

Download the files listed above.
Install the K12LTSP 2.1.1 distribution. When prompted in the Package Group Selection portion of the installation add "Software Development" as one of the selections (helps to have these packages installed for building the LTSP client Kernel for openMosix).
Copy the downloaded openMosix & LTSP files (ltsp_initrd_kit-3.0.5-i386.tgz) to your LTSP server.
Navigate to the folder containing the downloads for the project:
> cd /your-download-folder/
Install the openMosix kernel to the server via the command:
> rpm -ivh openmosix-kernel-2.4.18-openmosix4.i386.rpm
This adds the kernel to the grub bootloader and a link (/boot/vmlinuz-openmosix) that can be used for lilo (just make the appropriate entries in lilo.conf and run lilo -v).
Install the Userspace-tools "openmosix-tools-0.2.4-1.i386.rpm" to the server via the command:
> rpm -ivh openmosix-tools-0.2.4-1.i386.rpm
Install the openMosix kernel source files via the command:
> rpm -ivh openmosix-kernel-2.4.18-openmosix4.src.rpm
> cp /usr/src/redhat/SOURCES/linux-2.4.18.tar.gz /usr/src/
> cp /usr/src/redhat/SOURCES/openMosix-2.4.18-4.gz /usr/src/
> cp /your-download-folder/ltsp_initrd_kit-3.0.5-i386.tgz /usr/src/
> cd /usr/src
> tar xzvf ltsp_initrd_kit-3.0.5-i386.tgz
> tar xzvf linux-2.4.18.tar.gz
> mv /usr/src/linux /usr/src/linux-2.4.18-om4
> ln -s linux-2.4.18-om4 linux
> cd /usr/src/linux
> zcat ../openMosix-2.4.18-4.gz |patch -p1
> make mrproper
> mcedit /usr/src/linux/Makefile
At the top of this file, set "EXTRAVERSION = -om4" (without the quotation marks)
F10 to save & quit.
> cp /usr/src/ltsp_initrd_kit/config-2.4.19-ltsp-1 /usr/src/linux/.config
> make oldconfig (or menuconfig if you prefer)
Enable the openMosix options:
a) Yes to "openMosix process migration support"
b) NO to "Create the kernel with a "-openmosix" extension"
c) Yes to "openMosix File-System"
d) Defaults for remaining items.
> make dep
> make bzImage
> make modules
> make modules_install
> cp -R /lib/modules/2.4.18-om4 /opt/ltsp/i386/lib/modules/
> cd /usr/src/ltsp_initrd_kit
> mcedit buildk
Go to line number 255 (in the "Now, calculate some settings" section) which will read:
```
KERNEL_VER=${VERSION}.${PATCHLEVEL}.${SUBLEVEL}${EXTRAVERSION}
```

Edit this line to read:

KERNEL_VER=${VERSION}.${PATCHLEVEL}.${SUBLEVEL}-om4

F10 to save & exit
> ./buildk
buildk will ask you a few questions:
a) PCI/ISA network card: 1
b) Kernel source directory: /usr/src/linux
c) Size of intrd filesystem: 2000
d) Number of inodes to allocate in initrd filesystem: 200
Copy your new LTSP/openMosix client kernel to the proper location:
> cp vmlinuz-2.4.18-om4 /tftpboot/lts/
If you wish, create a symbolic link to the openMosix kernel for easier upgrading in the future (you won't have to edit the client filename line in dhcpd.conf later):
a) > cd /tftpboot/lts
b) > ln -s vmlinuz-2.4.18-om4 vmlinuz-openmosix
> cd /etc
> mcedit dhcpd.conf
Edit the filename "/lts/vmlinuz-2.4.18-ltsp" line using one of the two lines below depending on whether or not you created the symbolic link:
a) filename "/lts/vmlinuz-openmosix";
b) filename "/lts/vmlinuz-2.4.18-om4";

Add the following line (all on one line) in dhcpd.conf (above the trick from Peter comment):

option host-name = concat("ws", suffix( concat("00", binary-to-ascii( 10, 8, "", suffix(leased-address,1))),3));

F10 to save & exit
> service dhcpd restart
Copy & edit other files as indicated in Richard's How-To "Mosix Cluster with Diskless Nodes". Specifically, we are concerned with the following:
Create & save the mosix.map file:
> mcedit /etc/mosix.map
Enter the following text for mosix.map:
```
1 192.168.0.254 1
2 192.168.0.1 253
```
F10 to save & exit
> cp /etc/mosix.map /opt/ltsp/i386/etc/
(Note: exact copies of mosix.map should be maintained in both locations.)
Copy the Userspace-tools & other files for the LTSP clients:
> cp /sbin/setpe /opt/ltsp/i386/sbin/
> cp /bin/mosrun /opt/ltsp/i386/bin/
> cp /bin/mosmon /opt/ltsp/i386/bin/
> cp /bin/mosctl /opt/ltsp/i386/bin/
> cp /bin/migrate /opt/ltsp/i386/bin/
> cp /bin/touch /opt/ltsp/i386/bin/
> rm /opt/ltsp/i386/etc/hosts
> cp /etc/hosts /opt/ltsp/i386/etc/
> cp /etc/rc.d/init.d/openmosix /opt/ltsp/i386/etc/rc.openmosix
> mkdir /mfs
> mkdir /opt/ltsp/i386/mfs
> cd /etc/
> mcedit fstab
Add the following line to fstab:
```
none /mfs mfs dfsa=1 0 0
```
F10 to save & exit
> cd /opt/ltsp/i386/etc/
> mcedit fstab
Add the following line to fstab (yes, again! ;-):
```
none /mfs mfs dfsa=1 0 0
```
F10 to save & exit
> mcedit /opt/ltsp/i386/etc/rc.local

Add the following lines to the end of rc.local:

# openMosix startup section
echo 1 > /proc/hpc/admin/lstay
/etc/rc.openmosix start
mount /mfs
# End openMosix startup section

F10 to Save & exit
Once completed, and you verify that everything boots as it should, use the command > mosmon -t to check your work, hopefully you will see all of your openMosix nodes in action!

Some other quick notes once you have everything running properly and you're experienced with K12LTSP:

If you don't need a GUI on your LTSP/openMosix nodes, edit the lts.conf file to set the clients to start at "RUNLEVEL = 3".
Want to test the cluster? Try the openMosix stress test.

Well, that's it. Hopefully this How-To provides a complete step-by-step process for easily implementing openMosix in an LTSP client/server setting.

I would like to extend my thanks again to everyone involved in developing & supporting Open Source software. Particular thanks to the folks at the Linux Terminal Server Project, the K-12Linux Project, and the openMosix Project for making this concept a reality. Finally special thanks goes out again to Richard Camp (among other things, the dhcp.conf line is simply brilliant!), and the many individuals who have e-mailed me with questions, suggestions, and encouragement in this effort. Thank you!

Happy clustering!

James Jensen
jmsjnsnsatx@nojunkmailplease.yahoo.com

Revision 4.1
Saturday, October 26, 2002