Caveat: I have much more Linux experience than I had when I wrote this HOWTO. If I had to do it again, I'd probably use Gentoo instead of Debian, because I find it easier to write custom packages for Gentoo. Anyhow, here's my original HOWTO, preserved for posterity.


This HOWTO details the procedure I used to set up the abax cluster for NFS-rooted network booting. The system is useful in this case because it centralizes the installation in the head node (server), which makes maintaining, upgrading, or altering the computational nodes (clients) easier.

This procedure follows mainly Tim Brom's Microwulf configuration notes with two major differences.

  • Microwulf uses Ubuntu (gutsy?), and I'm using Debian etch.
  • Microwulf has a seperate partition for each client's root, populated with an independent installation from CD. I'm using a single partition for all of my clients, with the base system created using debootstrap (no CD).

For guidance in my deviations, I'm indebted to Bart Trojanowski's pxeboot and nfsroot notes and Falko Timme's notes on kernel compilation in Debian Etch.

Physical setup

Our cluster has one server with eight clients. The server has two network cards, eth0 and eth1. eth1 is connected to the outsize world (WAN). All of the clients have one network card, eth0. All of the eth0s are connected together through a gigabit switch (LAN).


Throughout this HOWTO, I will use # as the prompt for root, $ as the prompt for an unpriveledged user, and chroot# as the prompt for a root in a chrooted environment. File contents will be listed with the full path in the text introducing the listing. For example, path/to/file:

Contents of file

All files are complete with the exception of lines containing , in which case the meaning of the example should be clear from the context.

Basic server setup

Installing the OS

Boot the server with the Debian installation kernel following one of the options in the Debian installation guide. I netbooted my server from one of the client nodes following this procedure to set up the DHCP and TFTP servers on the client and untarring netboot.tar.gz in my tftpboot directory. After netbooting from a client, don't forget to take that client down so you won't have DHCP conflicts once you set up a DHCP server on your server.

Install Debian in whatever manner seems most appropriate to you. I partitioned my 160 GB drive manually according to

Mount pointTypeSize
`/` ext3280 MB
`/usr` ext320 GB
`/var` ext320 GB
`/swap` swap1 GB
`/tmp` ext35 GB
`/diskless`ext320 GB
`/home` ext393.7 GB

I went with a highly partitioned drive to ease mounting, since I will be sharing some partitions with my clients. To understand why partitioning is useful, see the Partition HOWTO.

You can install whichever packages you like, but I went with just the standard set (no Desktop, Server, etc.). You can adjust your installation later with any of (not an exhaustive list)

  • tasksel, command line, coarse-grained package control.
  • apt-get, command line, fine-grained package control.
  • aptitude, curses frontend for apt-get.
  • synaptic, gtk+ frontend for apt-get.
  • dpkg, command line, package-management without dependency checking.

The base install is pretty bare, but I don't need a full blown desktop, so I flesh out my system with:

# apt-get install xserver-xorg fluxbox fluxconf iceweasel xterm xpdf
# apt-get install build-essentials emacs21-nox

which gives me a bare-bones graphical system (fire it up with startx) and a bunch of critical build tools (make, gcc, etc.).

Configuring networking

We need to set up our server so that eth1 assumes it's appropriate static IP on the WAN, and eth0 assumes it's appropriate static IP on the LAN. We achieve this by changing the default /etc/network/interfaces to

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

allow-hotplug eth0           # start on boot & when plugged in
iface eth0 inet static       # static LAN interface

allow-hotplug eth1           # start on boot & when plugged in
#iface eth1 inet dhcp        # WAN DHCP interface (not used)
iface eth1 inet static       # WAN static interface
    address XXX.XXX.YYY.YYY
    broadcast XXX.XXX.127.255
    gateway XXX.XXX.ZZZ.ZZZ

where I've censored our external IPs for privacy. The netmask selects which addresses belong to which networks. The way we've set it up, all messages will be routed out eth0, and everything else will go through eth1 to it's gateway. See the Net-HOWTO for more details.

Remote booting

Server services

The clients will boot remotely using the Pre eXecution Environment (PXE). The boot procedure is

  1. Client powers on.
  2. Client BIOS comes up, detects attached devices, and looks for a DHCP server for advice on network booting.
  3. DHCP server gives client an IP address, domain name, host name, the IP address of the TFTP server, and the location of the bootloader on the TFTP server.
  4. Client gets bootloader from TFTP server.
  5. BIOS hands over control to bootloader.
  6. Bootloader gets kernel and initial ramdisk from TFTP server.
  7. Bootloader hands over control to kernel
  8. Kernel starts up the system, mounting root via NFS.
  9. … after this point, it's just like a normal boot process.

We can see that we need to set up DHCP, TFTP, and NFS servers (not necessarily on the same server, but they are in our case).


The pxe bootloader can be obtained with

# apt-get install syslinux

which installs it to /usr/lib/syslinux/pxelinux.0 along with a manual and some other syslinux tools.


Install a server with

# apt-get install dhcp

Configure the server with /etc/dhcpd.conf

allow bootp;  # maybe?
allow booting;# maybe?
option domain-name "";
option domain-name-servers XXX.XXX.XXX.XXX,YYY.YYY.YYY.YYY;

subnet netmask {
    range; # non-static IP range
    option broadcast-address;
    option routers; # Gateway server
    next-server;    # TFTP server
    filename "pxelinux.0";        # bootloader

    host n1 {
        hardware ethernet ZZ:ZZ:ZZ:ZZ:ZZ:ZZ;
        option root-path "";
        option host-name "n1";
    … more hosts for other client nodes …

This assigns the client a static hostname, domain name, and IP address according to it's ethernet address (aka MAC address). It also tells all the clients to ask the TFTP server on for the bootloader pxelinux.0. For extra fun, it tells the clients to send packets to the router at if they can't figure out where they should go, and to use particular DNS servers to resolve domain names to IP addresses. This gives them access to the outside WAN. I don't know yet if the booting options are necessary, since I don't know what they do.

We also need to ensure that the DHCP server only binds to eth0, since starting a DHCP server on your WAN will make you unpopular with your ISP. You should have the following /etc/default/dhcp:


Once the DHCP server is configured, you can start it with

# /etc/init.d/dhcp restart

Check that the server is actually up with

# ps -e | grep dhcp

and if it is not, look for error messages in

# grep -i dhcp /var/log/syslog


There are several TFTP server packages. We use atftpd here, but tftp-hpa is also popular. Install atftpd with

# apt-get install atftpd xinetd

where xinetd is a super-server (replacing inetd, see man xinetd for details). Configure atftpd with /etc/xinetd.d/atftpd

service tftp
    disable     = no
    socket_type = dgram
    protocol    = udp
    wait        = yes
    user        = nobody
    server      = /usr/sbin/in.tftpd
    server_args = --tftpd-timeout 300 --retry-timeout 5 --bind-address --mcast-port 1758 --mcast-addr --mcast-ttl 1 --maxthread 100 --logfile /var/log/atftpd.log --verbose=10 /diskless/tftpboot

Note that the server_args should all be on a single, long line, since I haven't been able to discover if xinetd recognizes escaped endlines yet. This configuration tells xinetd to provide TFTP services by running in.tftpd (the daemon form of atftpd) as user nobody. Most of the options we pass to in.tftpd involve multicasting, which I believe is only used for MTFTP (which pxelinux.0 doesn't use). --logfile /var/log/atftpd.log --verbose=10 logs lots of detail to /var/log/atftpd.log if it exists. You can create it with

# touch /var/log/atftpd.log
# chown nobody.nogroup /var/log/atftpd.log

The most important argument is /diskless/tftpboot, which specifies the root of the TFTP-served filesystem (feel free to pick another location if you would like). This is where we'll put all the files that the TFTP will be serving. It needs to be read/writable by nobody, so create it with

# mkdir \tftpboot
# chmod 777 tftpboot

(TODO: possibly set the sticky bit, remove writable?)

Finally, we need to restart the xinetd server so it notices the new atftpd server.

# /etc/init.d/xinetd restart

Check that the xinetd server is up with

# ps -e | grep xinetd

and look for error messages with

# grep -i dhcp /var/log/syslog

Just having xinetd up cleanly doesn't prove that atftpd is working though, it just shows that the atftpd configuration file wasn't too bungled. To actually test atftpd we need to wait until the Synthesis Section when we actually have files to test-transfer.


Install the NFS utilities on the server with

# apt-get install nfs-common nfs-kernel-server

We go with the kernel server because we want fast NFS, since we'll be doing a lot of it. Set the NFS server up to export the root file systems and the user's home directories with /etc/exports:

… other node root exports …
/diskless,no_root_squash,sync,no_subtree_check) # unnecessary
/home ,no_root_squash,sync,no_subtree_check)
/usr  ,no_root_squash,sync,no_subtree_check)

Then let the NFS server know we've changed the exports file with

# exportfs -av  # TODO: -r?

Test that the NFS server is working properly by sshing onto one of the clients and running

client# mkdir /mnt/n1
client# mount /mnt/n1
client# ls /mnt/n1
… some resonable contents …
client# umount /mnt/n1
client# rmdir /mnt/n1

Client setup

The only client setup that actually happens on the client is changing the BIOS boot order to preferentially boot from the network. Consult your motherboard manual for how to accomplish this. It should be simple once you get into the BIOS menu, which you generally do by pressing del, F2, F12, or some such early in your boot process. Everything else happens on the server.

Root file system

We want to install a basic Debian setup on our clients. Since each client doesn't have it's own, private partition, we need to install Debian using debootstrap.

# apt-get install debootstrap
# mkdir /diskless/n1
# debootstrap --verbose --resolve-deps etch /diskless/n1
# chroot /diskless/n1
chroot# tasksel install standard
chroot# dpkg-reconfigure locales
chroot# apt-get install kernel-image-2.6-686 openssh-server nfs-client

TODO: what get's installed with standard? See /usr/share/tasksel/debian-tasks.desc for a list of possible tasks and the debian docs for details on how a full installation from CD or netboot.

We can also add a few utilities so we can work in our chrooted environment

chroot# apt-get install emacs21-nox

Configuring /etc

The client will be getting its hostnames from the DHCP server, so remove the default

# rm /diskless/n1/etc/hostname

We also need to setup the fstab to mount /home and /usr from the server. In /diskless/n1/etc/fstab:

# /etc/fstab: static file system information.
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# automatically mount nfs root and proc through other means /home   nfs defaults,nolock        0       0 /usr     nfs defaults,nolock        0       0
# we're diskess so we don't need to mount the hard disk sda :)
#/dev/sda1       /               ext3    defaults,errors=remount-ro 0       1
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
/dev/fd0        /media/floppy0  auto    rw,user,noauto  0       0

Kernel and initial ramdisk

The kernel version number shows up often in this section. You can determine your kernel version number (in my case 2.6.18-6-686) with uname -r. Because kernel versions change fairly frequently, I'll use KERNEL_VERSION to denote the kernel version string.

Your kernel must be compiled with NFS root support if it's going to have an NFS root. You can determine whether your kernel supports NFS roots with

# grep 'ROOT_NFS' /diskless/n1/boot/config-KERNEL-VERSION

I didn't have it in my default debian etch 2.6.18-6-686 kernel, so I had to recompile my kernel (see the Kernel Appendix and Falko's notes). My compiled kernel had a version string 2.6.18-custom.

Most kernels boot using an initial ramdisk (a compressed root filesytem that lives in RAM). This ramdisk contains the necessary programs and scripts for booting the kernel. We need to create a ramdisk that can handle an NFS root, so chroot into your client filesystem and install some tools

chroot# apt-get install initramfs-tools 

Configure future ramdisks for NFS mounting with /etc/initramfs-tools/initramfs.conf:

# Configuration file for mkinitramfs(8). See initramfs.conf(5).
BOOT=nfs    # was BOOT=local

Compile a new initrd with

chroot# update-initramfs -u

If you compiled your own kernel as in Kernel Appendix after setting up initramfs.conf, an appropriate ramdisk should have been created automatically.

You can examine the contents of your ramdisk with

$ cp /diskless/n1/boot/initrd.img-2.6.18-6-686 initrd.img.gz
$ gunzip initrd.img.gz
$ mkdir initrd
$ cd initrd/
$ cpio -i --make-directories < ../initrd.img 


To configure PXE, we need to bring pxelinux.0 into our new tftpboot directory

# cp /usr/lib/syslinux/pxelinux.0 /diskless/tftpboot/

We also need to bring in our kernel image and initial ramdisk

# cd /diskless/tftpboot
# ln -s /diskless/n1/boot/initrd.img-2.6.18-custom
# ln -s /diskless/n1/boot/vmlinuz-2.6.18-custom

atftpd handles the symbolic links, but if your TFTP server doesn't, you'll have to copy the image and ramdisk over instead.

At this point you should test your TFTP server with test transfers. Install the atftp client

# apt-get install atftp

And attempt to transfer the important files.

$ atftp
tftp> status
Connected: port 69
Mode:       octet
Verbose:    off
Trace:      off
 tsize:     disabled
 blksize:   disabled
 timeout:   disabled
 multicast: disabled
mtftp variables
 client-port:   76
 listen-delay:  2
 timeout-delay: 2
Last command: quit
tftp> get pxelinux.0
tftp> get initrd.img-2.6.18-custom
tftp> get vmlinuz-2.6.18-custom
tftp> quit
$ ls -l
-rw-r--r-- 1 sysadmin sysadmin 4297523 2008-05-30 09:27 initrd.img-2.6.18-custom
-rw-r--r-- 1 sysadmin sysadmin   13480 2008-05-30 09:26 pxelinux.0
-rw-r--r-- 1 sysadmin sysadmin 1423661 2008-05-30 09:27 vmlinuz-2.6.18-custom

If this doesn't work, look for errors in /var/log/syslog and /var/log/atftpd.log and double check your typing in the atftpd configuration file.

The last stage is to configure the pxelinux.0 bootloader. Create a configuration directory in tftboot with

# mkdir /diskless/tftpboot/pxelinux.cfg

When each client loads pxelinux.0 during the boot, they look for a configuration file in pxelinux.cfg. The loader runs through a sequence of possible config file names, as described in pxelinux.doc. We'll have different rood directories for each of our nodes, so we need a seperate config for each of them. In order to make our configs machine-specific, we'll use the ethernet (MAC) address file-name scheme. That is, for a machine with MAC address AA:BB:CC:DD:EE:FF, we make the file pxelinux.cgf/01-aa-bb-cc-dd-ee-ff. TODO: base config on IP address. In /diskless/tftpboot/pxelinux.cfg/01-aa-bb-cc-dd-ee-ff:

default linux

label linux
  kernel vmlinuz-2.6.18-custom
  append root=/dev/nfs initrd=initrd.img-2.6.18-custom
         nfsroot=,tcp ip=dhcp rw

Note that the appended args should all be on a single, long line, since I haven't been able to discover if pxelinux recognizes escaped endlines yet. This file is basically like a grub or lilo config file, and you can get fancy with a whole menu, but since this is a cluster and not a computer lab, we don't need to worry about that. Note that this file was only for our first node (n1). You have to make copies for each of your nodes, with the appropriate file names and nfsroots.

The kernel options are fairly self explanatory except for the tcp for the nfsroot option, which says the client should mount the root directory using TCP based NFS. Traditional NFS uses UDP, which is faster, but possibly less reliable for large files (like our kernel and initrd). However I'm having trouble tracking down a reliable source for this. For now, consider the tcp a voodoo incantation to be attempted if the NFS booting isn't working.

You're done! Plug a monitor into one of the clients and power her up. Everything should boot smoothly off the server, without touching the client's harddrive.

Adding clients

To add a new client node nX to the cluster, we need to do the following (which can be combined into an add-client script). First, we need to create a root directory for the new client

# cd /diskless/
# cp -rp n1 nX

Now we need to export that directory

# echo '/diskless/nX,no_root_squash,sync,no_subtree_check)' >> /etc/exports
# exportfs -av

Finally, we need to set up the booting and DHCP options

# cd /diskless/tftpboot
# sed 's/\/diskless\/n1/\/diskless\/nX/' 01-xx-xx-xx-xx-xx-xx > 01-yy-yy-yy-yy-yy-yy
# echo '    host n8 {
        hardware ethernet YY:YY:YY:YY:YY:YY;
        option root-path "";
        option host-name "nX";
    }' >> /etc/dhcpd.conf
# /etc/init.d/dhcp restart


Compiling a kernel

See Falko's notes for an excellent introduction, and the NFS-root mini-HOWTO for NSF root particulars.

First, grab a bunch of useful compilation tools

chroot# apt-get install wget bzip2 kernel-package
chroot# apt-get install libncurses5-dev fakeroot build-essential initramfs-tools

Some of these (e.g. wget) should already be installed, but apt-get will realize this, so don't worry about it. Configure initramfs for building NFS root-capable initial ramdisks by setting up /etc/initramfs-tools/initramfs.conf as explained in the Kernel Section. For NSF root, your kernel needs the following options:

    → Networking support (`NET [=y]`)
      → Networking options
        → TCP/IP networking (`INET [=y]`)
          → IP: kernel level autoconfiguration (`IP_PNP =y`)
`ROOT_NFS` (`NET && NFS_FS=y && IP_PNP`)
  File systems
    → Network File Systems

I also used the build-in NFS client instead of the module. Here is a diff of the original debian etch conf vs. mine:

$ diff /diskless/n1/boot/config-2.6.18-6-686 .config
< # Sun Feb 10 22:04:18 2008
> # Thu May 29 23:59:47 2008
< # CONFIG_IP_PNP is not set

Compile your shiny, new kernel with

chroot# make-kpkg clean
chroot# fakeroot make-kpkg --initrd --append-to-version=-custom kernel_image kernel_headers

The new kernel packages are in the src directory

chroot# cd /usr/src
chroot# ls -l

Install the packages with

chroot# dpkg -i linux-image-2.6.18-custom_2.6.18-custom-10.00.Custom_i386.deb


No network devices available


IP-Config: No network devices available.

messages during the boot (after the kernel is successfully loaded!). According to this post, the problem is due to a missing kernel driver.

So I figured out what card I had:

# lspci
03:03.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller
03:04.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller

The ethernet HOWTO claimed that the e1000 drivers were required for Intel gigabit cards, and indeed I had the e1000 module mounted on my server:

# lsmod | less
e1000                 108480  0 

I reconfigured my kernel with (old vs new):

diff .config_mod_e1000 .config
< # Linux kernel version: 2.6.18-custom
< # Fri May 30 00:13:47 2008
> # Linux kernel version: 2.6.18
> # Fri May 30 22:21:29 2008
< CONFIG_E1000=m
> CONFIG_E1000=y

After which I recompiled and reinstalled the kernel as in the Kernel Appendix.

Waiting for /usr/

On booting a client, I noticed a Waiting for /usr/: FAILED message just before entering runlevel 2. I attribute the error to a faulty boot order on the client not mounting it's fstab filesystems before trying to run something in /usr/. There don't seem to be any serious side effects though, since the wait times out, and by the time I can log in to the node, /usr/ is mounted as it should be.