The root partition of the official x86 OpenWrt image is not very big, about 50 MiB.  Many find it too small after installing a few add-on packages.  Here I will cover the steps to expand it.  The resultant image can be used in a live USB (see Easy Live USB for x86 OpenWRT) or copied to a hard disk.

Procedure Outline

1. Get an uncompressed disk image.
2. Pad image to desired size
3. Attach the image file to a loop device
4. Edit image partition table to enlarge the root partition
5. Resize the file system in root partition
6. Detach the image from the loop device.

All commands below are run in Bash.

Uncompress Image File

Use whichever method you like to download an image file from OpenWrt (http://downloads.openwrt.org) and uncompress it using gzip.  For example, these two commands download and uncompress the 10.03.1-rc6 disk image.

bash$ wget --quiet http://downloads.openwrt.org/backfire/10.03.1-rc6/x86_generic/openwrt-x86-generic-combined-ext2.img.gz
bash$ gunzip openwrt-x86-generic-combined-ext2.img.gz

Alternative, you can just copy an image file from a live USB flash drive.  This will save you the trouble of restoring custom configurations.

Pad Disk Image

The next step is to use “dd” to increase the size of the disk image.

bash$ dd if=/dev/zero bs=1M count=50 >> openwrt-x86-generic-combined-ext2.img

This command appends 50 MiB of zeros to the end of the disk image:  “if=/dev/zero” tells dd to copy data from /dev/zero; “bs=1M” sets the block size to 1 MiB (1024*1024 bytes); “count=50” tells dd to copy 50 blocks.

Attach to Loop Device

Note:  All commands from this point to the end need to be run by a user with root privilege.

These commands find an unused loop device and attach it to the image file.

bash$ loop_dev=`losetup -f`
bash$ echo $loop_dev
/dev/loop3
bash$ losetup $loop_dev openwrt-x86-generic-combined-ext2.img

The first command uses “losetup -f” to find an unused device and stores the result in the shell variable loop_dev.  The “echo” command shows the device found.  Finally “losetup” attaches the device to the disk image.

Edit Partition Table

To expand a disk partition, it needs to be deleted first.  A new, larger partition is then created to take its place.  This new partition must start from the same sector as the old to prevent loss of data.

fdisk is used to manipulate the disk partition table.

bash$ fdisk -u=sectors -c=dos $loop_dev

The -u option asks fdisk to list partitions in sectors.  The -c option tells fdisk to operate in DOS compatibility mode.  $loop_dev is the loop device attached to the image file.

To see the existing partitions, type “p” at the fdisk prompt.

Command (m for help): p

Disk /dev/loop3: 107 MB, 107437568 bytes
16 heads, 63 sectors/track, 208 cylinders, total 209839 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

      Device Boot      Start         End      Blocks   Id  System
/dev/loop3p1   *          63        9071        4504+  83  Linux
/dev/loop3p2            9135      107855       49360+  83  Linux

fdisk shows /dev/loop3 has 209839 sectors.  It also lists two partitions.  The first one, /dev/loop3p1, is a small boot partition.  The second, /dev/loop3p2, is the root partition.  The root partition starts from sector 9135.  Make a note of this number.

Now delete the root partition and create a new one that covers all available space.

Command (m for help): d
Partition number (1-4): 2

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4, default 2): 2
First sector (9072-209838, default 9072): 9135
Last sector, +sectors or +size{K,M,G} (9135-209838, default 209838): 209838

Command (m for help): w
The partition table has been altered!

The “d” command asks fdisk to delete a partition, and “2” selects the second partition for deletion.  The “n” command asks fdisk to create a new partition.  “p” specifies a primary partition, and “2” selects the second primary partition.  The first sector of this partition is sector 9135, same as the deleted partition.  Its last sector is sector 209838, the default choice.  This is also the last sector on /dev/loop3.  Finally, the “w” command writes the new partition table through /dev/loop3 to the disk image.

Resize Root File System

The following commands will expand the root file system to the size of the root partition.

bash$ kpartx -a $loop_dev
/dev/mapper/loop3p1: mknod for loop3p1 failed: File exists

The “kpartx -a” command creates device nodes for the partitions in the disk image.  The output of “kpartx –a” (“mknod for loop3p1 failed”) seems to be a bug in my system.  As far as I can tell, the creation and deletion of loop3p1 occur normally.

Another thing worth noting:  kpartx and fdisk use different naming conventions.  kpartx uses “/dev/mapper/device_name”, for example “/dev/mapper/loop3p1”.  fdisk uses “/dev/device_name”, such as “/dev/loop3p1”.  This is because kpartx works with the device mapper.

Now run “fsck” to check the file system before resizing it.  In fact, some file systems can’t be resized until they are checked.

bash$ fsck -f /dev/mapper/loop3p2
fsck from util-linux 2.19.1
e2fsck 1.41.14 (22-Dec-2010)
Filesystem did not have a UUID; generating one.

Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/mapper/loop3p2: ***** FILE SYSTEM WAS MODIFIED *****
/dev/mapper/loop3p2: 957/6000 files (0.2% non-contiguous), 8173/49152 blocks

The “-f” option forces a run even when the file system seems clean.
Finally, resize the root file system.

bash$ resize2fs /dev/mapper/loop3p2
resize2fs 1.41.14 (22-Dec-2010)
Resizing the filesystem on /dev/mapper/loop3p2 to 100352 (1k) blocks.
The filesystem on /dev/mapper/loop3p2 is now 100352 blocks long.

bash$ kpartx -d $loop_dev

After resizing, “kpartx -d” reverses the changes made by “kpartx -a”.

Detach From Loop Device

The final step is to detach the image file from the loop device.

bash$ losetup –d $loop_dev

That’s it.  The disk image is now ready to be used in a live USB or copied to a hard disk.

Update 12/24/2011:  initramfs.img for Backfire 10.03.1 final is now available.

Update 11/16/2011:  If you want to a larger root partition for your installation (The official one is about 50 MB), see Expanding x86 OpenWrt Root Partition

Update 11/15/2011:  initramfs.img for Backfire 10.03.1-rc6 is available for download on SkyDrive.

Update 09/05/2011:  If your OpenWRT PC’s BIOS does not support USB boot, see Booting From USB Without BIOS Support.

In this blog I will cover steps to create an x86 OpenWRT live USB on a Windows 7 PC.  The steps are fairly simple.  Just downloading files, move them around, and type a few commands.  No Linux experience necessary.  They can be easily adapted to Windows Vista or Windows XP if you have an older PC.

With a little more work, the technique discussed here can be adapted to create a bootable-x86-ISO/live-USB combination, for PCs that can’t boot from USB.  That is something in my pipeline.

And of course Live USB can be created on a Linux system, but I don’t plan to write up the instructions.  Leave a comment if you need help with that.

Finally, the theory of operation comes after the instructions.  I think this live USB concept is a big improvement over my livecd idea (see Creating OpenWrt x86 Live CD with USB Support).  I plan to keep up with newer OpenWRT releases.

Background

Occasionally I want to try different OpenWRT configurations while keeping the main installation intact.  Although I can already do that with my custom live CDs , they are still cumbersome and slow when switching back and forth.  An even better method is to boot from USB to a selection of firmware versions or configurations.

To create a bootable live USB using the steps discussed here, you’ll need

• An x86 PC that can boot from USB or CD.  If you PC can boot from CD but not USB, see Booting From USB Without BIOS Support.
• A Windows PC for configuring the USB flash drive.
• A FAT or FAT32 USB flash drive.  Each OpenWRT installation needs about 60 MiB of space.  You do not need a large drive to start.  Installation can be moved from one drive to another.
• Syslinux for creating bootable USB.  Syslinux is free.  Its website is http://www.syslinux.org.  The download page is http://www.kernel.org/pub/linux/utils/boot/syslinux/. Update 09/16/2011: kernel.org has been down for a while, taking syslinux with it. I searched for a Syslinux mirror and Google returned MuntInternet in the Netherlands.

OpenWRT Files

Three components make up one OpenWRT configuration:  a Linux kernel, an initial rootfs image, and an OpenWRT disk image.  All three are stored as files on the USB drive, and must have the same firmware version.  For example, to boot your PC to OpenWRT Backfire 10.03, you need the 10.03 kernel, 10.03 OpenWRT disk image, and 10.03 initramfs image.

When changes are made to an OpenWRT router, they are stored in the disk image.  The linux kernel and rootfs image are not changed.  Therefore some configurations can use the same Linux kernel and rootfs image, as long as they are of the same version.  Also, the disk images are initially compressed (.gz format).  They are decompressed when used for the first time.

For clarity I gather files of the same version in one folder, and use the version number as folder names.

Official OpenWRT website has the kernels and disk images for many different platforms and versions.  The rootfs images are modified disk images that I created, and are hosted on SkyDrive.  Here are the links to all files.

Table updated on 12/24/2011

Creating A Three-Configuration Live USB

Here I will cover steps to create a live USB with three configurations:

1. Config A:  Version 10.03.  It needs three files:  10.03 kernel, 10.03 rootfs image, and 10.03 disk image.
2. Config B:  Version 10.03.1-rc5.  It also needs three files:  10.03.1-rc5 kernel, 10.03.1-rc5 rootfs image, and 10.03.1-rc5 disk image
3. Config C:  Also version 10.03.1-rc5.  It needs just one file,  its own 10.03.1-rc5 disk image.  Config B and C will use the same kernel and rootfs image.

These 7 files are stored in two folders:

• 10.03:  Files of Config A.
  • openwrt-x86-vmlinuz:  Backfire 10.03 Linux kernel
  • openwrt-x86-ext2.image.gz:  Backfire 10.03 disk image
  • initramfs.img:  Backfire 10.03 rootfs image
• 10.03.1-rc5:  Files of Config B & C
  • openwrt-x86-generic-vmlinuz:  10.03.1-rc5 Linux kernel
  • config-b-openwrt-x86-generic-combined-ext2.img.gz:  10.03.1-rc5 disk image, for Config B.
  • config-c-openwrt-x86-generic-combined-ext2.img.gz:  10.03.1-rc5 disk image, for Config C.
  • initramfs.img:  10.03.1-rc5 rootfs image.

To create the folders and contents:

1. Insert the USB drive.  For this discussion, I’ll assume the drive letter is “E:”.
2. Create two folders on “E:”:  “10.03” and “10.03.1-rc5”.
3. Follow the links given above, and download version 10.03 kernel, disk image, and rootfs image.  Move these files to “E:\10.03”.
4. Again, follow the links and download kernel, disk image, and rootfs image of version 10.03.1-rc5.  Move these files to “E:\10.03.1-rc5”
5. Navigate to “E:10.03.1-rc5”.  Make a copy of the disk image.  Name the new copy “config-c-openwrt-x86-generic-combined-ext2.img.gz”.
6. Rename the original from “openwrt-x86-generic-combined-ext2.img.gz” to “config-b-openwrt-x86-generic-combined-ext2.img.gz”.

When you’re done, the USB drive and folders should look like these snapshots.

Making The USB Drive Bootable

We’ll use Syslinux as the bootloader.

1. Download Syslinux.  Follow the link given above and choose the ZIP archive because it is easier to work with on a Windows PC.
2. Extract the content of the archive to a new folder.  For this discussion, let’s assume that folder is “C:\syslinux-4.04”.
3. Start a command prompt as administrator.  Press the Windows Logo key and type “command prompt” in the search box.  Right click on “Command Prompt” under “Programs”.  Select “Run as administrator.”  Answer “Yes” in the UAC pop-up window.  An administrator command prompt should now be open.
4. Go to the executable directory and run Syslinux.  Again, assuming the USB drive’s letter is “E:”, type
   • 64-bit system
     cd c:\syslinux-4.04\win64
.\syslinux64.exe -m -a E:
   • 32-bit system
     cd c:\syslinux-4.04\win32
.\syslinux.exe -m -a E:

   “-m”  places Syslinux on the master boot record of “E:” drive.  “-a” marks the partition active.

5. Label the flash drive.  In the same command prompt, type
   label E: OPENWRT
This disk label (“OPENWRT” here) will identify the USB drive to Linux kernel.  You can choose a different label, but remember to use it in the Syslinux configuration file discussed later.  Also, remember that Windows uses only uppercase letters in the label but Linux is case sensitive.

Configuring Syslinux

The final step is to create a configuration file for Syslinux.  This file is named “syslinux.cfg” and stored at the root level of the USB drive.  That is, “E:\syslinux.cfg”.

In “syslinux.cfg” list all OpenWRT configurations.  There are four lines for each configuration:

1. “label”: The label of a configuration. Entered at boot prompt to make a selection.
2. “kernel”: The path to a kernel file on the USB drive.  Use “/” instead “\” and discard the drive letter.  For example, the kernel file of of Config A is
   E:\10.03\openwrt-x86-vmlinuz
   That makes the Syslinux entry
   /10.03/openwrt-x86-vmlinuz
3. “initrd”: Path to rootfs image.  Again, no drive letter, and “/” instead of “\”
4. “append”: Kernel command line options.
   • “rootvol”: USB drive label.  Add the prefix “LABEL=”, and remember to use all uppercase letters.  For our USB drive, add the following text to the “append” line:
     rootvol=LABEL=OPENWRT
     “rootvol” is my addition to OpenWRT.
   • “rootimage”: Path to disk image of a configuration.  Discard drive letter and use “/”.  For example, Config B uses the disk image
     E:\10.03.1-rc5\config-b-openwrt-x86-generic-combined-ext2.img.gz
     so add the following text to Config B’s “append” line
     rootimage=/10.03.1-rc5/config-b-openwrt-x86-generic-combined-ext2.img.gz
     “rootimage” is also my addition.
   • “console”: Specify the linux console. If you have a video card and a monitor, use “console=tty0”. If your OpenWRT PC is headless, use “console=ttyS0,38400n8” to enable the serial port.
   • “reboot”: original from OpenWRT

These entries can create a simple user interface for selecting a configuration.

• “say”: Display a message.
• “default”: The default choice
• “prompt”: 1 to turn on boot prompt.
• “timeout”: time to wait for user’s selection, in 1/10 of a second. (100 = 10 seconds)
• “ontimeout”: automatically selected choice on timeout.

The following configuration file will display the three choices discussed here.  It will then wait 10 seconds for a user to enter “a”, “b”, or “c”.  If a selection is made, Syslinux boots the chosen configuration.  Otherwise, it boots Config A.

say a) Config A (10.03)

label a
kernel /10.03/openwrt-x86-vmlinuz
initrd /10.03/initramfs.img
append rootvol=LABEL=OPENWRT rootimage=/10.03/openwrt-x86-ext2.image.gz console=tty0 reboot=bios

say b) Config B (10.03.1-rc5)

label b
kernel /10.03.1-rc5/openwrt-x86-generic-vmlinuz
initrd /10.03.1-rc5/initramfs.img
append rootvol=LABEL=OPENWRT rootimage=/10.03.1-rc5/config-b-openwrt-x86-generic-combined-ext2.img.gz console=tty0 reboot=bios

say c) Config C (10.03.1-rc5)

label c
kernel /10.03.1-rc5/openwrt-x86-generic-vmlinuz
initrd /10.03.1-rc5/initramfs.img
append rootvol=LABEL=OPENWRT rootimage=/10.03.1-rc5/config-c-openwrt-x86-generic-combined-ext2.img.gz console=tty0 reboot=bios

say make a selection and press enter
default a
prompt 1
timeout 100
ontimeout a

Live USB Ready

That’s it.  Just eject the drive from Windows and you have a Live USB drive with two OpenWRT configurations.  When booting from this drive, you’ll see a list of 3 choices and have 10 seconds to choose.

Always Shutting Down Gracefully

I noticed that the FAT/FAT32 file systems are quite susceptible to data corruption. During development I lost many files by pressing the reset button on my PC. Always shut down your system gracefully to avoid data loss.

Why Booting from USB Fails

A desktop Linux system typically boots up in these steps:

1. A bootloader, usually Grub, loads Linux kernel and initial rootfs image into memory and starts kernel execution.
2. From initial rootfs Kernel starts an initialization program, usually “/init”, to set up essential system components, including root file system.
   • If the necessary drivers are not built in the kernel, they are loaded from kernel modules.
   • These modules are included in the initial rootfs because it is custom built for each system (by “dracut” or “mkinitrd” usually).
3. Finally, the control is transferred to the “real” initialization program on the root file system.

The official x86 OpenWRT distribution differs from  this flow.  It skips the second step.  The OpenWRT kernel itself initializes the root file system and transfer controls to it.  This isn’t a problem when the root file system resides in an ext2 partition of a IDE or SATA hard disk.  The ext2 and ATA drivers are already in the kernel.  The USB drivers however are not, and this turns booting from USB into a Catch-22:  The kernel can’t access the root file system without the USB modules, but the USB modules aren’t available because they are on the root file system.

Choosing an Alternative

One solution to this problem is to add the USB drivers and re-compile the kernel.  It works but has two problems.  It is opaque because changes in the kernel binary can’t be easily examined.  It also requires the kernel on the official disk image to be replaced.  That task is not simple for a novice.

Another solution is to use a combination of an initial rootfs image (initramfs.img) and Syslinux.  This is a much better solution.  The initial rootfs image can be easily examined.  Setting up Syslinux is much simpler than replacing the kernel.  Additionally, Syslinux works with FAT, which is the default file system on most USB drives, and it can be set up from a Windows PC, thus broadening the user base.

Setting Up USB Root

As mentioned earlier, a critical role of initramfs.img is to set up the root file system.  In this case, the root file system is a image file on a USB drive.  To set up the USB root, kernel runs “/init” of initramfs.img.  “/init” is a shell script.  It sets up the root file system in these steps:

1. Load all modules in initramfs.img.  They include drives for USB and FAT file system.  (See “Creating initramfs.img” on adding these modules.)
2. Using a parameter on kernel command line, identify the USB drive that holds the root file system.  Mount that USB drive.
3. Again using a kernel command line parameter, find the root file system image file.  Decompress the file if necessary.
4. Check the size of the image file.  Pad it to ensure enough space is allocated for the root partition.
5. Associate the image file with a loop device, run fsck, and mount it.  Now the root file system is online.
6. Add a step in OpenWRT’s pre-init flow to link “/dev/root” to the loop device.
7. Unmount the USB drive.
8. Call “switch_root” to start using the root file system.

#!/bin/sh
#
# Copyright (C) 2011 William H. Liao  All rights reserved.
#
# This copyrighted material is made available to anyone wishing to use,
# modify, copy, or redistribute it subject to the terms and conditions
# of the GNU General Public License.
#

# Kernel command line arguments:

# rootvol:
#	specify the volume that holds the ext2 root image file.
#	Valid options are
#		/dev/sdxy (e.g. /dev/sda1)
#		UUID=xxxx (as printed by 'blkid', but no quotes)
#		LABLE=yyyy ('yyyy' is case sensitive.  no quotes)
#
# rootimage:
#	File name of ext2 root image file.  Leading "/" optional

# chop off .gz from root image name

rootimage=${rootimage%.gz}

echo "Looking for $rootimage(.gz) on device $rootvol"

# root image file size, and start of root fs in image
# file.  Both are discovered using fsck on OpenWRT ext2
# image

rootimage_size=$((107856*512))
rootimage_offset=$((9135*512))

# Load modules to support USB and FAT

. /etc/functions.sh

load_modules /etc/modules.d/*

echo "Waiting 15 seconds for removable devices to stablize"
i=0
while [ $i -lt 15 ]; do
	sleep 1
	echo -n "."
	i=$(($i+1))
done
echo

# findfs is not built in OpenWRT busybox by default, so
# examine blkid output to find out the "rootvol" device.

mount proc /proc -t proc

root_dev=""
case "$rootvol" in
	/dev/*)
		root_dev=$rootvol
		;;
	UUID=*)
		# work in lower case because hexdecimal values
		# are case insensitive

		grep_string=`echo $rootvol | cut -d= -f2 | tr '[A-Z]' '[a-z]'`
		root_dev=`blkid | tr '[A-Z]' '[a-z]' | \
			fgrep uuid=\"$grep_string\" | cut -d: -f1`
		;;
	LABEL=*)
		grep_string=`echo $rootvol | cut -d= -f2`
		root_dev=`blkid | fgrep LABEL=\"$grep_string\" | cut -d: -f1`
		;;
esac

[ -z $root_dev ] && \
	{ echo "Root device not found" ; exit ; }
[ -b $root_dev ] || \
	{ echo "$root_dev not a block device" ; exit ; }

# Mount the "rootvol" and look for "rootimage"

mount $root_dev /mnt > /dev/null 2>&1 || \
	{ echo "Root volume not mounted" ; exit; }

if [ -r /mnt/$rootimage.gz ]; then
	[ -f /mnt/$rootimage ] && rm /mnt/$rootimage
	gunzip -f /mnt/$rootimage.gz || { echo "Can't decompress $rootimage.gz" ; exit ; }
fi

[ -r /mnt/$rootimage ] || { echo "No root image $rootimage on volume" ; umount /mnt ; exit; }

echo "root image found on $root_dev"

# Make sure the image file is at least as big as its partition
# table says.  If it isn't, enlarge it, to make sure enough space
# is allocated.

image_size=`ls -l /mnt/$rootimage | awk '{print $5}'`
if [ $image_size -lt $rootimage_size ]; then
	dd if=/dev/zero of=/mnt/$rootimage bs=1 seek=$image_size \
		count=$(($rootimage_size-$image_size)) ||
			{ echo "Can't pad root image" ; umount /mnt ; exit ; }
	echo "root image padded"
fi

# Now connect "rootimage" to /dev/root (same as /dev/loop0).  Use an
# offset to skip MBR, grub, boot partition, etc.

losetup /dev/loop0 /mnt/$rootimage -o $rootimage_offset ||
	{ echo "Can't set up root image loopback" ; umount /mnt ; exit ; }

# check the root image before mouting it

e2fsck -p /dev/root
case $? in
	0|1)
		: ;;
	2)
		reboot ;;
	*)
		echo "root FS image has errors"
		losetup -d /dev/loop0
		umount /mnt
		exit
		;;
esac

mount /dev/root /root ||
	{ echo "Can't mount root image" ; losetup -d /dev/loop0 ; umount /mnt ; exit ; }

# Add a script to create /dev/root

if [ ! -r /root/lib/preinit/21_add_root_dev ]; then
	cat << EOF > /root/lib/preinit/21_add_root_dev
#!/bin/sh

add_dev_root() {
	mknod /dev/loop0 b 7 0
	ln -s loop0 /dev/root
}

boot_hook_add preinit_essential add_dev_root
EOF
fi

# Finally, ready to switch root

echo "root image mounted, ready to switch"

umount -l /mnt	# lazy umount, since we're still using the usb drive
umount /proc

exec switch_root /root /etc/preinit

# Uh-oh, try cleaning up

umount /root
losetup -d /dev/loop0

Creating initramfs.img

“initramfs.img” is based on the official disk image, with the addition of USB and FAT modules and removal of others.  I will describe just the general steps here because a detail discussion will be too long.

1. Write the official x86 disk image to the storage of a kernel virtual machine.
2. Start the KVM and configure its network interface so it can access the internet
3. Use opkg to add and remove modules.  The final list of module is

   base-files	kmod-nls-base	kmod-usb2	libuci
   busybox	kmod-nls-cp437	libblkid	libuuid
   e2fsprogs	kmod-nls-iso8859-1	libc	losetup
   hotplug2	kmod-usb-core	libext2fs	opkg
   kernel	kmod-usb-ohci	libgcc	udevtrigger
   kmod-fs-vfat	kmod-usb-storage	libpthread
   kmod-loop	kmod-usb-uhci	librt

4. Turn off the KVM.
5. Use “kpartx” to make the OpenWRT root partition available on the host system.
6. Use “scripts/gen_initramfs_list.sh” from a kernel build directory to create a file list.  This list includes basic device nodes, “/init” script shown above, and the root file system of the OpenWRT KVM.
7. Edit the list to change absolute paths to relative.  Also delete unnecessary files like “.svn” or “.git”.
8. Use “usr/gen_init_cpio” from a kernel build directory to generate the initial rootfs.  Compress it with gzip.

History

12/24/2011:  Add note about 10.03.1 final.

11/16/2011:  Add note about expanding root.

11/15/2011:  Add note about 10.03.1-rc6.  Strike kernel.org down.

09/16/2011: Note kernel.org is down. Give muntinternet syslinux mirror.
09/07/2011:  Add “Why Booting From USB Fails”, “Choosing An Alternative”, “Setting Up USB Root”, and “Creating initramfs.img”

09/05/2011:  Add note abut Plop Boot Manager.  Change PC requirement.

Update 09/04/2011: If your PC’s BIOS doesn’t have USB boot support, see Booting From USB Without BIOS Support followed by Easy Live USB for x86 OpenWRT.

Update 09/01/2011:  If your PC can boot from USB, perhaps a live USB setup will be more useful.  See Easy Live USB for x86 OpenWRT.

I decided to run OpenWrt on an old x86 PC as my broadband router, but the precompiled binaries do not work on this PC.  It does not have a hard drive or bootable USB.  I had to create a binary that boots from CD but saves configuration settings to a USB drive.  Here I will describe the changes I made to create this binary.

Source URLs

These URLs point to OpenWrt Backfire 10.03, release on April 7, 2010.

OpenWrt File Systems

When running on its “native” platforms (routers), OpenWrt uses three file systems

• A read-only SquashFS volume with core files for booting.
• A writable JFFS2 volume for storage.
• A overlay file system (mini_fo) that overlays the writable storage volume on top of the read-only boot volume, creating a writable root file system.

For the CD/USB-drive combination to work, three things are needed:

• A replacement for the SquashFS volume.
• Replacing the JFFS2 volume with a USB flash drive.
• Modifying the initialization scripts to accommodate replacements.

Replacing SquashFS Volume

When OpenWrt boots from a router’s flash memory, Linux kernel mounts the flash device with SquashFS image at “/” and executes “/etc/preinit” to start initialization.

When booting from a CD, the kernel does not mount a file system at “/”.  Instead, it extracts files from an initramfs archive to populate the special file system rootfs.  After extraction “/init” is run to initialize the system.  “/init” in turn runs “/etc/preinit”.

To create an environment similar to SquashFS-booting, I modified “/init” to create a device to be mounted at “/”:

1. “/init” first creates “/dev/loop0” and “/dev/root”, both point to the loopback device #0.
2. It creates an image file in rootfs, associates it with “/dev/root”, and format it as an ext2 file system.
3. It copies files from rootfs to this image,
4. If the image isn’t large enough, “/init” repeats steps 2 and 3 with a larger image.
5. After files are copied, “/init” runs “switch_root” to move the ext2 image to “/” and to start “/etc/preinit”.
6. If for some reason these steps can not be completed, the original OpenWrt “/init” script is run.

After these changes, “/etc/preinit” sees no differences between booting from CD or SquashFS, with the exception that “/dev/root” point to “/dev/loop0” when booting from CD, and “/dev/mtdblock0” when booting from SquashFS.

Patch for “/init”:

Index: target/linux/generic-2.6/base-files/init
===================================================================
--- target/linux/generic-2.6/base-files/init	(revision 20728)
+++ target/linux/generic-2.6/base-files/init	(working copy)
@@ -1,6 +1,57 @@
 #!/bin/sh
 # Copyright (C) 2006 OpenWrt.org

+# The system is booted up from initramfs.  If
+# /dev/loop0 is available, we'll switch to that
+# instead.  Using /dev/loop0 allows us to use
+# extroot overlay
+
+try_loop0() {
+
+  # initialize /dev/root
+  which losetup > /dev/null 2>&1 || return 1
+  insmod loop > /dev/null 2>&1 || return 1
+  mknod /dev/loop0 b 7 0 || return 1
+  mknod /dev/root b 7 0 || return 1
+
+  # make a ${root_size} /dev/root, mount it,
+  # and copy files to it.  If it is too small,
+  # try a larger size until there is some
+  # space left
+  root_done=0
+  root_size=4000
+  cd /
+  while [ $root_done -ne 2 ]; do
+
+    # clean up if /new_root already exists
+    if [ -e /new_root ]; then
+      { umount /root ; losetup -d /dev/root; } >/dev/null 2>&1
+      rm -rf /new_root || return 1
+    fi
+
+    # make ${root_size} ext2 volume
+    echo "Trying root image @ ${root_size}K"
+    dd if=/dev/zero of=/new_root bs=1024 count=$root_size > /dev/null 2>&1
+    losetup /dev/root /new_root
+    mkfs.ext2 /dev/root > /dev/null 2>&1
+    mount -t ext2 /dev/root /root
+
+    # copy files
+    root_size=$((root_size+500)) # prep for next loop
+    { find . -xdev -type d | { cd /root ; xargs mkdir -p; }; } || continue
+    mkdir -p /root/storage || continue
+    { tar cf - `find . -xdev \( \! -type d -a \! -name new_root \)` |
+      { cd /root ; tar xf -; }; } > /dev/null 2>&1 || continue
+    root_done=$(($root_done+1))
+  done
+
+  exec switch_root -c /dev/console /root /etc/preinit
+}
+
+# Try loop0 first.  If that doesn't work, continue
+# with initramfs
+try_loop0
+
 INITRAMFS=1

 . /etc/preinit

Replacing JFFS2 Volume

OpenWrt already has the block-extroot package that allows systems to use external devices for storage.  The only task for me was to create a setting that works with as many hardware configurations as possible.  For that I modified the sample mount entry in the default fstab configuration:

• Change mount target from “/home” to “/storage”.
• Use the label “rootfs_data” to locate the file system.  This ensures block-extroot can still find the USB drive when other storage devices are added.
• Remove file system type for additional flexibility.
• Enable fsck.
• Set “is_rootfs” option as required by block-extroot.
• Enable the modified entry.

Patch for “fstab.config”:

Index: package/block-mount/files/fstab.config
===================================================================
--- package/block-mount/files/fstab.config	(revision 20728)
+++ package/block-mount/files/fstab.config	(working copy)
@@ -7,12 +7,12 @@
 	option anon_swap 0

 config mount
-	option target	/home
-	option device	/dev/sda1
-	option fstype	ext3
+	option target	/storage
+	option label	rootfs_data
 	option options	rw,sync
-	option enabled	0
-	option enabled_fsck 0
+	option enabled	1
+	option enabled_fsck 1
+	option is_rootfs 1

 config swap
 	option device	/dev/sda2

Modifying Initialization Scripts

In Backfire 10.03, block-extroot is written to work on systems that have MTD devices and JFFS2 volumes.  If either does not exist, the initialization scripts do not create the overlay file system.  This is a problem for my x86 system since it has neither MTD nor JFFS2.

The solution is to run the block-extroot scripts before checking for MTD or JFFS2:

• A new hook “do_mount_extroot” is created in the script file “75_mount_extroot”.  The name “75_mount_extroot” ensures “do_mount_extroot” is run before “do_mount_root” from “80_mount_root”, which checks for MTD and JFFS2.
• Two block-extroot functions, “determine_external_root” and “external_root_pivot”, found in “50_determine_usb_root” and “60_pivot_usb_root” respectively, are moved from “do_mount_root” to the new hook “do_mount_extroot”.
• block-extroot Makefile is modified to install “75_mount_extroot”.

Patches for block-extroot:

Index: package/block-extroot/files/75_mount_extroot
===================================================================
--- package/block-extroot/files/75_mount_extroot	(revision 0)
+++ package/block-extroot/files/75_mount_extroot	(revision 0)
@@ -0,0 +1,10 @@
+
+# Copyright (C) 2011 William H Liao
+
+do_mount_extroot() {
+    boot_run_hook preinit_mount_extroot
+}
+
+boot_hook_add preinit_main do_mount_extroot
+
+
Index: package/block-extroot/files/60_pivot_usb_root
===================================================================
--- package/block-extroot/files/60_pivot_usb_root	(revision 20728)
+++ package/block-extroot/files/60_pivot_usb_root	(working copy)
@@ -16,5 +16,5 @@
 	}
 }

-boot_hook_add preinit_mount_root external_root_pivot
+boot_hook_add preinit_mount_extroot external_root_pivot

Index: package/block-extroot/files/50_determine_usb_root
===================================================================
--- package/block-extroot/files/50_determine_usb_root	(revision 20728)
+++ package/block-extroot/files/50_determine_usb_root	(working copy)
@@ -46,5 +46,5 @@
 	UCI_CONFIG_DIR="$OLD_UCI_CONFIG_DIR"
 }

-boot_hook_add preinit_mount_root determine_external_root
+boot_hook_add preinit_mount_extroot determine_external_root

Index: package/block-extroot/Makefile
===================================================================
--- package/block-extroot/Makefile	(revision 20728)
+++ package/block-extroot/Makefile	(working copy)
@@ -55,6 +55,7 @@
 	$(INSTALL_DIR) $(1)/lib/preinit
 	$(INSTALL_DATA) ./files/50_determine_usb_root $(1)/lib/preinit/
 	$(INSTALL_DATA) ./files/60_pivot_usb_root $(1)/lib/preinit/
+	$(INSTALL_DATA) ./files/75_mount_extroot $(1)/lib/preinit/
 	$(INSTALL_DIR) $(1)/lib/preinit
 	echo "extroot_settle_time=\"$(CONFIG_EXTROOT_SETTLETIME)\"" >$(1)/lib/preinit/00_extroot.conf
 	$(INSTALL_DIR) $(1)/overlay

Other Enhancements

Since fsck is enabled for “rootfs_data”, block-extroot will run fsck although it does not know the file system type.  To work around this  I modified “fsck.sh” to try all known types.

Patch for “fsck.sh”:

Index: package/block-mount/files/fsck.sh
===================================================================
--- package/block-mount/files/fsck.sh	(revision 20728)
+++ package/block-mount/files/fsck.sh	(working copy)
@@ -12,22 +12,25 @@
 	local found_fsck=0

-	[ -n "$fsck_type" ] && [ "$fsck_type" != "swap" ] && {
 		grep -q "$device" /proc/swaps || grep -q "$device" /proc/mounts || {
 			[ -e "$device" ] && [ "$fsck_enabled" -eq 1 ] && {
 				for known_type in $libmount_known_fsck; do
-					if [ "$known_type" = "$fsck_fstype" ]; then
-						fsck_${known_type} "$device"
-						found_fsck=1
-						break
-					fi
+					{ [ "$known_type" = "$fsck_fstype" ] || [ -z "$fsck_fstype" ] || [ "$fsck_fstype" = "auto" ]; } && {
+						[ "$known_type" = "$fsck_fstype" ] ||
+							echo "Trying fsck_$known_type on $device..."
+						fsck_${known_type} "$device" && {
+							found_fsck=1
+							break
+						}
+					}
 				done
 				if [ "$found_fsck" -ne 1 ]; then
+					{ [ -z "$fsck_fstype" ] || [ "$fsck_fstype" = "auto" ]; } &&
+						echo "Giving up fsck on $device"
 					logger -t 'fstab' "Unable to check/repair $device; no known fsck for filesystem type $fstype"
 				fi
 			}
 		}
-	}
 }

 libmount_known_fsck=""

This change unfortunately runs into bug in the e2fsprogs package:  The function “fsck_e2fsck” in “e2fsck.sh” does not correctly return a status code.  I have an ugly fix.

Patch for “e2fsck.sh”:

Index: package/e2fsprogs/files/e2fsck.sh
===================================================================
--- package/e2fsprogs/files/e2fsck.sh	(revision 20728)
+++ package/e2fsprogs/files/e2fsck.sh	(working copy)
@@ -5,17 +5,21 @@
 #

 fsck_e2fsck() {
-	e2fsck -p "$device" 2>&1 | logger -t "fstab: e2fsck ($device)"
+	# long and complicated pipe because ash does not
+	# support pipefail like bash does
+	{ e2fsck -p "$device" 2>&1; echo "#### status $?"; } |
+	awk "
+		/^#### status/ { exit \$NF }
+		{ print \$0 | \"logger -t \\\"fstab: e2fsck ($device)\\\"\" }
+	"
 	local status="$?"
 	case "$status" in
 		0|1) ;; #success
 		2) reboot;;
-		4) echo "e2fsck ($device): Warning! Uncorrected errors."| logger -t fstab
-			return 1
-			;;
+		4) echo "e2fsck ($device): Warning! Uncorrected errors."| logger -t fstab ;;
 		*) echo "e2fsck ($device): Error $status. Check not complete."| logger -t fstab;;
 	esac
-	return 0
+	return $status
 }

 fsck_ext2() {

Finally, a bug in a kernel module Makefile prevents block-extroot using an ext4 volume as the “rootfs_data” volume.  ext4 depends on crc16, but the Makefile does not mark crc16 as required by block-extroot.

Patch for “other.mk”:

Index: package/kernel/modules/other.mk
===================================================================
--- package/kernel/modules/other.mk	(revision 20728)
+++ package/kernel/modules/other.mk	(working copy)
@@ -64,7 +64,7 @@
   TITLE:=CRC16 support
   KCONFIG:=CONFIG_CRC16
   FILES:=$(LINUX_DIR)/lib/crc16.$(LINUX_KMOD_SUFFIX)
-  AUTOLOAD:=$(call AutoLoad,20,crc16)
+  AUTOLOAD:=$(call AutoLoad,20,crc16,1)
 endef

 define KernelPackage/crc16/description

Configuring OpenWrt

Configuration for storing data on ext2 formatted USB drives:

·         Target system

o    X86

·         Target image

o    ramdisk:  Y

o    iso:  Y

·         Base system:

o    block-hotplug:  Y

o    busybox

§  Configuration

·         Linux System Utilities

o    losetup:  Y

·         Kernel Modules

o    Block Devices

§  kmod-loop:  Y

o    Filesystems

§  kmod-fs-ext2:  Y

o    USB Support

§  kmod-usb-core:  Y

§  kmod-usb-storage:  Y

§  kmod-usb-uhci:  Y

§  kmod-usb2:  Y

·         Utilities

o    Filesystem

§  e2fsprogs:  Y

o    disc

§  block-extroot:  Y

Creating ISO Image and Labelling USB Drive

After applying patches and configuring, the normal “make” command will compile OpenWrt and create the ISO image “openwrt-x86-iso.fs” in “bin/x86”.

Use the -L option of tune2fs to label a USB drive.

Future Plan

• Backfire 10.03.1
• Adding command line arguments to override “rootfs_data”

Other Thoughts

Why a new file system is needed to replace initramfs/rootfs:

• Can’t use “switch_root” with the overlay file system.  “switch_root” will delete rootfs content before switching, and this will make the read-only portion of the overlay file system empty.
• “pivot_root” doesn’t work with rootfs, and block-extroot scripts use “pivot_root”.   rootfs is special.  It belongs to the kernel and not mounted.  “pivot_root” can only swap mounted file systems.

Why the ext2 image on rootfs isn’t deleted by “switch_root”:

• When “switch_root” unlinks that image (it does do that), the link to the image is removed.  However, since the image is still associated with a lookback device, it still has an open file descriptor.  This file descriptor prevents the space occupied by the image from being freed.

Why is the “e2fsck.sh” patch so ugly?

• Because BusyBox’s ash does not have a “pipefail” feature as Bash does, a pipe always returns the status of its last command.  But for this script we want the exit status of the fsck command, which is not last.  So the fsck exit status is sent through the pipe to be extracted later.

How does OpenWrt shell scripts access UCI configuration files?

• OpenWrt has a whole set of shell functions for that purpose.  Many (“config”, “config_load”, “config_get”, “config_set”, “config_foreach”) are in “/etc/functions.sh”.  They create shell variables to store configuration values.  The names of these variables match UCI option names.

Regarding “rdinit=” and “init=” command line arguments:

• “rdinit=” is for rootfs only.  “init=” is for the mounted root file system only.  “rdinit=” defaults to “/init”.  “init=” has no default.
• The function “kernel_init” in “init/main.c” first checks for “rdinit=” in rootfs.  If it is not found, “rdinit=” is set to NULL, and “prepare_namespace” in “init/do_mounts.c” is called to mount the root file system.
• “kernel_init” then calls “init_post” (still “init/main.c”) to run the init process.  “init_post” doesn’t know whether it is dealing with kernel’s rootfs or a mounted file system.  It just tries “rdinit=” first, then “init=”, and finally some hard-coded names.

How to pass arguments to a module?

• It seems adding “module_name.options” to kernel command line works.  See how OpenWrt uses block2mtd when booting x86 from ext2.

Change History

2011-09-04: Add note about Plop Boot Manager.
2011-09-01:  Add note about new live USB method.