ILRI Research Computing

WebLab is a multifunctional bioinformatics analysis platform integrating diversified tools with unified, user-friendly web interface. However, WebLab is not a mere bioinformatics toolbox, but also offer a powerful data management function, group strategy and knowledge sharing mechanism, which brings considerable advance of efficiency for both wet bench and in silico scientists.

WebLab is complicated to install and we received a disk image from the developers to try and install at ILRI-BecA.

Test system using virtual machine is running here: http://172.26.0.234:5000

The image is really big, it is located in /export2/weblab:

• [alan@hpc-ilri: ~]$ ls -lh /export2/weblab/
  total 324G
  -rw-r--r--  1 root wheel 299G Jun 27 19:58 weblab.img
  -rw-r--r--  1 root root   25G Aug 19 08:56 weblab.tar.gz

It is a raw dump of a hard drive, master boot record, partition table, and partitions… everything. We can use sfdisk to examine the partition table:

• [root@hpc-ilri weblab]# sfdisk -l -uS weblab.img
  last_lba(): I don't know how to handle files with mode 81a4
  Disk weblab.img: cannot get size
  Disk weblab.img: cannot get geometry
  
  Disk weblab.img: 0 cylinders, 0 heads, 0 sectors/track
  Warning: The partition table looks like it was made
    for C/H/S=*/255/63 (instead of 0/0/0).
  For this listing I'll assume that geometry.
  Units = sectors of 512 bytes, counting from 0
     
     Device Boot    Start       End   #sectors  Id  System
  weblab.img1   *        63   1975994    1975932  83  Linux
  weblab.img2       1975995  21527099   19551105  82  Linux swap
  weblab.img3      21527100 625137344  603610245  83  Linux
  weblab.img4             0         -          0   0  Empty

So there's three partitons, sfdisk said each sector is 512 bytes so we can determine their sizes in human-readable numbers:

• weblab.img1: 942 megs, ext2
• weblab.img2: 9.3 gigs, swap (no filesystem)
• weblab.img3: 287 gigs, ext3

We obviously have no need for the swap partition, as we can just create our own. We can extract partitions one and three and manipulate them. This includes mounting to copy files or using directly with a virtualization application.

Booting off the image's boot partition is hopeless because its partition table is from the physical disk (300 gigs), so when we yank the partition out this will cause problems. It's better to dump the partitions, mount them, and copy everything we need into a new image. After that we can add a boot loader and new partitions.

Dump the first partition without the master boot record (where the boot loader and partition table live). Tell dd to skip 63 sectors to where partition 1 begins:

• [root@hpc-ilri weblab]# dd if=weblab.img of=weblab_partition1.img skip=63 count=1975932
  1975932+0 records in
  1975932+0 records out

See? It's just an ext2 file system:

• [root@hpc-ilri weblab]# file weblab_partition1.img 
  weblab_partition1.img: Linux rev 1.0 ext2 filesystem data (large files)

GRUB, a Linux bootloader, typically has problems booting off exotic filesystems, so it's usually safer to put everything needed for booting (kernel!) on an ext2 partition.

Mount it as a loop device to see what's on it:

• [root@hpc-ilri weblab]# mount -o loop weblab_partition1.img /mnt/floppy/
  [root@hpc-ilri weblab]# ls /mnt/floppy/
  boot  gentoo  gentoo.igz  grub

Mount the root partition to copy data directly to or from its filesystem. Use an offset which points at the starting sector of the image (from sfdisk, 21527100 sectors * 512 bytes per sector = offset of 11021875200 bytes):

• [root@hpc-ilri weblab]# mount -o loop,offset=11021875200 weblab.img /mnt/floppy/

Check the status of the mount…

• [root@hpc-ilri weblab]# mount
  /dev/md2 on / type ext3 (rw)
  ...
  ...
  /export2/weblab/weblab.img on /mnt/floppy type ext3 (rw,loop=/dev/loop0,offset=11021875200)
  [root@hpc-ilri weblab]# df -h
  Filesystem                  Size  Used Avail Use% Mounted on
  /dev/md2                     62G   28G   31G  48% /
  ...
  ...
  /export2/weblab/weblab.img  284G   40G  230G  15% /mnt/floppy

The partition is ext3 and there are only 40 gigs in use!

Create somewhere to copy the data, for example a 50 gigabyte empty file read from /dev/zero:

• [root@hpc-ilri weblab]# dd if=/dev/zero of=/export3/weblab_root.img bs=1M count=51200
  51200+0 records in
  51200+0 records out

Format the "drive" as ext3:

• [root@hpc-ilri weblab]# mkfs.ext3 /export3/weblab_root.img

Mount the new drive (it doesn't matter where, just somewhere else than the / partition above):

• [root@hpc-ilri weblab]# mount -o loop /export3/weblab_root.img /media/floppy/

Verify the used space:

• [root@hpc-ilri weblab]# df -h
  Filesystem                  Size  Used Avail Use% Mounted on
  /dev/md2                     62G   28G   31G  48% /
  ...
  ...
  /dev/sdd1                   459G   22G  415G   5% /media/Lacie
  /export2/weblab/weblab.img  284G   40G  230G  15% /mnt/floppy
  /export3/weblab_root.img     50G   84M   47G   1% /media/floppy

The / partition (mounted here at /mnt/floppy) has 40 gigs used, and our new, empty partition has 47 free, so copy the data using cp. Make sure you don't mix up the source and target for the copy operation (use ls to verify before!):

• [root@hpc-ilri weblab]# cp -a /mnt/floppy/* /media/floppy/

Read the note about cp -a in cp's man page:

• -a, --archive
       same as -dpR

It's the easiest way to recursively copy a bunch of files from one place to another while preserving time/date stamps as well as owner/group info.

It's probably mostly empty space (remember, this is a raw dump of another hard disk), but if you want the raw partiton for duplicating to a physical disk or something, extract it to somewhere where there is a lot of space available:

• [root@hpc-ilri weblab]# dd if=weblab.img of=/media/Lacie/weblab_root_partition.img skip=21527100 count=603610245

The MBR needs 512 bytes at the beginning of the disk/image for the boot loader code (GRUB) and the partition table. First, create an empty 512 byte file:

• [root@manatee2 alan]# dd if=/dev/zero of=mbr.raw bs=512 count=1

Then prepend the image of the boot partition with the embty mbr.raw:

• [root@manatee2 alan]# cat mbr.raw weblab_boot.img > weblab_boot_with_mbr.img

We extracted the boot partition without its boot loader, so we need to copy it to a new, empty image and then add a boot loader (GRUB).

First, create an empty 100 meg image:

• [root@manatee2 alan]# dd if=/dev/zero of=weblab_boot_new.raw bs=1M count=100
  100+0 records in
  100+0 records out
  104857600 bytes (105 MB) copied, 0.653573 seconds, 160 MB/s

Attach a loop device to the image so we can address it as a block device:

• [root@manatee2 alan]# losetup -f weblab_boot_new.raw

Create a partition on the block device (new, primary, 1, with default sectors, sizes, etc) and write the partition in fdisk:

• [root@manatee2 alan]# fdisk -u /dev/loop0

Delete and recreate the loop device so the kernel knows about the new partition table:

• [root@manatee2 alan]# losetup -d /dev/loop0
  [root@manatee2 alan]# losetup -f weblab_boot_new.raw

Now create a file system on the image's new partition using parted (mkfs, Yes, 1, ext2, Ignore, quit):

• [root@manatee2 alan]# parted /dev/loop0

Then mount the image's new file system on a loop device with an offset of 32256 bytes (63 sectors * 512 bytes per sector) as a loop device:

• [root@manatee2 alan]# mount -o loop,offset=32256 weblab_boot_alan.raw /mnt/weblab_boot_new/

Mount the original boot image:

• [root@manatee2 alan]# mount -o loop weblab_boot.img /mnt/weblab_boot_old/

Copy everything over using cp's archive functionality (preserving time, date, permissions, owner, etc):

• [root@manatee2 alan]# cp -a /mnt/weblab_boot_old/* /mnt/weblab_boot_new/

Unmount both images:

• [root@manatee2 alan]# umount /mnt/weblab_boot_new/
  [root@manatee2 alan]# umount /mnt/weblab_old/

We'll install grub right on the image (not using a block device). Start grub like this so it doesn't try to read all our disks (to create a device map):

• [root@manatee2 alan]# grub --device-map=/dev/null

Type the following commands in grub:

• grub> device (hd0) weblab_boot_new.raw
  grub> root (hd0,0)
  grub> setup (hd0)
  grub> quit

If you don't have any errors you should be able to boot qemu.

Having extracted the boot and root partitions, we can test them in a virtual machine:

• [root@manatee2 alan]# qemu-system-x86_64 -hda weblab_boot_new.raw -hdb weblab_root.img -m 192 -vnc :0 -no-kqemu -redir tcp:5000:10.0.2.15:80 -redir tcp:5002:10.0.2.15:22

• Mount the root image and change /etc/fstab entries
• Mount the root image and copy a new root hash to /etc/shadow

Note to self: qemu on 64 bit host with a 64 bit guest does not like to run with the kqemu kernel module! Start qemu with -no-kqemu if you're seeing errors/crashes like this:

• RAX=00007f392e505de8 RBX=000000006ffffeff RCX=0000000001dd4100 RDX=00007f392e2f2000
  RSI=00007f392e505f18 RDI=00007f392e507000 RBP=00007fff0cb6b1a0 RSP=00007fff0cb6b000
  R8 =000000006fffffff R9 =0000000070000021 R10=000000006ffffdff R11=000000006ffffe34
  R12=0000000001dd40c0 R13=000000006fffff40 R14=00007fff0cb6b268 R15=0000000000000002
  RIP=00007f392eeb77a8 RFL=00010206 [-----P-] CPL=3 II=0 A20=1 SMM=0 HLT=0
  ES =0000 0000000000000000 00000000 00000000
  CS =0033 0000000000000000 ffffffff 00affb00
  SS =002b 0000000000000000 ffffffff 00cff300
  DS =0000 0000000000000000 00000000 00000000
  FS =0000 00007f392f0ba6f0 00000000 00000000
  GS =0000 0000000000000000 00000000 00000000
  LDT=0000 0000000000000000 00000000 00008200
  TR =0040 ffff88000100c080 00002087 00008900
  GDT=     ffff880001007000 0000007f
  IDT=     ffffffff81981000 00000fff
  CR0=8005003b CR2=00007f392e505df0 CR3=00000000089d5000 CR4=000006e0
  DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
  DR6=0000000000000000 DR7=0000000000000000
  Unsupported return value: 0xffffffff