ILRI Research Computing

WebLab is a multifunctional bioinformatics analysis platform integrating diversified tools with unified, user-friendly web interface. However, WebLab is not a mere bioinformatics toolbox, but also offer a powerful data management function, group strategy and knowledge sharing mechanism, which brings considerable advance of efficiency for both wet bench and in silico scientists.

WebLab is complicated to install and we received a disk image from the developers to try and install at ILRI-BecA.

The image is really big, it is located in /export2/weblab:

• [alan@hpc-ilri: ~]$ ls -lh /export2/weblab/
  total 324G
  -rw-r--r--  1 root wheel 299G Jun 27 19:58 weblab.img
  -rw-r--r--  1 root root   25G Aug 19 08:56 weblab.tar.gz

It is a raw dump of a hard drive, master boot record, partition table, and partitions… everything. We can use sfdisk to examine the partition table:

• [root@hpc-ilri weblab]# sfdisk -l -uS weblab.img
  last_lba(): I don't know how to handle files with mode 81a4
  Disk weblab.img: cannot get size
  Disk weblab.img: cannot get geometry
  
  Disk weblab.img: 0 cylinders, 0 heads, 0 sectors/track
  Warning: The partition table looks like it was made
    for C/H/S=*/255/63 (instead of 0/0/0).
  For this listing I'll assume that geometry.
  Units = sectors of 512 bytes, counting from 0
     
     Device Boot    Start       End   #sectors  Id  System
  weblab.img1   *        63   1975994    1975932  83  Linux
  weblab.img2       1975995  21527099   19551105  82  Linux swap
  weblab.img3      21527100 625137344  603610245  83  Linux
  weblab.img4             0         -          0   0  Empty

So there's three partitons, sfdisk said each sector is 512 bytes so we can determine their sizes in human-readable numbers:

• weblab.img1: 942 megs, ext2
• weblab.img2: 9.3 gigs, swap (no filesystem)
• weblab.img3: 287 gigs, ext3

We obviously have no need for the swap partition, as we can just create our own. We can extract partitions one and two and manipulate them. This includes mounting to copy files or using directly with a virtualization application.

The MBR needs 512 bytes at the beginning of the disk/image for the boot loader code (GRUB) and the partition table. First, create an empty 512 byte file:

• [root@manatee2 alan]# dd if=/dev/zero of=mbr.raw bs=512 count=1

If you want to boot from the image extract it using this command:

• [root@hpc-ilri weblab]# dd if=weblab.img of=weblab_partition1.img count=1975995

You can see Linux recognizes the that file contains a boot sector:

• [root@hpc-ilri weblab]# file weblab_partition1.img
  weblab_partition1.img: x86 boot sector, code offset 0x48

This is a raw image which can be booted in a virtualization application like qemu, example:

• [root@hpc-ilri weblab]# qemu-system-x86_64 -hda weblab_partition1.img -vnc :0 -m 128

After which point you can watch the machine's boot process with a VNC viewer on port 5900.

If you want to read or copy files off the first partition then you need to extract it without the boot sector. Tell dd to skip 63 sectors to where partition 1 begins:

• [root@hpc-ilri weblab]# dd if=weblab.img of=weblab_partition1.img skip=63 count=1975932
  1975932+0 records in
  1975932+0 records out

See? It's just an ext2 file system:

• [root@hpc-ilri weblab]# file weblab_partition1.img 
  weblab_partition1.img: Linux rev 1.0 ext2 filesystem data (large files)

GRUB, a Linux bootloader, typically has problems booting off exotic filesystems, so it's usually safer to put everything needed for booting (kernel!) on an ext2 partition.

Mount it as a loop device to see what's on it:

• [root@hpc-ilri weblab]# mount -o loop weblab_partition1.img /mnt/floppy/
  [root@hpc-ilri weblab]# ls /mnt/floppy/
  boot  gentoo  gentoo.igz  grub

Mount the root partition to copy data directly to or from its filesystem. Use an offset which points at the starting sector of the image (from sfdisk, 21527100 sectors * 512 bytes per sector = offset of 11021875200 bytes):

• [root@hpc-ilri weblab]# mount -o loop,offset=11021875200 weblab.img /mnt/floppy/

Check the status of the mount…

• [root@hpc-ilri weblab]# mount
  /dev/md2 on / type ext3 (rw)
  ...
  ...
  /export2/weblab/weblab.img on /mnt/floppy type ext3 (rw,loop=/dev/loop0,offset=11021875200)
  [root@hpc-ilri weblab]# df -h
  Filesystem                  Size  Used Avail Use% Mounted on
  /dev/md2                     62G   28G   31G  48% /
  ...
  ...
  /export2/weblab/weblab.img  284G   40G  230G  15% /mnt/floppy

The partition is ext3 and there are only 40 gigs in use!

Create somewhere to copy the data, for example a 50 gigabyte empty file read from /dev/zero:

• [root@hpc-ilri weblab]# dd if=/dev/zero of=/export3/weblab_root.img bs=1M count=51200
  51200+0 records in
  51200+0 records out

Format the "drive" as ext3:

• [root@hpc-ilri weblab]# mkfs.ext3 /export3/weblab_root.img

Mount the new drive (it doesn't matter where, just somewhere else than the / partition above):

• [root@hpc-ilri weblab]# mount -o loop /export3/weblab_root.img /media/floppy/

Verify the used space:

• [root@hpc-ilri weblab]# df -h
  Filesystem                  Size  Used Avail Use% Mounted on
  /dev/md2                     62G   28G   31G  48% /
  ...
  ...
  /dev/sdd1                   459G   22G  415G   5% /media/Lacie
  /export2/weblab/weblab.img  284G   40G  230G  15% /mnt/floppy
  /export3/weblab_root.img     50G   84M   47G   1% /media/floppy

The / partition (mounted here at /mnt/floppy) has 40 gigs used, and our new, empty partition has 47 free, so copy the data using cp. Make sure you don't mix up the source and target for the copy operation (use ls to verify before!):

• [root@hpc-ilri weblab]# cp -a /mnt/floppy/* /media/floppy/

Read the note about cp -a in cp's man page:

• -a, --archive
       same as -dpR

It's the easiest way to recursively copy a bunch of files from one place to another while preserving time/date stamps as well as owner/group info.

It's probably mostly empty space (remember, this is a raw dump of another hard disk), but if you want the raw partiton for duplicating to a physical disk or something, extract it to somewhere where there is a lot of space available:

• [root@hpc-ilri weblab]# dd if=weblab.img of=/media/Lacie/weblab_root_partition.img skip=21527100 count=603610245

Having extracted the boot and root partitions, we can test them in a virtual machine:

• [root@manatee2 alan]# qemu-system-x86_64 -kernel rescue64 -initrd initram.igz -append "root=/dev/sdb" -hda weblab_boot.img -hdb weblab_root.img -m 512 -vnc :0 -no-kqemu -redir tcp:5000:10.0.2.15:80 -redir tcp:5001:10.0.2.15:443

• Mount the root image and change /etc/fstab entries
• Mount the root image and copy a new root hash to /etc/shadow

Note to self: qemu on 64 bit host with a 64 bit guest does not like to run with the kqemu kernel module! Start qemu with -no-kqemu if you're seeing errors/crashes like this:

• RAX=00007f392e505de8 RBX=000000006ffffeff RCX=0000000001dd4100 RDX=00007f392e2f2000
  RSI=00007f392e505f18 RDI=00007f392e507000 RBP=00007fff0cb6b1a0 RSP=00007fff0cb6b000
  R8 =000000006fffffff R9 =0000000070000021 R10=000000006ffffdff R11=000000006ffffe34
  R12=0000000001dd40c0 R13=000000006fffff40 R14=00007fff0cb6b268 R15=0000000000000002
  RIP=00007f392eeb77a8 RFL=00010206 [-----P-] CPL=3 II=0 A20=1 SMM=0 HLT=0
  ES =0000 0000000000000000 00000000 00000000
  CS =0033 0000000000000000 ffffffff 00affb00
  SS =002b 0000000000000000 ffffffff 00cff300
  DS =0000 0000000000000000 00000000 00000000
  FS =0000 00007f392f0ba6f0 00000000 00000000
  GS =0000 0000000000000000 00000000 00000000
  LDT=0000 0000000000000000 00000000 00008200
  TR =0040 ffff88000100c080 00002087 00008900
  GDT=     ffff880001007000 0000007f
  IDT=     ffffffff81981000 00000fff
  CR0=8005003b CR2=00007f392e505df0 CR3=00000000089d5000 CR4=000006e0
  DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
  DR6=0000000000000000 DR7=0000000000000000
  Unsupported return value: 0xffffffff