Differences

This shows you the differences between two versions of the page.

--- weblab [2010/05/22 14:19] – external edit 127.0.0.1
+++ weblab [2018/04/30 10:12] (current) – Meh, remove this old crap. aorth
@@ Line 1: / Line 1: @@
-===== WebLab =====
-[[http://weblab.cbi.pku.edu.cn/|WebLab]] is a multifunctional bioinformatics analysis platform integrating diversified tools with unified, user-friendly web interface. However, WebLab is not a mere bioinformatics toolbox, but also offer a powerful data management function, group strategy and knowledge sharing mechanism, which brings considerable advance of efficiency for both wet bench and in silico scientists.
-WebLab is complicated to install and we received a disk image from the developers to try and install at ILRI-BecA.
-Test system using virtual machine is running here: http://172.26.0.234:5000
-The image is **really** big, it is located in ''/export2/weblab'':
-  * <code>$ ls -lh /export2/weblab/
-total 324G
--rw-r--r--  1 root wheel 299G Jun 27 19:58 weblab.img
--rw-r--r--  1 root root   25G Aug 19 08:56 weblab.tar.gz</code>
-===== Working with the image =====
-==== Find partitions ====
-It is a //raw// dump of a hard drive, master boot record, partition table, and partitions... everything.  We can use sfdisk to examine the partition table:
-  * <code># sfdisk -l -uS weblab.img
-last_lba(): I don't know how to handle files with mode 81a4
-Disk weblab.img: cannot get size
-Disk weblab.img: cannot get geometry
-Disk weblab.img: 0 cylinders, 0 heads, 0 sectors/track
-Warning: The partition table looks like it was made
-  for C/H/S=*/255/63 (instead of 0/0/0).
-For this listing I'll assume that geometry.
-Units = sectors of 512 bytes, counting from 0
-   Device Boot    Start       End   #sectors  Id  System
-weblab.img1   *        63   1975994    1975932  83  Linux
-weblab.img2       1975995  21527099   19551105  82  Linux swap
-weblab.img3      21527100 625137344  603610245  83  Linux
-weblab.img4             0         -          0   0  Empty</code>
-So there's three partitons, ''sfdisk'' said each sector is 512 bytes so we can determine their sizes in human-readable numbers:
-  * weblab.img1: 942 megs, ext2
-  * weblab.img2: 9.3 gigs, swap (no filesystem)
-  * weblab.img3: 287 gigs, ext3
-We obviously have no need for the swap partition, as we can just create our own.  We can extract partitions one and three and manipulate them.  This includes mounting to copy files or using directly with a virtualization application.
-Booting off the image's boot partition is hopeless because its partition table is from the physical disk (300 gigs), so when we yank the partition out this will cause problems.  It's better to dump the partitions, mount them, and copy everything we need into a new image.  After that we can add a boot loader and new partitions.
-Overview:
-  - Dump partition
-  - Create new, empty partition image
-  - Mount old and new images
-  - Copy data from old -> new image
-  - Edit system settings (/etc/fstab, GRUB config, etc) to allow system to boot
-  - Unmount images
-  - Install GRUB on new boot partition
-==== Creating /boot partition ====
-We extracted the boot partition without its boot loader, so we need to copy it to a new, empty image and then add a boot loader (GRUB).
-First, dump partition one from the weblab image, skip 63 sectors to where partition one begins:
-  * <code># dd if=weblab.img of=weblab_boot.img skip=63 count=1975932
-1975932+0 records in
-1975932+0 records out</code>
-Now, create an empty 100 meg image for our new boot partition:
-  * <code># dd if=/dev/zero of=weblab_boot_new.raw bs=1M count=100
-+0 records in
-+0 records out
-104857600 bytes (105 MB) copied, 0.653573 seconds, 160 MB/s</code>
-Attach a loop device to the image so we can address it as a block device:
-  * <code># losetup -f weblab_boot_new.raw</code>
-Create a partition on the block device (**n**ew, **p**rimary, **1**, with default sectors, sizes, etc) and **w**rite the partition in ''fdisk'':
-  * <code># fdisk -u /dev/loop0</code>
-Delete and recreate the loop device so the kernel knows about the new partition table:
-  * <code># losetup -d /dev/loop0
-# losetup -f weblab_boot_new.raw</code>
-Now create a file system on the image's new partition using ''parted'' (mkfs, Yes, 1, ext2, Ignore, quit):
-  * <code># parted /dev/loop0</code>
-Then mount the image's new file system on a loop device with an offset of 32256 bytes (63 sectors * 512 bytes per sector) as a loop device:
-  * <code># mkdir /mnt/weblab_boot_new
-# mount -o loop,offset=32256 weblab_boot_alan.raw /mnt/weblab_boot_new/</code>
-Mount the original boot image:
-  * <code># mkdir /mnt/weblab_boot_old
-# mount -o loop weblab_boot.img /mnt/weblab_boot_old/</code>
-Copy everything over using cp's archive functionality (preserving time, date, permissions, owner, etc):
-  * <code># cp -a /mnt/weblab_boot_old/* /mnt/weblab_boot_new/</code>
-Unmount both images:
-  * <code># umount /mnt/weblab_boot_new/
-# umount /mnt/weblab_old/</code>
-We'll install grub right on the image (not using a block device).  Start ''grub'' like this so it doesn't try to read all our disks (to create a device map):
-  * <code># grub --device-map=/dev/null</code>
-Type the following commands in grub:
-   * <code>grub> device (hd0) weblab_boot_new.raw
-grub> root (hd0,0)
-grub> setup (hd0)
-grub> quit</code>
-If you don't have any errors you should be able to follow a simialar procedure to create a root partition.
-==== Creating / partition ====
-Because the root partition is so big it's a pain in the ass to dump, so we'll just mount it directly.  Use an offset which points at the starting sector of the image (from ''sfdisk'', 21527100 sectors * 512 bytes per sector = offset of 11021875200 bytes):
-  * <code># mount -o loop,offset=11021875200 weblab.img /mnt/floppy/</code>
-Check the status of the mount...
-  * <code># mount
-/dev/md2 on / type ext3 (rw)
-...
-...
-/export2/weblab/weblab.img on /mnt/floppy type ext3 (rw,loop=/dev/loop0,offset=11021875200)
-# df -h
-Filesystem                  Size  Used Avail Use% Mounted on
-/dev/md2                     62G   28G   31G  48% /
-...
-...
-/export2/weblab/weblab.img  284G   40G  230G  15% /mnt/floppy</code>
-The partition is ext3 and there are only 40 gigs in use!
-Create somewhere to copy the data, for example a 50 gigabyte empty file read from /dev/zero:
-  * <code># dd if=/dev/zero of=/export3/weblab_root.img bs=1M count=51200
-+0 records in
-+0 records out</code>
-Format the "drive" as ext3:
-  * <code># mkfs.ext3 /export3/weblab_root.img</code>
-Mount the new drive (it doesn't matter where, just somewhere else than the / partition above):
-  * <code># mount -o loop /export3/weblab_root.img /media/floppy/</code>
-Verify the used space:
-  * <code># df -h
-Filesystem                  Size  Used Avail Use% Mounted on
-/dev/md2                     62G   28G   31G  48% /
-...
-...
-/export2/weblab/weblab.img  284G   40G  230G  15% /mnt/floppy
-/export3/weblab_root.img     50G   84M   47G   1% /media/floppy</code>
-The / partition (mounted here at /mnt/floppy) has 40 gigs used, and our new, empty partition has 47 free, so copy the data using ''cp''.  Make sure you don't mix up the source and target for the copy operation (use ''ls'' to verify before!):
-  * <code># cp -a /mnt/floppy/* /media/floppy/</code>
-Unmount images:
-Read the note about ''cp -a'' in cp's man page:
-  * <code>-a, --archive
-     same as -dpR</code>
-It's the easiest way to recursively copy a bunch of files from one place to another while preserving time/date stamps as well as owner/group info.
-===== Booting qemu =====
-Having extracted the boot and root partitions, we can test them in a virtual machine:
-  * <code>qemu-system-x86_64 -hda weblab_boot.raw -hdb weblab_root.img -m 1024 -vnc :0 -no-kqemu -redir tcp:5000:10.0.2.15:80 -redir tcp:5001:10.0.2.15:443 -redir tcp:5002:10.0.2.15:22 -smp 2</code>
-  * Mount the root image and change ''/etc/fstab'' entries to point to the correct drives
-  * Mount the root image and copy a new root hash to ''/etc/shadow''
-  * Mount the boot image to change grub's ''menu.lst'' to reflect the new root device
-===== Troubleshooting =====
-==== qemu crashes during boot ====
-Note to self: qemu on 64 bit host with a 64 bit guest does not like to run with the ''kqemu'' kernel module!  Start qemu with ''-no-kqemu'' if you're seeing errors/crashes like this:
-  * <file>RAX=00007f392e505de8 RBX=000000006ffffeff RCX=0000000001dd4100 RDX=00007f392e2f2000
-RSI=00007f392e505f18 RDI=00007f392e507000 RBP=00007fff0cb6b1a0 RSP=00007fff0cb6b000
-R8 =000000006fffffff R9 =0000000070000021 R10=000000006ffffdff R11=000000006ffffe34
-R12=0000000001dd40c0 R13=000000006fffff40 R14=00007fff0cb6b268 R15=0000000000000002
-RIP=00007f392eeb77a8 RFL=00010206 [-----P-] CPL=3 II=0 A20=1 SMM=0 HLT=0
-ES =0000 0000000000000000 00000000 00000000
-CS =0033 0000000000000000 ffffffff 00affb00
-SS =002b 0000000000000000 ffffffff 00cff300
-DS =0000 0000000000000000 00000000 00000000
-FS =0000 00007f392f0ba6f0 00000000 00000000
-GS =0000 0000000000000000 00000000 00000000
-LDT=0000 0000000000000000 00000000 00008200
-TR =0040 ffff88000100c080 00002087 00008900
-GDT=     ffff880001007000 0000007f
-IDT=     ffffffff81981000 00000fff
-CR0=8005003b CR2=00007f392e505df0 CR3=00000000089d5000 CR4=000006e0
-DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
-DR6=0000000000000000 DR7=0000000000000000
-Unsupported return value: 0xffffffff</file>