Differences

This shows you the differences between two versions of the page.

--- weblab [2009/08/27 10:28] – 172.26.0.166
+++ weblab [2018/04/30 10:12] (current) – Meh, remove this old crap. aorth
@@ Line 1: / Line 1: @@
-===== WebLab =====
-[[http://weblab.cbi.pku.edu.cn/|WebLab]] is a multifunctional bioinformatics analysis platform integrating diversified tools with unified, user-friendly web interface. However, WebLab is not a mere bioinformatics toolbox, but also offer a powerful data management function, group strategy and knowledge sharing mechanism, which brings considerable advance of efficiency for both wet bench and in silico scientists.
-WebLab is complicated to install and we received a disk image from the developers to try and install at ILRI-BecA.
-The image is **really** big, it is located in ''/export2/weblab'':
-  * <code>[alan@hpc-ilri: ~]$ ls -lh /export2/weblab/
-total 324G
--rw-r--r--  1 root wheel 299G Jun 27 19:58 weblab.img
--rw-r--r--  1 root root   25G Aug 19 08:56 weblab.tar.gz</code>
-===== Working with the image =====
-==== Find partitions ====
-It is a //raw// dump of a hard drive, master boot record, partition table, and partitions... everything.  We can use sfdisk to examine the partition table:
-  * <code>[root@hpc-ilri weblab]# sfdisk -l -uS weblab.img
-last_lba(): I don't know how to handle files with mode 81a4
-Disk weblab.img: cannot get size
-Disk weblab.img: cannot get geometry
-Disk weblab.img: 0 cylinders, 0 heads, 0 sectors/track
-Warning: The partition table looks like it was made
-  for C/H/S=*/255/63 (instead of 0/0/0).
-For this listing I'll assume that geometry.
-Units = sectors of 512 bytes, counting from 0
-   Device Boot    Start       End   #sectors  Id  System
-weblab.img1   *        63   1975994    1975932  83  Linux
-weblab.img2       1975995  21527099   19551105  82  Linux swap
-weblab.img3      21527100 625137344  603610245  83  Linux
-weblab.img4             0         -          0   0  Empty</code>
-So there's three partitons, ''sfdisk'' said each sector is 512 bytes so we can determine their sizes in human-readable numbers:
-  * weblab.img1: 942 megs, ext2
-  * weblab.img2: 9.3 gigs, swap (no filesystem)
-  * weblab.img3: 287 gigs, ext3
-We obviously have no need for the swap partition, as we can just create our own.  We can extract partitions one and two and manipulate them.  This includes mounting to copy files or using directly with a virtualization application.
-==== /boot partition ====
-If you want to boot from the image extract it using this command:
-  * <code>[root@hpc-ilri weblab]# dd if=weblab.img of=weblab_partition1.img count=1975995</code>
-You can see Linux recognizes the that file contains a boot sector:
-  * <code>[root@hpc-ilri weblab]# file weblab_partition1.img
-weblab_partition1.img: x86 boot sector, code offset 0x48</code>
-This is a **raw** image which can be booted in a virtualization application like ''qemu'', example:
-  * <code>[root@hpc-ilri weblab]# qemu-system-x86_64 -hda weblab_partition1.img -vnc :0 -m 128</code>
-After which point you can watch the machine's boot process with a VNC viewer on port 5900.
-=== To copy files off ===
-If you want to read or copy files off the first partition then you need to extract it without the boot sector.  Tell ''dd'' to skip 63 sectors to where partition 1 begins:
-  * <code>[root@hpc-ilri weblab]# dd if=weblab.img of=weblab_partition1.img skip=63 count=1975932
-1975932+0 records in
-1975932+0 records out</code>
-See? It's just an ext2 file system:
-  * <code>[root@hpc-ilri weblab]# file weblab_partition1.img
-weblab_partition1.img: Linux rev 1.0 ext2 filesystem data (large files)</code>
-GRUB, a Linux bootloader, typically has problems booting off exotic filesystems, so it's usually safer to put everything needed for booting (kernel!) on an ext2 partition.
-Mount it as a loop device to see what's on it:
-  * <code>[root@hpc-ilri weblab]# mount -o loop weblab_partition1.img /mnt/floppy/
-[root@hpc-ilri weblab]# ls /mnt/floppy/
-boot  gentoo  gentoo.igz  grub</code>
-==== Extract data from / partition ====
-Mount the root partition to copy data directly to or from its filesystem.  Use an offset which points at the starting sector of the image (from ''sfdisk'', 21527100 sectors * 512 bytes per sector = offset of 11021875200 bytes):
-  * <code>[root@hpc-ilri weblab]# mount -o loop,offset=11021875200 weblab.img /mnt/floppy/</code>
-Check the status of the mount...
-  * <code>[root@hpc-ilri weblab]# mount
-/dev/md2 on / type ext3 (rw)
-...
-...
-/export2/weblab/weblab.img on /mnt/floppy type ext3 (rw,loop=/dev/loop0,offset=11021875200)
-[root@hpc-ilri weblab]# df -h
-Filesystem                  Size  Used Avail Use% Mounted on
-/dev/md2                     62G   28G   31G  48% /
-...
-...
-/export2/weblab/weblab.img  284G   40G  230G  15% /mnt/floppy</code>
-The partition is ext3 and there are only 40 gigs in use!
-Create somewhere to copy the data, for example a 50 gigabyte empty file read from /dev/zero:
-  * <code>[root@hpc-ilri weblab]# dd if=/dev/zero of=/export3/weblab_root.img bs=1M count=51200
-+0 records in
-+0 records out</code>
-Format the "drive" as ext3:
-  * <code>[root@hpc-ilri weblab]# mkfs.ext3 /export3/weblab_root.img</code>
-Mount the new drive (it doesn't matter where, just somewhere else than the / partition above):
-  * <code>[root@hpc-ilri weblab]# mount -o loop /export3/weblab_root.img /media/floppy/</code>
-Verify the used space:
-  * <code>[root@hpc-ilri weblab]# df -h
-Filesystem                  Size  Used Avail Use% Mounted on
-/dev/md2                     62G   28G   31G  48% /
-...
-...
-/dev/sdd1                   459G   22G  415G   5% /media/Lacie
-/export2/weblab/weblab.img  284G   40G  230G  15% /mnt/floppy
-/export3/weblab_root.img     50G   84M   47G   1% /media/floppy</code>
-The / partition (mounted here at /mnt/floppy) has 40 gigs used, and our new, empty partition has 47 free, so copy the data using ''cp''.  Make sure you don't mix up the source and target for the copy operation (use ''ls'' to verify before!):
-  * <code>[root@hpc-ilri weblab]# cp -a /mnt/floppy/* /media/floppy/</code>
-Read the note about ''cp -a'' in cp's man page:
-  * <code>-a, --archive
-     same as -dpR</code>
-It's the easiest way to recursively copy a bunch of files from one place to another while preserving time/date stamps as well as owner/group info.
-=== Extracting the entire partition ===
-It's probably mostly empty space (remember, this is a raw dump of another hard disk), but if you want the raw partiton for duplicating to a physical disk or something, extract it to somewhere where there is a lot of space available:
-  * <code>[root@hpc-ilri weblab]# dd if=weblab.img of=/media/Lacie/weblab_root_partition.img skip=21527100 count=603610245</code>
-==== Adding a bootloader to /boot ====
-The MBR needs 512 bytes at the beginning of the disk/image for the boot loader code (GRUB) and the partition table.  First, create an empty 512 byte file:
-  * <code>[root@manatee2 alan]# dd if=/dev/zero of=mbr.raw bs=512 count=1</code>
-Then prepend the image of the boot partition with the embty mbr.raw:
-  * <code>[root@manatee2 alan]# cat mbr.raw weblab_boot.img > weblab_boot_with_mbr.img</code>
-  *
-==== Copy boot data ====
-We extracted the boot partition without its boot loader, so we need to copy it to a new, empty image and then add a boot loader (GRUB).
-First, create an empty 100 meg image:
-  * <code>[root@manatee2 alan]# dd if=/dev/zero of=weblab_boot_new.raw bs=1M count=100
-+0 records in
-+0 records out
-104857600 bytes (105 MB) copied, 0.653573 seconds, 160 MB/s</code>
-Attach a loop device to the image so we can address it as a block device:
-  * <code>[root@manatee2 alan]# losetup -f weblab_boot_new.raw</code>
-Create a partition on the block device (**n**ew, **p**rimary, **1**, with default sectors, sizes, etc) and **w**rite the partition in ''fdisk'':
-  * <code>[root@manatee2 alan]# fdisk -u /dev/loop0</code>
-Delete and recreate the loop device so the kernel knows about the new partition table:
-  * <code>[root@manatee2 alan]# losetup -d /dev/loop0
-[root@manatee2 alan]# losetup -f weblab_boot_new.raw</code>
-Now create a file system on the image's new partition using ''parted'' (mkfs, Yes, 1, ext2, Ignore, quit):
-  * <code>[root@manatee2 alan]# parted /dev/loop0</code>
-Then mount the image's new file system on a loop device with an offset of 32256 bytes (63 sectors * 512 bytes per sector) as a loop device:
-  * <code>[root@manatee2 alan]# mount -o loop,offset=32256 weblab_boot_alan.raw /mnt/weblab_boot_new/</code>
-Mount the original boot image:
-  * <code>[root@manatee2 alan]# mount -o loop weblab_boot.img /mnt/weblab_boot_old/</code>
-Copy everything over using cp's archive functionality (preserving time, date, permissions, owner, etc):
-  * <code>[root@manatee2 alan]# cp -a /mnt/weblab_boot_old/* /mnt/weblab_boot_new/</code>
-Unmount both images:
-  * <code>[root@manatee2 alan]# umount /mnt/weblab_boot_new/
-[root@manatee2 alan]# umount /mnt/weblab_old/</code>
-We'll install grub right on the image (not using a block device).  Start ''grub'' like this so it doesn't try to read all our disks (to create a device map):
-  * <code>[root@manatee2 alan]# grub --device-map=/dev/null</code>
-Type the following commands in grub:
-   * <code>grub> device (hd0) weblab_boot_new.raw
-grub> root (hd0,0)
-grub> setup (hd0)
-grub> quit</code>
-If you don't have any errors,
-===== Booting qemu =====
-Having extracted the boot and root partitions, we can test them in a virtual machine:
-  * <code>[root@manatee2 alan]# qemu-system-x86_64 -kernel rescue64 -initrd initram.igz -append "root=/dev/sdb" -hda weblab_boot.img -hdb weblab_root.img -m 512 -vnc :0 -no-kqemu -redir tcp:5000:10.0.2.15:80 -redir tcp:5001:10.0.2.15:443</code>
-  * Mount the root image and change ''/etc/fstab'' entries
-  * Mount the root image and copy a new root hash to ''/etc/shadow''
-===== Troubleshooting =====
-Note to self: qemu on 64 bit host with a 64 bit guest does not like to run with the ''kqemu'' kernel module!  Start qemu with ''-no-kqemu'' if you're seeing errors/crashes like this:
-  * <file>RAX=00007f392e505de8 RBX=000000006ffffeff RCX=0000000001dd4100 RDX=00007f392e2f2000
-RSI=00007f392e505f18 RDI=00007f392e507000 RBP=00007fff0cb6b1a0 RSP=00007fff0cb6b000
-R8 =000000006fffffff R9 =0000000070000021 R10=000000006ffffdff R11=000000006ffffe34
-R12=0000000001dd40c0 R13=000000006fffff40 R14=00007fff0cb6b268 R15=0000000000000002
-RIP=00007f392eeb77a8 RFL=00010206 [-----P-] CPL=3 II=0 A20=1 SMM=0 HLT=0
-ES =0000 0000000000000000 00000000 00000000
-CS =0033 0000000000000000 ffffffff 00affb00
-SS =002b 0000000000000000 ffffffff 00cff300
-DS =0000 0000000000000000 00000000 00000000
-FS =0000 00007f392f0ba6f0 00000000 00000000
-GS =0000 0000000000000000 00000000 00000000
-LDT=0000 0000000000000000 00000000 00008200
-TR =0040 ffff88000100c080 00002087 00008900
-GDT=     ffff880001007000 0000007f
-IDT=     ffffffff81981000 00000fff
-CR0=8005003b CR2=00007f392e505df0 CR3=00000000089d5000 CR4=000006e0
-DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
-DR6=0000000000000000 DR7=0000000000000000
-Unsupported return value: 0xffffffff</file>