upgrade_notes:hpc_june_2011
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| upgrade_notes:hpc_june_2011 [2011/06/20 11:53] – [Dendroscope] aorth | upgrade_notes:hpc_june_2011 [2011/12/01 09:05] (current) – removed aorth | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ====== HPC installation, | ||
| - | Notes documenting the installation of the new HPC server in June, 2011. The machine is a Dell PowerEdge R910. | ||
| - | |||
| - | Machine specifications: | ||
| - | * Quad Eight-core Xeon X7560 | ||
| - | * 128 GB RAM | ||
| - | * 16 500GB, 7200RPM Serial-attached SCSI drives | ||
| - | * Dell PowerEdge RAID Controller (PERC) H700, {{: | ||
| - | |||
| - | ====== Pre-installation notes ====== | ||
| - | |||
| - | ===== Hardware RAID ===== | ||
| - | Hardware RAID configuration set up to provide the following virtual/ | ||
| - | * Disks 0,1,2,3 -> RAID5 | ||
| - | * Disks 4, | ||
| - | * Disks 12,13,14,15 -> RAID5 | ||
| - | |||
| - | ... where disks are physically laid out as follows: | ||
| - | | Disk 0 | Disk 4 | Disk 8 | Disk 12 | | ||
| - | | Disk 1 | Disk 5 | Disk 9 | Disk 13 | | ||
| - | | Disk 2 | Disk 6 | Disk 10 | Disk 14 | | ||
| - | | Disk 3 | Disk 7 | Disk 11 | Disk 15 | | ||
| - | |||
| - | The first disk group must not be too large because we need to boot from it. CentOS uses the legacy GRUB1 bootloader, which only supports the MSDOS partition table format, meaning that partitions cannot be larger than ~2 TB (http:// | ||
| - | |||
| - | It seems the hardware RAID controller automatically sets partition tables to GPT when you create virtual disks, and CentOS' | ||
| - | |||
| - | Enter parted and ignore any errors it spits out in the beginning: | ||
| - | < | ||
| - | mklabel | ||
| - | yes | ||
| - | msdos | ||
| - | q</ | ||
| - | |||
| - | After that you can switch back to the installer (Ctrl-Alt-F6) and partition away. | ||
| - | |||
| - | ====== Post-installation notes ====== | ||
| - | Now SSH to the new server' | ||
| - | * HPC: 192.168.5.3 | ||
| - | * HPCTEST: 192.168.5.4 | ||
| - | ===== Preparation ===== | ||
| - | ==== Rsync with ACL support ==== | ||
| - | CentOS' | ||
| - | < | ||
| - | wget http:// | ||
| - | tar zxf rsync-3.0.8.tar.gz | ||
| - | ./configure --prefix=/ | ||
| - | make | ||
| - | sudo make install</ | ||
| - | Now we can use this special version of rsync later by calling: ''/ | ||
| - | |||
| - | ===== Rocks service pack ===== | ||
| - | Install the latest Rocks service pack. See documentation here: http:// | ||
| - | |||
| - | ===== Users, groups, etc ===== | ||
| - | |||
| - | Migrate existing users, groups, and home folders from the old server. | ||
| - | |||
| - | ==== On the old server ==== | ||
| - | Backup a few things and prepare for user/group migration... | ||
| - | === /etc === | ||
| - | Create a tarball of the system configuration files: | ||
| - | < | ||
| - | |||
| - | === Rocks MySQL databases === | ||
| - | Rocks uses its own instance of mysql, in addition to the " | ||
| - | < | ||
| - | export HISTFILE=/ | ||
| - | for dbname in `/ | ||
| - | exit</ | ||
| - | |||
| - | === System MySQL databases === | ||
| - | The regular CentOS system MySQL databases... just in case! Make sure mysqld is running then: | ||
| - | < | ||
| - | export HISTFILE=/ | ||
| - | for dbname in `mysql -u root -p' | ||
| - | exit</ | ||
| - | |||
| - | === Storix backup tool === | ||
| - | Commercial tape backup software. | ||
| - | < | ||
| - | tar cjf / | ||
| - | tar cjf / | ||
| - | ==== On the new server ==== | ||
| - | < | ||
| - | Copy the tarball from the old server: | ||
| - | < | ||
| - | tar jxf etc_june_2011.tar.bz2</ | ||
| - | |||
| - | === Migrate users and groups in /etc...=== | ||
| - | * Manually copy all but system users, from around segoli at UID 658, into /// | ||
| - | * Delete old/ | ||
| - | * Delete directories: | ||
| - | * Copy users' passwords (same as users above) into /// | ||
| - | * Delete old/ | ||
| - | * Copy all but system groups into /// | ||
| - | * Delete old/ | ||
| - | * Make sure to copy important group memberships like ssh, gcc, wheel, etc... | ||
| - | * Copy groups' | ||
| - | * Delete old/ | ||
| - | |||
| - | === / | ||
| - | Allow the wheel group to use sudo, find and uncomment the following line: | ||
| - | < | ||
| - | |||
| - | === Edit passwd to reflect home directory location === | ||
| - | * Replace ''/ | ||
| - | * Run '' | ||
| - | |||
| - | === Copy homes === | ||
| - | * rsync -avz --exclude " | ||
| - | * (or... with --delete if you're absolutely sure!!! | ||
| - | * not sure if the condor user is support to have a home? | ||
| - | * Anne Fischer and Segoli are special cases, their home folders were on export2, with a symlink in / | ||
| - | * Copy Anne's stuff: | ||
| - | * '' | ||
| - | * Copy Segolip' | ||
| - | * Add '' | ||
| - | * Sync: ''/ | ||
| - | |||
| - | === Root home === | ||
| - | Temporarily enable password authentication for root on the old HPC, then copy his home directory over... | ||
| - | < | ||
| - | rsync -avz 192.168.5.4:/ | ||
| - | |||
| - | Copy important things like configs, ssh keys, etc... | ||
| - | |||
| - | === Copy / | ||
| - | Make sure to exclude some old/unused stuff... | ||
| - | < | ||
| - | |||
| - | === Copy / | ||
| - | The system' | ||
| - | < | ||
| - | Then copy everything over, exlcuding some old/unused stuff. | ||
| - | < | ||
| - | Clean up old mysql databases which don't exist anymore (they are sym links to non-existent places): | ||
| - | < | ||
| - | find . -type l -exec rm {} \;</ | ||
| - | |||
| - | === Symlink for legacy /mnt/export === | ||
| - | < | ||
| - | |||
| - | === Test SSH as your user! === | ||
| - | Now the home directories and users/ | ||
| - | ===== Configure yum ===== | ||
| - | Rocks 5.4 is based on CentOS 5.5, so you can use the repositories directly from CentOS. | ||
| - | |||
| - | ==== CentOS-Base.repo ==== | ||
| - | Copy a '' | ||
| - | < | ||
| - | # | ||
| - | # The mirror system uses the connecting IP address of the client and the | ||
| - | # update status of each mirror to pick mirrors that are updated to and | ||
| - | # geographically close to the client. | ||
| - | # unless you are manually picking other mirrors. | ||
| - | # | ||
| - | # If the mirrorlist= does not work for you, as a fall back you can try the | ||
| - | # remarked out baseurl= line instead. | ||
| - | # | ||
| - | # | ||
| - | |||
| - | [base] | ||
| - | name=CentOS-$releasever - Base | ||
| - | mirrorlist=http:// | ||
| - | # | ||
| - | gpgcheck=1 | ||
| - | gpgkey=file:/// | ||
| - | |||
| - | #released updates | ||
| - | [updates] | ||
| - | name=CentOS-$releasever - Updates | ||
| - | mirrorlist=http:// | ||
| - | # | ||
| - | gpgcheck=1 | ||
| - | gpgkey=file:/// | ||
| - | |||
| - | #packages used/ | ||
| - | [addons] | ||
| - | name=CentOS-$releasever - Addons | ||
| - | mirrorlist=http:// | ||
| - | # | ||
| - | gpgcheck=1 | ||
| - | gpgkey=file:/// | ||
| - | |||
| - | #additional packages that may be useful | ||
| - | [extras] | ||
| - | name=CentOS-$releasever - Extras | ||
| - | mirrorlist=http:// | ||
| - | # | ||
| - | gpgcheck=1 | ||
| - | gpgkey=file:/// | ||
| - | |||
| - | #additional packages that extend functionality of existing packages | ||
| - | [centosplus] | ||
| - | name=CentOS-$releasever - Plus | ||
| - | mirrorlist=http:// | ||
| - | # | ||
| - | gpgcheck=1 | ||
| - | enabled=0 | ||
| - | gpgkey=file:/// | ||
| - | |||
| - | #contrib - packages by Centos Users | ||
| - | [contrib] | ||
| - | name=CentOS-$releasever - Contrib | ||
| - | mirrorlist=http:// | ||
| - | # | ||
| - | gpgcheck=1 | ||
| - | enabled=0 | ||
| - | gpgkey=file:/// | ||
| - | |||
| - | ==== Excludes ==== | ||
| - | We don't want to update packages that Rocks depends on, or else we might break it! See the [[https:// | ||
| - | < | ||
| - | |||
| - | ==== Test ==== | ||
| - | Try to test the new repository using '' | ||
| - | ===== Software Configuration / Installation ===== | ||
| - | |||
| - | ==== SSH configuration ==== | ||
| - | === SSH Host Keys === | ||
| - | Copy the SSH keys from the backup of the old server: | ||
| - | < | ||
| - | rm -rf / | ||
| - | cp backup_old_hpc/ | ||
| - | === ssh_config === | ||
| - | Disable Protocol 1 in the system-wide SSH client configuration, | ||
| - | < | ||
| - | |||
| - | === sshd_config === | ||
| - | Make the following changes to the SSH daemon' | ||
| - | < | ||
| - | |||
| - | GSSAPIAuthentication no | ||
| - | |||
| - | # only allow members of the " | ||
| - | AllowGroups | ||
| - | |||
| - | === / | ||
| - | Host-based allow/deny for remote login... | ||
| - | |||
| - | === Restart sshd === | ||
| - | < | ||
| - | |||
| - | === Test === | ||
| - | Test from another machine to see if you can log in without it complaining about host keys! | ||
| - | |||
| - | ==== From yum ==== | ||
| - | Some random, one-off things from the top of my head: | ||
| - | < | ||
| - | |||
| - | Install all CentOS updates: | ||
| - | < | ||
| - | ==== Environment Modules ==== | ||
| - | https:// | ||
| - | < | ||
| - | rocks enable roll modules | ||
| - | cd / | ||
| - | rocks create distro | ||
| - | rocks run roll modules >> installroll.sh | ||
| - | sh installroll.sh | ||
| - | rm installroll.sh</ | ||
| - | |||
| - | === Module files === | ||
| - | Copy module files from HPC to / | ||
| - | < | ||
| - | |||
| - | Edit module files to reflect current application locations! | ||
| - | < | ||
| - | cd / | ||
| - | ln -sv 2.2.25+ latest</ | ||
| - | |||
| - | Edit paths too! | ||
| - | < | ||
| - | cd / | ||
| - | sed -i ' | ||
| - | ==== R Statistics ==== | ||
| - | http:// | ||
| - | < | ||
| - | cd R-2.13.0 | ||
| - | ./configure --prefix=/ | ||
| - | make | ||
| - | sudo make install</ | ||
| - | |||
| - | Repeat the process for R 2.12.2 and 2.11.0, just in case users have those versions installed, we don't want to cause incompatibilities for them! Also, make sure the latest version has a symlink for itself! | ||
| - | < | ||
| - | cd / | ||
| - | ln -sv 2.13.0 latest</ | ||
| - | |||
| - | Make a similar one for the module file... | ||
| - | ==== NCBI BLAST+ ==== | ||
| - | === Install === | ||
| - | ftp:// | ||
| - | < | ||
| - | tar zxf ncbi-blast-2.2.25+-x64-linux.tar.gz | ||
| - | sudo cp -r ncbi-blast-2.2.25+ / | ||
| - | |||
| - | Create a symlink for the " | ||
| - | < | ||
| - | cd / | ||
| - | ln -sv 2.2.25+ latest</ | ||
| - | |||
| - | === Setup BLAST+ databases === | ||
| - | Download a few databases from NCBI. I think a good location on the new server would be / | ||
| - | < | ||
| - | cd / | ||
| - | for name in nr.*gz; do tar zxf $name; done | ||
| - | / | ||
| - | for name in *tar.gz; do tar zxf $name; done | ||
| - | rm *.gz *.md5</ | ||
| - | |||
| - | === Test BLAST+ === | ||
| - | Pass the BLASTDB variable to blastp to see if it can find the nr database we just unzipped. | ||
| - | < | ||
| - | |||
| - | ==== Structure ==== | ||
| - | http:// | ||
| - | < | ||
| - | cd structure_kernel_src | ||
| - | make | ||
| - | sudo mkdir -p / | ||
| - | sudo cp structure / | ||
| - | |||
| - | ==== MrBayes ==== | ||
| - | http:// | ||
| - | < | ||
| - | cd mrbayes-3.1.2 | ||
| - | sudo mkdir -p / | ||
| - | sudo cp mb / | ||
| - | |||
| - | ==== BEAST ==== | ||
| - | http:// | ||
| - | < | ||
| - | sudo mkdir -p / | ||
| - | sudo cp -r BEASTv1.6.1 / | ||
| - | |||
| - | ==== Python ==== | ||
| - | http:// | ||
| - | < | ||
| - | cd Python-2.7.1 | ||
| - | ./configure --prefix=/ | ||
| - | make | ||
| - | sudo make install</ | ||
| - | |||
| - | === Python - NumPy === | ||
| - | http:// | ||
| - | < | ||
| - | cd numpy-1.6.0 | ||
| - | / | ||
| - | sudo / | ||
| - | |||
| - | === Python - BioPython === | ||
| - | http:// | ||
| - | < | ||
| - | cd biopython-1.57 | ||
| - | / | ||
| - | sudo / | ||
| - | |||
| - | ==== Dendroscope ==== | ||
| - | http:// | ||
| - | < | ||
| - | sudo ./ | ||
| - | Tell the installer to install it to /// | ||
| - | |||
| - | ==== 454 tools ==== | ||
| - | < | ||
| - | sudo su - | ||
| - | mv ~aorth/454 / | ||
| - | chown -R root:root / | ||
| - | ln -s / | ||
| - | |||
| - | ==== IPtables firewall rules ==== | ||
| - | Add the following to /// | ||
| - | < | ||
| - | # accept SSH from 192.168.5.1 (ILRI corporate network) | ||
| - | #-A INPUT -m state --state NEW -p tcp --dport ssh --source 192.168.5.1 -j ACCEPT | ||
| - | # need a way to specify the range of IP (ie, NOT 192.168.5.1, | ||
| - | #-I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --set | ||
| - | #-I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP | ||
| - | |||
| - | # NFS for ILRI corporate network | ||
| - | -A INPUT -s 192.168.5.1/ | ||
| - | -A INPUT -s 192.168.5.1/ | ||
| - | -A INPUT -s 192.168.5.1/ | ||
| - | -A INPUT -s 192.168.5.1/ | ||
| - | -A INPUT -s 192.168.5.1/ | ||
| - | -A INPUT -s 192.168.5.1/ | ||
| - | -A INPUT -s 192.168.5.1/ | ||
| - | -A INPUT -s 192.168.5.1/ | ||
| - | -A INPUT -s 192.168.5.1/ | ||
| - | -A INPUT -s 192.168.5.1/ | ||
| - | -A INPUT -s 192.168.5.1/ | ||
| - | |||
| - | # samba rules for ILRI corporate network | ||
| - | -A INPUT -s 192.168.5.1/ | ||
| - | -A INPUT -s 192.168.5.1/ | ||
| - | -A INPUT -s 192.168.5.1/ | ||
| - | -A INPUT -s 192.168.5.1/ | ||
| - | -A INPUT -s 192.168.5.1/ | ||
| - | |||
| - | #Zabbix Monitering | ||
| - | #-A INPUT -m state --state NEW -p tcp --dport 10050 -j ACCEPT | ||
| - | |||
| - | # SWAT rules | ||
| - | -A INPUT -m state --state NEW -p tcp --dport 901 -j ACCEPT</ | ||
| - | |||
| - | |||
| - | ==== Samba ==== | ||
| - | === Install === | ||
| - | < | ||
| - | === Configure === | ||
| - | - Edit /// | ||
| - | - < | ||
| - | server string = Samba Server Version %v | ||
| - | |||
| - | netbios name = HPC | ||
| - | |||
| - | interfaces = lo eth1 | ||
| - | hosts deny = ALL | ||
| - | hosts allow = 127. 192.168.5.1/ | ||
| - | |||
| - | load printers = no | ||
| - | printing = bsd | ||
| - | printcap name = /dev/null | ||
| - | disable spoolss = yes | ||
| - | |||
| - | |||
| - | ;[printers] | ||
| - | ; | ||
| - | ; path = / | ||
| - | ; | ||
| - | ; guest ok = no | ||
| - | ; | ||
| - | ; | ||
| - | |||
| - | - Convert the old smbpasswd file to tdbsam (this is using the smbpasswd file we backed up): | ||
| - | - '' | ||
| - | - Start samba: | ||
| - | - '' | ||
| - | - Set samba to start when the system boots: | ||
| - | - '' | ||
| - | |||
| - | |||
| - | |||
| - | ==== Apache ==== | ||
| - | === / | ||
| - | Move the stock Rocks index.html | ||
| - | < | ||
| - | cd / | ||
| - | mv index.html index.html.rocks</ | ||
| - | Synchronize the other files and folders from the old server: | ||
| - | < | ||
| - | === / | ||
| - | Copy the contents of cgi-bin | ||
| - | < | ||
| - | |||
| - | === httpd.conf === | ||
| - | Change at least the following in Apache' | ||
| - | < | ||
| - | < | ||
| - | Options FollowSymLinks | ||
| - | AllowOverride Options | ||
| - | </ | ||
| - | < | ||
| - | #UserDir disable | ||
| - | UserDir public_html | ||
| - | </ | ||
| - | </ | ||
| - | |||
| - | === Other configs === | ||
| - | Assuming you have a backup of the old HPC's ///etc// in your folder, copy the following to the new server' | ||
| - | < | ||
| - | cp etc/ | ||
| - | |||
| - | === mod_perl === | ||
| - | Certain CGI web applications will need mod_perl and some perl modules, lets install them to preempt any problems! | ||
| - | < | ||
| - | |||
| - | === Restart Apache === | ||
| - | < | ||
| - | |||
| - | ==== Viroblast ==== | ||
| - | A cool web-based BLAST+ interface. | ||
| - | |||
| - | ===Install=== | ||
| - | < | ||
| - | cd / | ||
| - | tar xvfp / | ||
| - | cd viroblast</ | ||
| - | |||
| - | === Setup databases === | ||
| - | First, make symlinks to the various installed databases (pre-formatted from NCBI): | ||
| - | < | ||
| - | ln -sv / | ||
| - | |||
| - | Then setup the '' | ||
| - | < | ||
| - | blastp: test_aa_db => Protein test database, nr => NCBI nr (June 2011)</ | ||
| - | |||
| - | === Update the blast+ version === | ||
| - | Viroblast 2.2 comes with BLAST+ 2.2.24, symlink the latest installed version we have: | ||
| - | < | ||
| - | ln -sv / | ||
| - | |||
| - | === Apache configuration === | ||
| - | Tell Apache to load '' | ||
| - | < | ||
| - | DirectoryIndex viroblast.php | ||
| - | | ||
| - | # Allow slightly-larger file sizes to be uploaded for viroblast | ||
| - | php_value upload_max_filesize 10M | ||
| - | php_value post_max_size 10M | ||
| - | </ | ||
| - | |||
| - | # redirect requests to paracel' | ||
| - | < | ||
| - | RewriteRule ^/ | ||
| - | </ | ||
| - | |||
| - | Restart Apache: | ||
| - | < | ||
| - | |||
| - | Make sure to test with a small data set to make sure it works! | ||
| - | |||
| - | ==== MySQL ==== | ||
| - | Start the service so it can be initialized for the first time: | ||
| - | < | ||
| - | |||
| - | Reset the root user's password then secure the installation (remove anonymous users and test database): | ||
| - | < | ||
| - | export HISTFILE=/ | ||
| - | / | ||
| - | / | ||
| - | / | ||
| - | exit</ | ||
| - | |||
| - | ==== Not done yet!! ==== | ||
| - | * MySQL databases for formtools, pasa, others? | ||
| - | * Check ncbirc and make sure they are not conflicting with new stuff | ||
| - | * quotas | ||
| - | ===== Disk Quotas ===== | ||
| - | * See: https:// | ||
| - | * See: https:// | ||
| - | * See: http:// | ||
| - | * See: http:// | ||
| - | |||
| - | * student/ | ||
| - | * " | ||
| - | |||
| - | ==== Implementation ==== | ||
| - | * Add " | ||
| - | * Restart the computer | ||
| - | * Create quota files on the file system and check current disk usage: | ||
| - | * '' | ||
| - | * | ||
| - | |||
| - | ===== Hardware RAID controller tools ===== | ||
| - | The Dell PERC H700 is a re-branded LSI MegaRaid: | ||
| - | < | ||
| - | 01:00.0 RAID bus controller: LSI Logic / Symbios Logic LSI MegaSAS 9260 (rev 04)</ | ||
| - | |||
| - | Drivers can be found here: http:// | ||
| - | ==== Installation ==== | ||
| - | < | ||
| - | |||
| - | ==== Testing ==== | ||
| - | List available controllers: | ||
| - | < | ||
upgrade_notes/hpc_june_2011.1308570786.txt.gz · Last modified: by aorth