Differences

This shows you the differences between two versions of the page.

--- brainstorms:research_computing_storage_infrastructure_2012 [2012/05/25 08:46] – created aorth
+++ brainstorms:research_computing_storage_infrastructure_2012 [2012/05/25 15:56] – [Proposed architecture] aorth
@@ Line 1: / Line 1: @@
 ====== Research computing storage infrastructure, 2012 ======
-Brainstorming the current and future needs of the ILRI research computing storage infrastructure (we're currently swimming in compute capacity).
+Brainstorming the current status and future needs of the ILRI research computing storage infrastructure.
+===== Current situation =====
+  * HPC (June, 2011)
+    * ~6TB of usable disk space, ~2.5TB in use right now
+  * boran (database + VM server, January, 2012)
+    * ~1.5TB usable disk space, ~20GB in use right now
+===== Timeline =====
+  * **May 18**: Alan, Isaac, Etienne, and Mark meet to discuss projects and upcoming storage requirements.  Notable:
+    * very real possibility of a getting an Illumina MiSeq (shorter reads, but lots and lots of overlapping data)
+    * Cassava genome project?
+  * **May 24**: Alan and Isaac meet with NetApp storage representative ("GK" <gkumawat@techno-associates.co.ke>), facilitated by George Ogoti from ICT
+    * Existing NetApp is expandable, there are various options we can explore
+    * GK is going to get us a quote for the following infrastructure
+      * RAID-DP (NetApp's version of RAID6, 2 disk failure)
+      * Site redundancy (storage syncs nightly via fiber to ICRAF)
+      * ILRI site will have two controllers for high availability
+      * ICRAF site will have one controller (less critical storage)
+      * Capacity 12TB or 24TB (with usable space roughly half of each figure)
+===== Proposed NetApp architecture =====
+Proposed architecture assuming we expand ICT's existing NetApp rack with extra controllers and storage.
+{{ :brainstorms:research_computing_storage_2012.png?nolink |}}
+**Key points**:
+  * Raw storage is sliced in several chunks and shared appropriately
+  * NetApp exports CIFS shares to corporate clients and servers (users authenticate with Active Directory credentials)
+  * NetApp exports iSCSI block devices to Linux servers in order to allow them to manage their own storage/access/users directly in the OS
+===== Alternatives =====
+  * Build our own storage, ala Backblaze "pods": http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/
+  * Using FreeBSD + ZFS?