This is an old revision of the document!

Research computing storage infrastructure, 2012

Brainstorming the current status and future needs of the ILRI research computing storage infrastructure.

Current situation

HPC (June, 2011)
- ~6TB of usable disk space, ~2.5TB in use right now
boran (database + VM server, January, 2012)
- ~1.5TB usable disk space, ~20GB in use right now

May 18: Alan, Isaac, Etienne, and Mark meet to discuss projects and upcoming storage requirements. Notable:
- very real possibility of a getting an Illumina MiSeq (shorter reads, but lots and lots of overlapping data)
- Cassava genome project?
May 24: Alan and Isaac meet with NetApp storage representative ("GK" gkumawat@techno-associates.co.ke), facilitated by George Ogoti from ICT
- Existing NetApp is expandable, there are various options we can explore
- GK is going to get us a quote for the following infrastructure
  - RAID-DP (NetApp's version of RAID6, 2 disk failure)
  - Site redundancy (storage syncs nightly via fiber to ICRAF)
  - ILRI site will have two controllers for high availability
  - ICRAF site will have one controller (less critical storage)
  - Capacity 12TB or 24TB (with usable space roughly half of each figure)
June 13, 2012:
- Got the quote back from GK at Techno Associates, two options:
  - 12TB dual configurations: $40,000
  - 24TB dual configurations: $48,000
- We need to talk to Ian Moore to see what he thinks
- It's possible we now use ICT's NetApp to provide 1-2TB for GIS server, then building some custom solution for the DMZ

Proposed architecture assuming we expand ICT's existing NetApp rack with extra controllers and storage.

Key points:

Raw storage is sliced in several chunks and shared appropriately
NetApp exports CIFS shares to corporate clients and servers (users authenticate with Active Directory credentials)
NetApp exports iSCSI block devices to Linux servers in order to allow them to manage their own storage/access/users directly in the OS