Differences

This shows you the differences between two versions of the page.

--- raid [2009/11/16 13:59] – 172.26.0.166
+++ raid [2009/11/19 07:07] – 172.26.0.166
@@ Line 50: / Line 50: @@
 unused devices: <none></code>
 ==== Repair RAID ====
+When a disk is failing you might see errors in the system logs from smartd like this:
+<file>Device: /dev/hda, 1 Offline uncorrectable sectors</file>
+In that case you need to replace the drive.  First, look at the RAID configuration to see which partitions are in use by which arrays.  For example:
+<code># cat /proc/mdstat
+Personalities : [raid1] [raid0]
+md3 : active raid1 hdd1[1] hda1[0]
+blocks [2/2] [UU]
+md1 : active raid1 hdd3[1] hda3[0]
+      26627648 blocks [2/2] [UU]
+md2 : active raid0 hdd5[1] hda5[0]
+      36868608 blocks 256k chunks
+md4 : active raid1 hdd6[1] hda6[0]
+      2168640 blocks [2/2] [UU]
+md0 : active raid1 hdd2[1] hda2[0]
+      30716160 blocks [2/2] [UU]
+unused devices: <none></code>
-Setting a disk faulty/failed:
+Because it is ''/dev/hda'' which was having problems, set all its RAID1 partitions as failed and remove them:
+<code># mdadm /dev/md0 --fail /dev/hda2 --remove /dev/hda2
-# mdadm --fail /dev/md0 /dev/hdc1
+# mdadm /dev/md1 --fail /dev/hda3 --remove /dev/hda3
+# mdadm /dev/md3 --fail /dev/hda1 --remove /dev/hda1
-DO NOT run this every on a raid0 or linear device or your data is toasted!
+# mdadm /dev/md4 --fail /dev/hda6 --remove /dev/hda6
+# umount /dev/md2
-Removing a faulty disk from an array:
+# mdadm --stop /dev/md2</code>
+<note warning> You must Shutdown the server before you physically remove the drive! </note>
-# mdadm --remove /dev/md0 /dev/hdc1
+Shut the server down and replace the faulty drive with a new one.  After booting your drive letters may have shifted around, so just be sure to verify which is which before proceeding.
+Clone the partition table from the good drive to the bad one:
+<code># sfdisk -d /dev/hda | sfdisk --force /dev/hdd</code>
+Verify the new partitions can be seen:
+<code># partprobe -s
+/dev/hda: msdos partitions 1 2 3 4 <5 6>
+/dev/hdd: msdos partitions 1 2 3 4 <5 6>
+/dev/sda: msdos partitions 1
+/dev/sdb: msdos partitions 1
+/dev/sdc: msdos partitions 1
+</code>
+Re-create the scratch partition (RAID0):
+<code># mdadm --create --verbose /dev/md2 --level=0 --raid-devices=2 /dev/hda5 /dev/hdd5
+# mkfs.ext3 /dev/md2
+# mount /dev/md2 /scratch</code>
+You can now add the new partitions back to the RAID1 arrays:
+<code># mdadm /dev/md0 --add /dev/hdd2
+# mdadm /dev/md1 --add /dev/hdd3
+# mdadm /dev/md3 --add /dev/hdd1
+# mdadm /dev/md4 --add /dev/hdd6</code>
+After adding you can monitor the progress of the RAID rebuilds by looking in ''/proc/mdstat'':
+<file>Personalities : [raid1] [raid0]
+md3 : active raid1 hdd1[1] hda1[0]
+blocks [2/2] [UU]
+md1 : active raid1 hdd3[2] hda3[0]
+      26627648 blocks [2/1] [U_]
+      [===================>.]  recovery = 95.4% (25407552/26627648) finish=0.7min speed=28648K/sec
+md2 : inactive hda5[0]
+      18434304 blocks
+md4 : active raid1 hdd6[2] hda6[0]
+      2168640 blocks [2/1] [U_]
+        resync=DELAYED
+md0 : active raid1 hdd2[1] hda2[0]
+      30716160 blocks [2/2] [UU]
+unused devices: <none></file>
 Clearing any previous raid info on a disk (eg. reusing a disk from another decommissioned raid array)
@@ Line 81: / Line 140: @@
 How do you know which physical disk is broken to replace it?
+f
 ===== Hardware RAID =====