Screwed by 3GB

At work I have a server setup to store all the backup data from the other servers. Backing up to disk over the network is much faster than backing up all those servers to tape and is a lot cheaper. I have a RAID (Redundant Array of Independent Disks) system setup in this server to prevent any data loss if any of the disks should go bad. There are a few different flavors of RAID; the one I'm using is RAID 1, or mirroring. This involves two "identical" disks having the same data. When data is written to the array it gets put on both disks at the same time. In the event that one of the disks should fail, the other one will continue to function and no data will be lost. Obviously if both disks fail you're screwed.

When I was planning this server I ordered two 200GB disks for the backups. These disks were identical; they were the same manufacturer, make, model and size. So when I created the array I used the maximum amount of space on the disks for the partitions. It worked great for a year or so. Last week one of those disks failed. It was still under warranty, but I bought a new one instead because if I tried to get a replacement they would want the old disk back and it still has all sorts of private data on it that I can't remove.

I ordered a new 200GB disk last week (the day the old one failed) and it arrived yesterday morning. I transplanted the drives and run the "raidhotadd" command expecting to have the system recognize the new disk and add it the array. It didn't. After some hair pulling I found out that the new disk is actually smaller than the original. Both times I asked for 200GB disks and both times I received 200GB disks (according to the box and the labels on the disks), but the original pair were actually 203GB. You can't mirror two disks when one of them is 3GB larger than the other.

Luckily I was had enough room on another drive in that server copy all the data so I could recreate the partitions on the one old drive and the one new drive to around 185GB so they would both match and could mirror each other. I purposely left some room in case this happens again and I get a disk that is 199GB.

Jun 27 09:26:37 storage kernel: hde: dma_intr: status=0x51 { DriveReady SeekComplete Error }

Jun 27 09:26:37 storage kernel: hde: dma_intr: error=0x40 { UncorrectableError }, LBAsect=24746384, high=1, low=7969168, sector=24746320

Jun 27 09:26:37 storage kernel: end_request: I/O error, dev 21:01 (hde), sector 24746320

Jun 27 09:26:37 storage kernel: raid1: Disk failure on hde1, disabling device.

Jun 27 09:26:37 storage kernel: ^IOperation continuing on 1 devices

Jun 27 09:26:37 storage kernel: raid1: mirror resync was not fully finished, restarting next time.

Jun 27 09:26:37 storage kernel: md: updating md0 RAID superblock on device

Jun 27 09:26:37 storage kernel: md: hdg1 [events: 00000002]<6>(write) hdg1's sb offset: 184747392

Jun 27 09:26:37 storage kernel: md: recovery thread got woken up ...

Jun 27 09:26:37 storage kernel: md0: no spare disk to reconstruct array! -- continuing in degraded mode

Jun 27 09:26:37 storage kernel: md: (skipping faulty hde1 )

Jun 27 09:26:39 storage kernel: raid1: mirror resync was not fully finished, restarting next time.