13 October 2009Own project · Storage & file systemsOwn post

Software RAID in 2009: my mdadm cheat sheet for disks, spares and GRUB

Summary

In October 2009 my server odin ran a software RAID 5 for data and a RAID 1 with a hot spare for the system, managed with mdadm. So that I would not have to search every time a disk was replaced, I noted down the commands: copying partitions, removing and adding disks, swapping spares, writing GRUB to every disk and understanding the auto-read-only state. This post from my old blog explains the steps with the output from back then and shows what has changed since.

The setup in 2009

My server was called odin and had three disks: two SATA disks as sda and sdb and an older IDE disk as hda. Each disk had two partitions. The first partitions formed a RAID 1 for the system (md2), the second ones a RAID 5 of just over 200 GB for data (md1). The commands were preceded by a foreword I would write differently today:

“RAID-5 makes no sense in software (XOR should be done in hardware in a RAID). I did it anyway so as not to have to buy additional hardware (disks/RAID5 controller).”

Preparing, removing and adding disks

A new disk needs the same partitioning as the others. sfdisk dumps one disk’s table as text and writes it to the other:

odin:~# sfdisk -d /dev/sda | sfdisk /dev/hda

mdadm then takes the partitions out of the array and puts them back in. Adding starts the rebuild immediately, and /proc/mdstat shows its progress:

odin:~# mdadm --manage --remove /dev/md1 /dev/hda2
odin:~# mdadm --manage --add /dev/md1 /dev/hda2
odin:~# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active raid5 hda2[3] sdb2[1] sda2[2]
      214836992 blocks level 5, 64k chunk, algorithm 2 [3/2] [_UU]
      [=>...................]  recovery =  7.0% (7588176/107418496) finish=30.6min speed=54199K/sec

The output is compact but complete: [3/2] means three disks are expected and two are active, [_UU] shows which one is missing. A RAID 5 of three disks survives exactly one failure; during this half hour of rebuilding it had no reserve.

A hot spare for the system

For the system RAID 1 I used the third partition as a hot spare. It sits idle and steps in as soon as an active disk fails. In /proc/mdstat it is marked with (S):

odin:~# mdadm --manage --add /dev/md2 /dev/hda1
odin:~# cat /proc/mdstat
md2 : active raid1 hda1[2](S) sda1[0] sdb1[1]
      9767424 blocks [2/2] [UU]

The test was the interesting part: I marked sdb1 as faulty by hand. The spare hda1 took over immediately and was synchronised. Then I removed sdb1 and added it again, and now sdb1 was the new spare. The roles had swapped without the system ever being without a mirror.

odin:~# mdadm /dev/md2 --fail /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md2
odin:~# mdadm /dev/md2 --remove /dev/sdb1
mdadm: hot removed /dev/sdb1
odin:~# mdadm /dev/md2 --add /dev/sdb1
mdadm: re-added /dev/sdb1
odin:~# cat /proc/mdstat
md2 : active raid1 sdb1[2](S) sda1[0] hda1[3]
      9767424 blocks [2/1] [U_]
      [==>..................]  recovery = 13.7% (1348288/9767424) finish=4.5min speed=31144K/sec

GRUB on every disk

A mirrored system is useless if only one disk has a boot loader. If exactly that disk fails, the machine no longer boots. So after every replacement I wrote GRUB, still the old version 0.97 at the time, to all disks. In the GRUB shell, find locates the disks with /boot, root selects one and setup writes the boot loader to its start:

odin:~# grub
grub> find /boot/grub/stage1
 (hd0,0)
 (hd1,0)
 (hd2,0)
grub> root (hd0,0)
 Filesystem type is ext2fs, partition type 0xfd
grub> setup (hd0)
grub> root (hd1,0)
grub> setup (hd1)
grub> root (hd2,0)
grub> setup (hd2)
grub> quit

Partition type 0xfd stands for “Linux raid autodetect”. GRUB 0.97 could read from a RAID 1 partition because the md metadata version 0.90 common at the time lived at the end of the partition: to the boot loader, each half of the mirror looked like an ordinary ext file system.

What auto-read-only means

After a reboot md1 suddenly showed (auto-read-only), and my comment was a question mark: the state supposedly prevented a rebuild “in the next hours”. The kernel documentation is more precise: read-auto behaves like read-only, i.e. no resync and no metadata written, but switches to normal operation on the first write. An array that nobody is writing to simply waits. If you do not want to wait, switch it by hand:

odin:~# mdadm --readwrite /dev/md1

To try out: RAID 1 from two files

The note ended with an experiment: a RAID 1 made of two files that look like disks via loop devices. To this day it is the best way to practise mdadm without risk.

dd if=/dev/zero of=/export/ftp_space1/raid1_file0 bs=10240 count=10240
dd if=/dev/zero of=/export/ftp_space2/raid1_file1 bs=10240 count=10240
losetup /dev/loop0 /export/ftp_space1/raid1_file0
losetup /dev/loop1 /export/ftp_space1/raid1_file1
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/loop0 /dev/loop1
mkfs.ext3 /dev/md0
mount /dev/md0 /export/ftpraid0

Seen from today there is a typo: the second file is created in ftp_space2, but losetup accesses ftp_space1. The intention was a mirror across two different shares, an idea that a 2007 Hackaday project about a RAID of network shares also shows. Today losetup -f --show saves searching for free loop devices.

What has changed since

XOR in software is no longer a problem: at boot the kernel measures which XOR routine is fastest on the CPU and uses SIMD instructions for it. For small servers, hardware RAID controllers are more of a risk, because after a failure you need the same controller. mdadm arrays, in contrast, can be assembled on any Linux machine.
The write hole: if a RAID 5 crashes in the middle of a write, data and parity can diverge. md can close this today, with a journal on a separate disk or with the Partial Parity Log (--consistency-policy=ppl).
GRUB 2 and UEFI: with BIOS you write the boot loader to every disk with grub-install, on Debian also via dpkg-reconfigure grub-pc. With UEFI the boot loader lives on the EFI system partition, which does not simply belong in an md RAID. You create one on every disk and keep them identical.
GPT instead of MBR: sfdisk handles GPT these days too; with sgdisk -R you copy the table and then assign new GUIDs with sgdisk -G.
Check regularly: echo check > /sys/block/md1/md/sync_action reads all blocks and compares the redundancy. This reveals creeping read errors before a rebuild needs those blocks.
Disk names: hda no longer exists; IDE disks are called sdX today too. mdadm does not care, it recognises members by their metadata.

Numbers at a glance

The note “Raid Helper” was in my old Plone blog under /dokumente/raid-helper, last modified on 13 October 2009.
For RAID 5, md has two remedies against the write hole: a journal device and the Partial Parity Log.
The read-auto state switches to normal operation on the first write.
Writing check to /sys/block/mdX/md/sync_action starts a full check of the redundancy.

References

Remarks

I took this post over in 2026 from the archived version of January 2010. Commands and output are verbatim; for GRUB I shortened the success messages of setup.
Odin, Hödur, Tyr: back then my machines were named after Norse gods, a habit that has lasted until today’s firewall frigg.

Original note in the Web Archive