Diagnosing and Recovering a Degraded RAID in Linux
06 Oct 2026, 12:58:10
RAID improves the fault tolerance of a disk subsystem, but the failure of one or more drives can cause the array to enter a degraded state.A degraded RAID means that the array is still operational, but one or more of its components are missing or faulty. Depending on the RAID level, the server may continue operating without data loss. However, the failure of another drive may result in the loss of the entire array.
Therefore, when a degraded RAID is detected, it is important to determine the cause first and only then proceed with recovery.
1. How to determine whether software RAID is being used
The most common software RAID implementation in Linux is Linux Software RAID (mdadm).Check for RAID with:
cat /proc/mdstatExample of a healthy RAID1:Personalities : [raid1]Here:
md0 : active raid1 sda1[0] sdb1[1]
1047552 blocks super 1.2 [2/2] [UU]
[2/2] [UU]means that both drives are working.
Example of a degraded RAID:
md0 : active raid1 sda1[0]Here:
1047552 blocks super 1.2 [2/1] [U_]
[2/1] [U_]means:
- [2/1] — the array is configured for 2 devices, but only 1 is currently active;
- [U_] — the first component is working, while the second is missing.
2. Get detailed RAID information
Use:mdadm --detail /dev/md0For example:/dev/md0:The following parameters are especially important: State, Raid Devices, Active Devices, Working Devices, Failed Devices, Spare Devices.
Version : 1.2
Raid Device : 2
Active Devices : 1
Working Devices : 1
Failed Devices : 1
Spare Devices : 0
State : clean, degraded
| Field | Meaning |
|---|---|
| State | Overall array state: clean (everything is written and the array is healthy), active (writing is in progress), degraded (redundancy has been lost), recovering (recovery is in progress on an added drive), resyncing (synchronization is in progress), FAILED (the array has failed). Flags can be combined, for example clean, degraded. |
| Raid Devices | Number of devices that should be present in the array according to its configuration. For a two-drive RAID1, this is 2. |
| Active Devices | Number of devices currently active in the array and containing up-to-date data. If this is lower than Raid Devices, the array is degraded. |
| Working Devices | Number of devices that are operational (Active plus spare drives). These are working components, including spare devices. |
| Failed Devices | Number of devices marked as failed (faulty) and excluded from operation. |
| Spare Devices | Number of spare drives in the array. They do not store data until needed and are automatically activated to replace a failed drive. |
State : clean, degraded - the array is operational but in a degraded state.
3. Determine which drive has been removed from the RAID
Command:mdadm --detail /dev/md0usually displays a device table:Number Major Minor RaidDevice StateIn this case, /dev/sdb1 is marked as faulty. This means that the component has been removed from the array. However, you should not immediately conclude that the physical drive itself is faulty.
0 8 1 0 active sync /dev/sda1
1 8 17 1 faulty /dev/sdb1
The component may have been removed due to:
- a read/write error;
- a temporary connection problem;
- a RAID controller issue;
- a faulty cable;
- a drive failure;
- a power failure;
- an error within the array itself.
4. Check the condition of the drives
First, check which storage devices are detected by the system:lsblkAdditionally:lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS,MODEL,SERIALFor SATA/SAS drives:smartctl -a /dev/sdaFor NVMe:smartctl -a /dev/nvme0or:nvme smart-log /dev/nvme0Pay particular attention to:- Reallocated Sector Count;
- Current Pending Sector;
- Offline Uncorrectable;
- SMART overall-health;
- read/write errors;
- Media and Data Integrity Errors for NVMe.
5. Check system messages
The reason why a drive was removed from the array can often be found in the kernel log:dmesg -T | grep -iE "error|fail|raid|md|ata|scsi|I/O"Also:journalctl -k -p warning..alertLook for messages such as:I/O errorSuch messages can help determine whether the problem was caused by the drive itself or by the communication path to it, such as a cable, port, or power supply.
ataX: hard resetting link
blk_update_request: I/O error
md/raid1:md0: Disk failure
md0: recovery
6. Monitor RAID status in real time
During RAID recovery, you can monitor the array status with:watch cat /proc/mdstatFor example:md0 : active raid1 sda1[0] sdb1[2]After recovery is complete, you should see:
1047552 blocks [2/1] [U_]
[=====>...............] recovery = 25.4%
[UU]and the array state should change to normal/clean.
7. If the drive is physically healthy
Sometimes a drive is removed from the array due to a temporary error while the drive itself remains fully operational.In this case, first remove its old entry from the RAID:
mdadm /dev/md0 --remove /dev/sdb1Then add it again:mdadm /dev/md0 --add /dev/sdb1A rebuild will then begin.Check the progress with:
cat /proc/mdstatImportant: do not perform --remove and --add blindly. Before doing so, make sure that you are working with the correct RAID array, partition, and drive.8. If the drive is faulty
General replacement procedure:- Make sure the backup is up to date (before starting the rebuild, as it places additional load on the remaining drives).
- Identify the failed drive (array, partition, and serial number).
- Check SMART and system logs.
- If the drive is still listed in the array: use --fail, then --remove.
- Physically replace the drive.
- Make sure the system detects the new drive.
- Copy the partition layout.
- Add the partition to the RAID.
- Wait for the rebuild to complete.
- Install the bootloader on the new drive.
- Check the RAID and system configuration.
mdadm /dev/md0 --fail /dev/sdb1
mdadm /dev/md0 --remove /dev/sdb1For hot-swap servers, the drive can be replaced without shutting down the server if the server and storage subsystem support this. Otherwise, the procedure depends on the hardware.9. Prepare the new drive
After installing the new drive, first make sure Linux detects it:lsblkFor example:sda 1.8TBefore adding the new drive to the RAID, make sure that it has a suitable size and the correct partition layout.
sdb 1.8T
If the RAID was created from partitions, the new drive usually needs to be partitioned in the same way.
For example:
fdisk -l /dev/sdbFor GPT partitioning, you can use:sgdisk -R=/dev/sdb /dev/sdaGPT (sgdisk): with -R=, the target drive is specified first, while the source is the last argument.However, do not run this command without checking the devices: confusing /dev/sda and /dev/sdb can overwrite data.
After copying the partition table, generate new partition GUIDs:
sgdisk -G /dev/sdbMBR (sfdisk):sfdisk -d /dev/sda | sfdisk /dev/sdbThe exact commands depend on the structure of the original array and the disk partitioning scheme.10. Add the new drive to the RAID and restore the bootloader
After preparing the partition:mdadm --add /dev/md0 /dev/sdb1Check:cat /proc/mdstatDuring recovery, you may see:md0 : active raid1 sda1[0] sdb1[2]After completion:
976630336 blocks super 1.2 [2/1] [U_]
[=====>...............] recovery = 28.5%
[UU]The array is fully operational again.
Bootloader. RAID1 protects data, but not the bootloader. If it is not installed on the new drive, the server may fail to boot if the second drive fails.
BIOS/MBR or BIOS/GPT:
grub-install /dev/sdbUEFI: the system partition (ESP) is usually not part of the RAID. Copy its contents to the ESP on the new drive and create a UEFI entry:efibootmgr -c -d /dev/sdb -p 1 -L "Linux (disk 2)" -l '\EFI\\shimx64.efi'The exact commands depend on the distribution and partitioning scheme.11. What is a rebuild?
A rebuild is the process of restoring RAID redundancy.During RAID recovery, the system reads data from the remaining healthy components of the array and writes the required data to the new drive.
During a rebuild:
- disk load increases;
- I/O increases;
- server performance may decrease;
- the process may take a significant amount of time.
Therefore, a long recovery time should not automatically be considered a sign of a problem.
12. Control the recovery speed
For Linux Software RAID, recovery parameters can be checked with:cat /proc/sys/dev/raid/speed_limit_minand:cat /proc/sys/dev/raid/speed_limit_maxFor example:1000These values affect the rebuild/recovery speed range.
200000
The minimum can be temporarily increased if you need to speed up the process:
echo 50000 > /proc/sys/dev/raid/speed_limit_minOn a production server, you should not unnecessarily set the recovery speed too high. Excessive recovery speed can significantly increase the load on the storage subsystem and negatively affect running services. Changes made through /proc are lost after reboot. For persistent values, use sysctl.13. Verify the result
After the rebuild is complete:cat /proc/mdstatThe expected state is:[UU]Also:
mdadm --detail /dev/md0For example:State : cleanThis means that both RAID components are operational.
Active Devices : 2
Working Devices : 2
Failed Devices : 0
14. Check mdadm configuration and initramfs
After recovery, it is useful to make sure that the array configuration has been saved.Depending on the distribution, the information may be located in:
Debian/Ubuntu: /etc/mdadm/mdadm.confCheck:
RHEL/CentOS/Rocky/Alma: /etc/mdadm.conf
cat /etc/mdadm/mdadm.conf
cat /etc/mdadm.confOn some systems, the configuration can be generated or updated using:mdadm --detail --scanCompare the output with the contents of the configuration file. If the array has been added or modified, update the configuration without duplicating ARRAY entries, then rebuild the initramfs:Debian/Ubuntu
update-initramfs -uRHEL-based distributionsdracut -fReplacing a drive normally does not change the array UUID, but initramfs should be updated after any configuration changes.It is important that the system can correctly assemble the RAID after reboot.
15. If the RAID becomes degraded again
If the array removes the drive again after it has been added:[U_]do not repeatedly run:
mdadm --addThe underlying cause must be identified.Check:
dmesg -T | grep -iE "error|fail|raid|md|ata|scsi|I/O"and SMART:smartctl -a /dev/sdXPossible causes include:- a faulty drive;
- a SATA/SAS link problem;
- a faulty backplane;
- a RAID controller problem;
- unstable power;
- errors on a specific port;
- recurring I/O errors.
16. Hardware RAID is a separate case
Not all RAID arrays are managed through mdadm.If the server uses a hardware RAID controller, such as:
- Dell PERC;
- Broadcom/LSI MegaRAID;
- HPE Smart Array;
For some controllers, for example:
storcli /c0 /showor:perccli /c0 /showThe exact command depends on the controller manufacturer and model.For hardware RAID, it is important to distinguish between a RAID logical volume and physical drives.
The operating system may see a single logical disk, while several physical drives are operating inside the RAID controller.
17. What not to do without verification
When working with a degraded RAID, commands that modify the array structure are especially dangerous.Do not blindly execute:
mdadm --create
mdadm --add
mdadm --remove
mdadm --zero-superblockor:mkfson a suspicious device.You should also not rely solely on device names such as /dev/sda, /dev/sdb, or /dev/sdc.
Device names may change after a reboot.
For drive identification, it is better to use:
lsblk -o NAME,SIZE,MODEL,SERIALand verify the serial numbers.Common RAID states
In /proc/mdstat:| State | Meaning |
|---|---|
| [UU] | All RAID1 components are operational |
| [U_] | One of the two components is missing |
| recovery | Recovery is in progress |
| resync | Synchronization is in progress |
| reshape | The array geometry is being changed |
| check | A scheduled array check is in progress |
| State | Meaning |
|---|---|
| clean | The array is in a normal state |
| clean, degraded | The array is operational but has lost redundancy |
| active sync | The device is operational and synchronized |
| faulty | The device is marked as faulty |
| removed | The device has been removed from the array |
| spare | Spare device |
Minimal set of commands for Linux Software RAID
cat /proc/mdstat
mdadm --detail /dev/md0
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS,MODEL,SERIAL
dmesg -T | grep -iE "error|fail|raid|md|ata|scsi|I/O"
smartctl -a /dev/sdXAfter recovery:cat /proc/mdstat
mdadm --detail /dev/md0The ideal result for RAID1 is:[UU]and:
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Conclusion
Diagnosis of a degraded RAID should begin not with immediately replacing the drive, but with determining which component has been removed from the array and why.For Linux Software RAID, the main tools are cat /proc/mdstat, mdadm, lsblk, smartctl, dmesg, and journalctl.
The main rule is simple: do not perform RAID-modifying operations until the array, device, and cause of the failure have been clearly identified. Choosing the wrong drive or using mdadm --create, mkfs, or --zero-superblock incorrectly can result in data loss.