Blog — Data Recovery and Cybersecurity
Technical analyses, prevention tips and data recovery news from the experts at SOS Data Recovery, Swiss laboratory since 2006.
RAID or NAS Failure: The Mistakes That Turn a Fault Into Permanent Loss
When a RAID, a NAS or a server fails, the right reflex is to switch everything off and rebuild nothing: in most cases, it is not the initial failure that loses the data, but the repair attempt that follows: a rebuild launched on an already fragile system, a RAID deleted then recreated, drives handled again and again. A degraded RAID can often be recovered in full, provided the situation is not made worse. Concretely: shut the system down, do not attempt any reconstruction, and hand only the hard drives to a laboratory (the server is of no use).
What follows explains why these actions matter, and where the traps lie, with two real-life stories that nearly swept everything away.
RAID in three minutes
A RAID (Redundant Array of Independent Disks) is an assembly of several drives seen by the system as a single volume, designed to provide redundancy: the ability to survive the failure of one or more drives. You still need to know which redundancy is being discussed, because not all levels are equal.
The mirror RAID (RAID 1) automatically duplicates the contents of one drive onto another. It tolerates the loss of one drive of the pair, but is expensive in capacity: half the total volume serves for the copy.
Striping (RAID 0) does the opposite. There is no redundancy, and the word "RAID" is here almost a misuse. The file is cut into small pieces spread across several drives written simultaneously. The splitting is done at the software level, very quickly, and writing in parallel gains a great deal of speed. The downside is brutal: a single drive failing, and everything is lost. On three drives in RAID 0, the risk of loss is considerable.
RAID 5 and RAID 6 seek the balance between speed and safety. The file is spread as in striping, but a parity (an XOR) is also calculated and placed on a drive according to a cycle, so that it is never always the same drive that receives it. This calculation makes writing a little slower than pure striping. In return, RAID 5 tolerates the failure of one drive (and sacrifices the equivalent of one drive in volume); RAID 6 tolerates two drives (and sacrifices two drives in volume). At least three drives are needed for a RAID 5.
RAID 0+1 or 1+0 combines mirroring and striping, one way or the other depending on the build, to combine speed and redundancy.
| Level | Redundancy | Min. drives | Fault tolerance | Capacity cost |
|---|---|---|---|---|
| RAID 0 (striping) | None | 2 | 0 drives (total loss if 1 fails) | 0 % |
| RAID 1 (mirror) | Duplication | 2 | 1 drive of the pair | 50 % |
| RAID 5 | Parity | 3 | 1 drive | 1 drive |
| RAID 6 | Double parity | 4 | 2 drives | 2 drives |
| RAID 1+0 | Mirror + striping | 4 | 1 drive per mirror | 50 % |
One point deserves to be stated from the outset, because it is the source of many dramas: a RAID is not a backup. It protects against the hardware failure of a drive, nothing more. It will save you neither from an accidental deletion, nor from ransomware, nor from a fire, nor from a configuration error. For that, you need a genuine backup strategy, separate from the RAID.
How a RAID goes down
The failures we see at the laboratory follow a few recurring scenarios.
The loss of the configuration. A RAID exists only thanks to its configuration: the information that describes how the drives assemble into a volume. This configuration is found either on a physical controller card or written on the drives themselves, generally in several copies. If the controller card is faulty, the virtual RAID disappears. And even when the configuration is replicated on the drives, a problem can spread to all the copies and cause the assembly to be lost.
Degraded mode. When a drive fails, the system detects it and switches to degraded mode: nothing more is stored on the failing drive, and the array keeps running with one drive short. This is the RAID's safety net, but a single-use net. On a RAID 5 in degraded mode, if a second drive fails, all the data is lost. Hence the importance of reacting quickly, provided the system's alerts have indeed been configured and received. Many RAIDs run in degraded mode for weeks without anyone knowing.
The rebuild: the trap almost everyone underestimates
A rebuild is the automatic reconstruction of the contents of a replaced drive, from the data and the parity present on the other drives of the RAID. On paper, it is the normal procedure. In practice, it is the most dangerous moment in the whole life of a RAID.
Here is why. The drives of the same server are almost always identical: same capacity, same model, out of the same factory in the same period, from the same manufacturer. They have then run exactly the same number of hours, in the same conditions. They therefore have, more or less, the same age and the same fatigue.
Now a rebuild is not an innocuous operation: it reads every sector, from the first to the last, of all the remaining drives. It is the most intense strain a drive can undergo. And it happens very regularly that a second drive, as tired as the one that has just failed, chooses precisely this moment to give way. On a RAID 5, the system then finds itself unable to reconstruct anything: two missing drives, no sufficient parity, data lost.
This is the cruel paradox of the RAID: the procedure meant to repair the failure is also the one that turns it into a catastrophe.
The mistakes that kill
Beyond the rebuild, certain human reactions almost always make the situation worse.
Deleting and recreating the RAID. This is the most destructive mistake. Faced with a system that no longer mounts, one thinks one is doing the right thing by deleting the RAID configuration to recreate a new one. But if you do not know exactly how the RAID had been built (the precise order of the drives, the block size), you write a false structure over the existing structure, and you blur the trails that would have allowed everything to be reconstructed.
We found ourselves one day facing a textbook case. An IT specialist managed a RAID system of three drives, in a server that contained four or five. Following a problem, he erased everything and rebuilt a RAID using all five drives at once. The result was a real puzzle, but through sheer work, we were nonetheless able to recover the data.
Multiplying the handling. The general principle to remember: the more you work with the RAID management software, the more restores, rebuilds and command-line commands you launch, the more risk you take. Each operation spins drives that are perhaps already damaged, and each additional rotation of a dying drive may be the last.
The right action, and what a laboratory does in your place
The right reaction is therefore counterintuitive: do nothing more. Switch off the system, and bring the hard drives to a laboratory, only the drives, the server or the NAS are not necessary.
What we do then is the opposite of a live rebuild. The first step is a bit-by-bit copy of each drive: we freeze the state of each one, like a photograph, and we protect the original units. From there, we never again work on the original drives, but on their copies.
Then comes the virtual reconstruction of the RAID. By analysing the copies, we determine the exact order of the drives, the block size, the parity scheme, and we reassemble the volume on the fly to extract the data from it. The original drives, meanwhile, no longer spin: they are safe.
An important clarification: with us, none of this carries the risk of destruction of a classic rebuild. Our bit-by-bit copies are themselves kept read-only, we never relaunch the RAID cards' software nor any live reconstruction, and we reassemble the RAID virtually, on the fly. Our process writes nothing onto your data. The only real limit is the physical state of the drives: what is genuinely unreadable on a platter stays so, but never because of the handling.
For very large volumes (several terabytes, sometimes 60, 70 or 80 TB spread over sixteen drives or more), the approach is the same on a larger scale: bit-by-bit safety copies of all the drives, then extraction of the data to a new NAS that the client provides. Once the operation is finished, the client leaves with their new NAS, data already restructured as it was, ready to use.
Two pieces of advice that go against the grain
Our experience has led us to recommend two things that common logic advises against.
Mix the brands of your drives. Everyone recommends identical drives (same model, same product number) for a homogeneous system running at the same speed. That is right in terms of performance. But it is also the guarantee that all your drives will have the same age, the same wear and the same susceptibility to failure, hence the cascade failures during a rebuild. We instead recommend drives with identical or very similar performance, but of different brands or series. On a RAID 1+0, for example, one can build the RAID 0 with one brand and its mirror with another: the day a series defect strikes one brand, the other holds firm.
One story illustrates this risk perfectly. We were called in for a NAS entirely fitted with SSDs. All these SSDs shared a firmware defect: an internal value, in memory, exceeded a threshold after a certain running time, on the order of two to three years. Past this threshold, the firmware bugged out, the SSD refused to initialise and was no longer detected. The client, on a RAID 5, ran into a first problem, switched off the server, restarted, inserted a new SSD, restarted again. But this restart relaunched all the SSDs at the same time, which all reached the fateful value simultaneously. All at once, no drive worked anymore. With different brands, this common defect would not have struck all the drives at the same instant.
Steer clear of proprietary hybrid RAIDs. Some brands offer in-house hybrid RAID systems, which for example create virtual volumes from several elements assembled in a manner specific to the manufacturer. On paper, it is convenient. In the event of a failure, a bug or a corruption of the configuration, it is a nightmare: these assemblies are hard to understand when nothing has been documented or recorded by the technicians who set them up. A standard, well-documented RAID is always more recoverable than an opaque proprietary system.
Frequently asked questions
What to do when a RAID or a NAS fails?
Switch the system off immediately and do not attempt any reconstruction. Do not erase the RAID, do not launch a rebuild, do not reinsert a drive. Remove the hard drives (noting their order if possible) and hand them to a specialised laboratory: it is the handling, not the failure, that most often loses the data.
Why is a RAID rebuild dangerous?
Because it reads every sector of all the remaining drives, which have the same age and the same wear as the one that has just failed. This intense strain often makes a second drive fail during the operation. On a RAID 5, the loss of a second drive during the rebuild leads to the total loss of the data.
Is a RAID a backup?
No. A RAID protects only against the hardware failure of one or two drives depending on the level. It protects neither against an accidental deletion, nor against ransomware, nor against a fire, nor against a configuration error. A genuine backup, separate and ideally off-site, remains indispensable.
Can the data of a RAID 5 whose two drives have failed be recovered?
Often, yes, if the situation has not been made worse. In the laboratory, a bit-by-bit copy of each drive is made, then the RAID is reconstructed virtually from the copies to extract the data. The original drives are no longer strained, which protects what can still be protected.
Should the complete server be brought to the laboratory?
No, the hard drives are enough. The configuration is reconstructed from the drives themselves. Bringing only the drives simplifies the logistics and avoids handling the equipment needlessly.
Should identical drives be used in a RAID?
That is the common practice, but it has a flaw: identical drives age and weaken at the same rate, hence the cascade failures during a rebuild. We instead recommend drives with similar performance but of different brands or series, so that a common defect does not strike them all at the same time.