Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Why is that happening?

Slow performance I could understand (though RAID rebuild is usually sequential, not random), but how does SMR cause these "dropouts"?



Hardware RAID typically requires a disk to respond within a certain number of ms, either reporting a successful write or a failure to write. SMR, because it requires a whole section of the drive to be re-written everytime you need to modify a single block of that section, may have very slow response times, and result in timeouts. On a timeout, the hardware RAID controller assumes the disks is offline and drops the disk from the array.

The same happened with WD Green drives, which are not graded for RAID. Their error correction logic typically allows for a lot more attempts to read the data than a datacentre drive, resulting in timeouts and the drive dropping out of the array (which is why it is a very bad idea to put a consumer drive in a hardware RAID array).

Now WD Red NAS are meant for RAID arrays but I suspect WD assumed they would be used for software RAID only, which typically doesn't have those timeouts. But if so, it should be clearly stated.


Isn't that behaviour of Greens only a problem when the drive goes bad? In which case you want it to get kicked out of the array. Or do you mean this will happen to a new Green drive with too high a probability?


The problem is that taking some time to read the data doesn't mean the drive has gone bad, at worst perhaps a sector has gone bad, and even then, it might even still be readable, just a bit slow. And apparently it is very common on large capacity green disks after a few months.

The typical bad scenario (and I have been burned by this) is that let's say you use RAID5, and one sector goes bad. While the disk tries to read it, the RAID controller kicks the disk out of the array because of the timeout. Now you need to replace the disk and rebuild the array. In the rebuilt, as you are doing a full read or all disks, you are pretty likely to find another sector that is slow to read on one of the other disks (particularly on multi-TB disks). And then the controller will kick another disk from the array. And now you lost everything.

Also why data scrubbing is pretty important in NAS.


Thanks for explanation. That was surely a bad experience. Luckily there are ways to overcome this incompatibility, such as increasing the timeout value for a drive in the OS, see the Linux wiki

https://raid.wiki.kernel.org/index.php/Timeout_Mismatch


They have long (deep) error recoveries, and most of them don't have configurable SCT ERC. If used on Linux, where the default command (queue) timer is 30 seconds, a bad sector could result in the drive not reporting success or failure well beyond 30 seconds at which point the kernel does a link reset thinking the drive is unresponsive. On SATA drives, resetting the drive clears the whole queue, all commands are lost, so it's not clear what sector has the problem. This prevents whatever healing properties RAID has. e.g. on md RAID (includes mdadm and LVM managed RAIDs these days) an explicit read error by the drive comes with a sector LBA value, and md will know what stripe its a member of (or its mirror) and will overwrite the bad sector with reconstructed data, fixing it. But if there are either write errors or a bunch of link resets, md will consider the drive faulty (kicking it).

https://raid.wiki.kernel.org/index.php/Timeout_Mismatch

It's important to know this kernel command timer and SCT ERC mismatch is (a) common and (b) affects mdadm, LVM, Btrfs and maybe ZFS RAID on Linux. I'm not really sure whether the kernel command timer applies to ZoL or if a vdev has its own policy.


Thanks for explaining. The kernel wiki describes an easy solution to this.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: