"An ongoing study on datasets of several Petabytes have shown that there can be 'silent data corruption' at rates much larger than one might naively expect from the expected error rates in RAID arrays and the expected probability of single bit uncorrected errors in hard disks," began a recent query on the Linux kernel mailing list asking where the errors might be introduced. Alan Cox replied, "its almost entirely device specific at every level." He then continued on with some general information, tracing the path of the data from the drive, through the cable and bus, into main memory and the CPU cache, as well as over the network, "once its crossing the PCI bus and main memory and CPU cache its entirely down to the system you are running what is protected and how much. Note that a lot of systems won't report ECC errors unless you ask." Alan continued:

"The next usual mess is network transfers. The TCP checksum strength is questionable for such workloads but the ethernet one is pretty good. Unfortunately lots of high performance people use checksum offload which removes much of the end to end protection and leads to problems with iffy cards and the like. This is well studied and known to be very problematic but in the market speed sells not correctness."

Regarding the specific study in question, Alan noted, "for drivers/ide there are *lots* of problems with error handling so that might be implicated (would want to do old [versus] new ide tests on the same h/w which would be very intriguing)."

Alan Cox posted an updated LibATA PATA (IDE) status report on the lkml. Improved from a previous status report [story] he noted, "current kernels now support HPA (Host Protected Area) but default to honouring it. Probably a wrong default for PATA but we need to decide the right way to expose it nicely." He went on to note, "no PATA hotplug support yet. Need warmplug helpers for some chipsets (eg some intel ICH) to avoid risk of hangs."

Later in the report he listed around 40 chipsets describing LibATA's PATA support for each, ranging from "rock solid" for ATIIXP and "solid" for AMD, TRIFLEX, MARVELL, MPIIX, OLDPIIX, NETCELL, RZ1000, SERVERWORKS, SIL680, and VIA, to "still experimental" for NS87410 and "no idea" for IT8213. When asked about support for PowerPC drivers, Alan replied, "I'm not aware of anyone having done any PPC ports yet, although a couple of people have asked and said they would look at it. Currently our coverage is incomplete for some embedded and obscure platforms, of which the macintrash is probably the least 'obscure'."