"An ongoing study on datasets of several Petabytes have shown that there can be 'silent data corruption' at rates much larger than one might naively expect from the expected error rates in RAID arrays and the expected probability of single bit uncorrected errors in hard disks," began a recent query on the Linux kernel mailing list asking where the errors might be introduced. Alan Cox replied, "its almost entirely device specific at every level." He then continued on with some general information, tracing the path of the data from the drive, through the cable and bus, into main memory and the CPU cache, as well as over the network, "once its crossing the PCI bus and main memory and CPU cache its entirely down to the system you are running what is protected and how much. Note that a lot of systems won't report ECC errors unless you ask." Alan continued:

"The next usual mess is network transfers. The TCP checksum strength is questionable for such workloads but the ethernet one is pretty good. Unfortunately lots of high performance people use checksum offload which removes much of the end to end protection and leads to problems with iffy cards and the like. This is well studied and known to be very problematic but in the market speed sells not correctness."

Regarding the specific study in question, Alan noted, "for drivers/ide there are *lots* of problems with error handling so that might be implicated (would want to do old [versus] new ide tests on the same h/w which would be very intriguing)."

From:	Bruce Allen [email blocked]
Subject: ECC and DMA to/from disk controllers
Date:	Mon, 10 Sep 2007 07:19:35 -0500 (CDT)

Dear LKML,

Apologies in advance for potential mis-use of LKML, but I don't know where 
else to ask.

An ongoing study on datasets of several Petabytes have shown that there 
can be 'silent data corruption' at rates much larger than one might 
naively expect from the expected error rates in RAID arrays and the 
expected probability of single bit uncorrected errors in hard disks.

The origin of this data corruption is still unknown.  See for example 
http://cern.ch/Peter.Kelemen/talk/2007/kelemen-2007-C5-Silent_Corruptions.pdf

In thinking about this, I began to wonder about the following.  Suppose 
that a (possibly RAID) disk controller correctly reads data from disk and 
has correct data in the controller memory and buffers.  However when that 
data is DMA'd into system memory some errors occur (cosmic rays, 
electrical noise, etc).  Am I correct that these errors would NOT be 
detected, even on a 'reliable' server with ECC memory?  In other words the 
ECC bits would be calculated in server memory based on incorrect data from 
the disk.

The alternative is that disk controllers (or at least ones that are meant 
to be reliable) DMA both the data AND the ECC byte into system memory. 
So that if an error occurs in this transfer, then it would most likely be 
picked up and corrected by the ECC mechanism.  But I don't think that 
'this is how it works'.  Could someone knowledgable please confirm or 
contradict?

Cheers,
 	Bruce



From: Alan Cox [email blocked]
Subject: Re: ECC and DMA to/from disk controllers
Date:	Mon, 10 Sep 2007 14:54:15 +0100
 Windsor, Berkshire, SL4 1TE, Y Deyrnas Gyfunol. Cofrestrwyd yng Nghymru a
 Lloegr o'r rhif cofrestru 3798903

> In thinking about this, I began to wonder about the following.  Suppose 
> that a (possibly RAID) disk controller correctly reads data from disk and 
> has correct data in the controller memory and buffers.  However when that 
> data is DMA'd into system memory some errors occur (cosmic rays, 
> electrical noise, etc).  Am I correct that these errors would NOT be 
> detected, even on a 'reliable' server with ECC memory?  In other words the 
> ECC bits would be calculated in server memory based on incorrect data from 
> the disk.

Architecture specific.

> The alternative is that disk controllers (or at least ones that are meant 
> to be reliable) DMA both the data AND the ECC byte into system memory. 
> So that if an error occurs in this transfer, then it would most likely be 
> picked up and corrected by the ECC mechanism.  But I don't think that 
> 'this is how it works'.  Could someone knowledgable please confirm or 
> contradict?

Its almost entirely device specific at every level. Some general
information and comment however

- Drives normally do error correction and shouldn't be fooled very often
by bad bits. 
- The ECC level on the drive processors and memory cache vary by vendor.
Good luck getting any information on this although maybe if you are Cern
sized they will talk

After the drive we cross the cable. For SATA this is pretty good, and
UDMA data transfer is CRC protected. For PATA the data is but not the
command block so on PATA there is a minute chance you send the CRC
protected block to the wrong place

Once its crossing the PCI bus and main memory and CPU cache its entirely
down to the system you are running what is protected and how much. Note
that a lot of systems won't report ECC errors unless you ask.

If you have hardware RAID controllers its all vendor specific including
CPU cache etc on the card etc.

The next usual mess is network transfers. The TCP checksum strength is
questionable for such workloads but the ethernet one is pretty good.
Unfortunately lots of high performance people use checksum offload which
removes much of the end to end protection and leads to problems with iffy
cards and the like. This is well studied and known to be very problematic
but in the market speed sells not correctness. 

From the paper type II sounds like slab might be a candidate kernel side
but also CPU bugs as near OOM we will be paging hard and any L2 cache page
out/page table race from software or hardware would fit what it describes,
especially the transient nature

Type III wrong block on PATA fits with the fact the block number isn't
protected and also the limits on the cache quality of drives/drive
firmware bugs. 

For drivers/ide there are *lots* of problems with error handling so that
might be implicated (would want to do old v new ide tests on the same h/w
which would be very intriguing).

Stale data from disk cache I've seen reported, also offsets from FIFO
hardware bugs (The LOTR render farm hit the latter and had to avoid UDMA
to avoid a hardware bug)

Chunks of zero sounds like caches again, would be interesting to know
what hardware changes occurred at the point they began to pop up and what
software.

We also see chipset bugs under high contention some of which are
explained and worked around (VIA ones in the past), others we see are
clear correlations - eg between Nvidia chipsets and Silicon Image SATA
controllers.

• Archive of above thread
• KernelTrap interview with Alan Cox