From: Ingo Molnar <mingo@...>
Subject: Re: [patch] scsi: fix crash in gdth_timeout()
 [1]Date: Oct 15, 3:27 pm 2007

* Boaz Harrosh <bharrosh@panasas.com> wrote:

> > Honestly, the driver is probably rarely run by people that lack the 
> > hardware, I bet...
> 
> It was all "flight by instruments only". I called for HW testers and 
> none came forward. All these changes, apart from "successful downgrade 
> to messy" where also needed in order to push important changes to 
> scsi.
> 
> But a little bird said that QEMU might simulate this HW. SO I guess it 
> is QEMU time for me.

heh. Incidentally i was thinking about using KVM for automated testing. 
Important pieces of hardware should get an in-KVM simulator/emulator, 
that way developers who do not own that hardware can do functionality 
testing too. So basically the highest-quality drivers would have an 
"inverse driver" in KVM, which simulates the hardware. (that model is 
evidently useful to the hardware maker even for new hardware: it can 
then also be used to test the Linux compatibility and Linux performance 
of future planned releases of the hardware, etc.)

	Ingo
-


From: Linus Torvalds <torvalds@...>
Subject: Re: [patch] scsi: fix crash in gdth_timeout()
 [1]Date: Oct 15, 3:38 pm 2007

On Mon, 15 Oct 2007, Ingo Molnar wrote:
> 
> heh. Incidentally i was thinking about using KVM for automated testing. 

Using emulators to test device drivers is almost certain to be pointless.

The problem with device drivers tends to be timing issues, odd hardware 
interactions, and lots of strange (and sometimes undocumented) behaviour 
and dependencies (eg things like "you have to wait 50us after setting the 
reset bit until the hardware has actually reset").

These are all things that you'd generally not catch in emulation - because 
the emulation by necessity is only going to be a very weak picture of the 
real thing.

So I suspect you can find the easy stuff, but only by writing insanely 
complex device model descriptions in the emulator environment itself, and 
only for those device models that have actually been written.

Can it be donein theory? Sure. Practically? Not so sure. Would it help? I 
suspect the effort to do the device model would be many times bigger than 
*any* conceivable effort to just fix the driver bugs as they get found 
through other means.

So we could perhaps have *really* good emulated hardware for a few models 
of hw out there, but likely they'd be fewer and less varied platforms than 
most kernel developers end up having hidden under their desk anyway.. 

			Linus
-


From: Alan Cox <alan@...>
Subject: Re: [patch] scsi: fix crash in gdth_timeout()
 [1]Date: Oct 15, 4:08 pm 2007

On Mon, 15 Oct 2007 12:38:06 -0700 (PDT)
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> 
> 
> On Mon, 15 Oct 2007, Ingo Molnar wrote:
> > 
> > heh. Incidentally i was thinking about using KVM for automated testing. 
> 
> Using emulators to test device drivers is almost certain to be pointless.

For some things. I do it a bit because you can use it to fake
failures that are tricky to do in the real world. It won't tell you the
driver works but its suprisingly good for testing for races (forcing IRQ
delivery at specific points), buggy hardware you don't posess, and things
like media failures and timeouts your real hardware refuses to do.

Alan
-


From: Linus Torvalds <torvalds@...>
Subject: Re: [patch] scsi: fix crash in gdth_timeout()
 [1]Date: Oct 15, 4:15 pm 2007

On Mon, 15 Oct 2007, Alan Cox wrote:
> 
> For some things. I do it a bit because you can use it to fake
> failures that are tricky to do in the real world. It won't tell you the
> driver works but its suprisingly good for testing for races (forcing IRQ
> delivery at specific points), buggy hardware you don't posess, and things
> like media failures and timeouts your real hardware refuses to do.

Heh. I do agree that you likely find bugs, even if quite often it's 
exactly because the behaviour is something that will never happen on real 
hardware.

But failure testing is very useful - I forget who it was who debugged some 
driver by taking a CD and just scrathing it mercilessly to induce read 
errors ;)

Having a really *bad* HW emulator can certainly work that way too, even if 
it also would probably end up hitting just a few of the potential error 
paths..

			Linus
-

From: Ingo Molnar <mingo@...>
Subject: Re: [patch] scsi: fix crash in gdth_timeout()
 [1]Date: Oct 15, 6:23 pm 2007

* Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Mon, 15 Oct 2007, Ingo Molnar wrote:
> > 
> > heh. Incidentally i was thinking about using KVM for automated 
> > testing.
> 
> Using emulators to test device drivers is almost certain to be 
> pointless.

something like that wont enable 100% coverage (or even reasonable 
coverage for most hardware), so it's no replacement for actual hard 
testing, but it could push out the domain of minimally tested code quite 
a bit and increase the quality of the kernel. Races are always tough and 
so are bugs on the side of the hardware, but it's the silly boot-time 
crash showstoppers and "device does not work anymore" mistakes that 
causes us to lose most of the testers and early adopters.

I'm not really worried about the 1% of bugs that are tough to find and 
fix (because they are actually fun to find and fix), i'm worried about 
the 99% easy and boring bugs - because they annoy users just as much as 
the exciting bugs do. If we fix them faster then there's more time (and 
more tester stamina) left for the harder to find bugs.

so i think adding redundancy in form of a simplified hw emulator can 
certainly not hurt and fundamentally increases robustness - and it will 
definitely reduce the chance for a whole host of stupid bugs that are 
not in the hardware but are in the ~4 million lines of Linux driver 
codebase. Limits and scalability would also be testable: "if i put 32 of 
these networking cards into a Linux box, will the Linux driver blow up".

not that i think this is realistic for any significant portion of the 
hardware currently - unless some hw maker starts doing it. But KVM will 
have a good portion of the core PC platform emulated (APIC, etc.) - and 
that's a nice step forward already.

	Ingo
-