"It looks to be about 2.1% increase in time to do the make/mount/unmount operations with the marker patches in place and no blktrace operations," Alan Brunelle summarized some benchmarks testing the overhead of the kernel markers patches. He continued, "with the blktrace operations in place we see about a 3.8% decrease in time to do the same ops." Block layer maintainer Jens Axboe responded favorably, "thanks for running these numbers. I don't think you have to bother with it more. My main concern was a performance regression, increasing the overhead of running blktrace." He added, "I'd say the above is Good Enough for me," acking the kernel marker patches.

Jens went on to muse, "I do wonder about that performance _increase_ with blktrace enabled. I remember that we have seen and discussed something like this before, it's still a puzzle to me..." Mathieu Desnoyers agreed, "interesting question indeed," going on to suggest possible future tests to understand the unexpected performance increase. blktrace is a block layer IO tracing tool for providing detailed information about request queue operations, originally developed by Jens Axboe and merged into the mainline kernel in 2.6.17-rc1.

Mathieu Desnoyers posted an updated version of his Linux Kernel Markers patchset explaining, "following Christoph Hellwig's suggestion, aiming at a Linux Kernel Markers inclusion for 2.6.24, I made a simplified version of the Linux Kernel Markers. There are no more dependencies on any other patchset." He continued, "the modification only involved turning the immediate values into static variables and adapting the documentation accordingly. It will have a little more data cache impact when disabled than the version based on the immediate values, but it is far less complex." The patch includes documentation which explains:

"A marker placed in code provides a hook to call a function (probe) that you can provide at runtime. A marker can be 'on' (a probe is connected to it) or 'off' (no probe is attached). When a marker is 'off' it has no effect, except for adding a tiny time penalty (checking a condition for a branch) and space penalty (adding a few bytes for the function call at the end of the instrumented function and adds a data structure in a separate section). When a marker is 'on', the function you provide is called each time the marker is executed, in the execution context of the caller. When the function provided ends its execution, it returns to the caller (continuing from the marker site)."

In a series of ten patches, Mathieu Desnoyers posted an updated version of the Linux Kernel Markers. In the first patch he explains the need for markers:

"With the increasing complexity of today's user-space application and the wide deployment of SMP systems, the users need an increasing understanding of the behavior and performance of a system across multiple processes/different execution contexts/multiple CPUs. In applications such as large clusters (Google, IBM), video acquisition (Autodesk), embedded real-time systems (Wind River, Monta Vista, Sony) or sysadmin/programmer-type tasks (SystemTAP from Redhat), a tool that permits tracing of kernel-user space interaction becomes necessary."

Mathieu goes on to explain that in complex system finding bugs can be even more difficult due to the rarity of their occurance, "one can therefore only hope at having the best conditions to statistically reproduce the bug while extracting information from the system. Some bugs have been successfully found at Google using their ktrace tracer only because they could enable it on production machines and therefore recreate the same context where the bug happened." He then added, "therefore, it makes sense to offer an instrumentation set of the most relevant events occurring in the Linux that can have the smallest performance cost possible when not active while not requiring a reboot of a production system to activate. This is essentially what the markers are providing."