"After posting some benchmarks involving cfs, I got some feedback, so I decided to do a follow-up that'll hopefully fill in the gaps many people wanted to see filled," Rob Hussey began. He added, "this time around I've done the benchmarks against 2.6.21, 2.6.22-ck1, and 2.6.23-rc6-cfs-devel (latest git as of 12 hours ago)." Rob briefly summarized, "the only analysis I'll offer is that both sd and cfs are improvements, and I'm glad that there is a lot of work being done in this area of linux development. Much respect to Con Kolivas, Ingo Molnar, and Roman Zippel, as well all the others who have contributed."

Referring to a chart in which the blue line represented the CFS process scheduler, and the green line represented the SD "staircase" process scheduler, Ingo Molnar noted, "heh - am i the only one impressed by the consistency of the blue line in this graph? :-) [ and the green line looks a bit like a .. staircase? ]" He acknowledged some slowdown in CFS compared to SD in one of the benchmarks, "-ck1 is 0.8% faster in this particular test." Ingo then explained, "many things happened between 2.6.22-ck1 and 2.6.23-cfs-devel that could affect performance of this test. My initial guess would be sched_clock() overhead." In further testing he applied a low-res-sched-clock that resulted in better performance for CFS leading him to conclude, "the performance difference between -ck and -cfs-devel seems to be mostly down to the more precise (but slower) sched_clock() introduced in v2.6.23 and to the startup penalty of freshly created tasks." When asked if the low-res-sched-clock was likely to be merged, Ingo replied:

"I don't think so - we want precise/accurate scheduling before performance. (otherwise tasks working off the timer tick could steal away cycles without being accounted for them fairly, and could starve out all other tasks.) Unless the difference was really huge in real life - but it isn't."

Greg KH and Chris Wright have been maintaining a -stable 2.6.x.y patchset for the 2.6.x and 2.6.(x-1) kernels since March of 2005. Thus, with the current stable release being 2.6.22, they maintain -stable patches for 2.6.22 and 2.6.21. 2.4 stable kernel maintainer Willy Tarreau noted the currently high patch rate in each of the 2.6 -stable trees and decided to maintain -stable patches against the 2.6.20 tree until things calm down. Adrian Bunk also continues to maintain a -stable 2.6.16 branch of the Linux kernel. Willy explained about his new 2.6.20 -stable patches:

"I proposed Chris and Greg to continue issuing a few more 2.6.20 releases during the time needed for 2.6.21 and 2.6.22 to show a significant drop in their patch rates, which hopefully will be just a matter of a few releases.

"My goal is *not* to do all the hard work they do, but just to backport from their patches those which are meaningful for 2.6.20. For this reason, 2.6.20 releases will always be slightly late and should not contain patches not merged in more recent releases."

Following up to a bug report against the 2.6.22 kernel, Andrew Morton and Linus Torvalds offered some tips on how to debug kernel problems. Andrew first pointed to netconsole.txt for instructions on setting up a netconsole, "when the machine has stalled, see if you can get a task trace with ALT-SYSRQ-t. This will require CONFIG_MAGIC_SYSRQ=y and possibly setting ignore_loglevel on the kernel boot command line."

Linus Torvalds suggested "git bisect" as an alternative, "[it] will take some time, but is really a lot easier" He explains, "there's almost 7000 commits in between 2.6.21 and 22, but that still means that in about fourteen recompiles/reboots, "git bisect" should tell us where your problem starts, which will hopefully make it obvious what the problem is (or at least pinpoint it a *lot*)." He goes on to detail how to install git, obtain the latest kernel, and run "git bisect", "doing a git bisect isn't really that hard, but fourteen compiles/reboots will take some time (well, the compiles will, the reboots aren't that bad). But even if you're not a git user, it really is very simple". Specifically, he notes, "start the 'git bisect' with 'git bisect good v2.6.21', 'git bisect bad v2.6.22', and it will pick a kernel version about half-way between the two points, and you can now start testing. For each kernel you try, if it boots fine, do 'git bisect good', otherwise boot into a working kernel, and then do 'git bisect bad'. Git will then pick the next 'halfway' kernel for that case."

Included in Andrew Morton's potential 2.6.23 merge list [story] were a series of patches to make the x86-64 architecture tickless. Andi Kleen, the x86-64 maintainer replied, "I'm sceptical about the dynticks code. It just rips out the x86-64 timing code completely, which needs a lot more review and testing. Probably not .23." Linus Torvalds agreed, "we are *not* going to do another 'rip everything out, and replace it with new code' again. Over my dead body. We're going to do this thing gradually, or not at all." He went on to explain "the patch-set itself actually looks fine, as far as I'm concerned. But it does seem to have that 'enable everything in one go' problem. I'd much rather see one time source at a time being converted, and enabled then and there, so that when people report problems and do a bisection, if it was HPET that broke, you get the commit that changed HPET."

In response to the pains caused by the original dyntick merge in 2.6.21, Ingo Molnar acknowledged, "we had 12 -hrt/dynticks merge related regressions between 2.6.21-rc1 and -final, and 4 after final." He went on to point out, "it's all pretty quiet today on the dynticks regressions front. (there are no open regressions in either the upstream i386 code or in the devel patches we are aware of)." As to the source of the bugs, he explained, "the majority of the above bugs were in the infrastructure code. (the worst was the generic resume/suspend one fixed in 2.6.21.2) And sadly, a fair number of the infrastructure bugs we introduced during the frentic clockevents/dynticks rewrites/redesigns we did between .20 and .21. That was a royally stupid mistake for us to do - instead of patiently waiting for the bugs to be shaken out we destabilized the infrastructure. (it was a 'lets make this thing so nice that it's impossible to reject' instintic gut reaction.)" Linus replied, "one thing I'll happily talk about is that while 2.6.21 was painful, you and Thomas in particular were both very responsible about the thing, so no, I'm not at all complaining or worried about it in that sense! I just really _really_ wish we could have two fairly stable releases in a row. I think 2.6.22 has the potential to be a pretty good setup, and I'd really like to avoid having another 2.6.21 immediately afterwards."

Linux creator Linus Torvalds announced the official release of the 2.6.22 kernel, "it's out there now (or at least in the process of mirroring out - if you don't see everything, give it a bit of time)." He summarized the changes since 2.6.22-rc7 [story]:

"Not a whole lot of changes since -rc7: some small architecture changes (ppc, mips, blackfin), and most of those are defconfig updates. Various driver fixes: new PCI ID's along with some ide, ata and networking fixes (for example - the magic wireless libertas ioctl's got removed, they may be re-added later, hopefully in a more generic form, but in the meantime this doesn't make a release with new interfaces that aren't universally liked)."

The previous stable kernel, 2.6.21, was released a little over two months ago on April 25'th [story]. An overview of all the changes merged into the latest version of the kernel is maintained in the Kernel Newbies wiki. Included in the list of changes are the SLUB allocator which replaced the slab allocator, a new wireless stack, a new firewire stack [story], and support for the Blackfin architecture. Source level changes can be tracked via the gitweb interface to Linus' kernel tree.

Linux creator Linus Torvalds released the 2.6.22-rc7 kernel saying, "it's hopefully (almost certainly) the last -rc before the final 2.6.22 release, and we should be in pretty good shape. The flow of patches has really slowed down and the regression list has shrunk a lot." He briefly summarized the changes in this latest release candidate, "the patches are mostly trivial fixes, a few new device ID's, and the appended shortlog really does pretty much explain it," adding, "final testing always appreciated, of course".

The previous stable kernel, 2.6.21, was released a little over two months ago on April 25'th [story]. An overview of all the changes merged into the latest version of the kernel is maintained in the Kernel Newbies wiki. Included in the list of changes are the SLUB allocator which replaced the slab allocator, a new wireless stack, a new firewire stack [story], and support for the Blackfin architecture. Source level changes can be tracked via the gitweb interface to Linus' kernel tree.

Con Kolivas [interview] continues to maintain the performance oriented -ck patchset that he started in early 2004 [story], "this patchset is designed to improve system responsiveness and interactivity. It is configurable to any workload but the default -ck patch is aimed at the desktop and -cks is available with more emphasis on serverspace." In Con's latest release, 2.6.21-ck1, he notes that he has updated the patchset to include his improved SD cpu scheduler [story], "the staircase-deadline cpu scheduler has replaced the old staircase design in this version."

Con goes on to explain, "the staircase-deadline cpu scheduler can be set in either purely forward-looking mode for absolutely rigid fairness and cpu distribution according to nice level, or it can allow a small per-process history to smooth out cpu usage perturbations common in interactive tasks by enabling this sysctl. While small fairness issues can arise with this enabled, overall fairness is usually still strongly maintained and starvation is never possible. Enabling this can significantly smooth out 3d graphics and games." Swap prefetch [story] is also among the patches included in the -ck patchset.

Following the release of the 2.6.21 kernel [story] Andrew Morton [interview] posted a list of patches in his -mm kernel, summarizing for each his plans as to whether or not they wil be pushed upstream for inclusion in the upcoming 2.6.22 kernel. He noted, "the overall stability in recent -mm's was not sufficiently high and we ran out of time to find all the bugs. I shouldn't have merged all those patches last week - they contained an exceptional amount of garbage. This all means that more bugs than usual will probably leak into mainline, and we'll have to fix them there." He went on to add, "I've been ducking most non-bugfix patches recently. I have ~200 feature and cleanup patches queued for later consideration, so people who sent those will be hearing from me eventually."

Following the release announcement of the 2.6.21 Linux kernel [story], Adrian Bunk noted that he no longer planned to track regressions [story]. He explained, "if we would take 'no regressions' seriously, it might take 4 or 5 months between releases due to the lack of developer manpower for handling regressions. But that should be considered OK if avoiding regressions was considered more important than getting as quick as possible to the next two week regression-merge window."

Linus Torvalds disagreed with Adrian's view that increasing the length of the release cycle would improve stability, "regressions _increase_ with longer release cycles. They don't get fewer." He went on to add, "you are ignoring the reality of development. The reality is that you have to balance things. If you have a four-month release cycle, where three and a half months are just 'wait for reports to trickle in from testers', you simply won't get _anything_ done. People will throw their hands up in frustration and go somewhere else." He continued:

"Do you really think bugs get fixed faster just because there wasn't a release? Quite the reverse. Bugs get _found_ faster thanks to a release (simply because you tend to get more information thanks to more users), giving the stable people more information, causing the bugs to be able to be found and fixed _more_quickly_ in the stable release than if we had waited for four months to release 2.6.21."

Linux creator Linus Torvalds announced the release of the 2.6.21 kernel, "if the goal for 2.6.20 was to be a stable release (and it was), the goal for 2.6.21 is to have just survived the big timer-related changes and some of the other surprises (just as an example: we were apparently unlucky enough to hit what looks like a previously unknown hardware errata in one of the ethernet drivers that got updated etc)." Regarding the the dynticks code which was merged in -rc1 [story] he said, "the big change during 2.6.21 is all the timer changes to support a tickless system (and even with ticks, more varied time sources). Thanks (when it no longer broke for lots of people ;) go to Thomas Gleixner and Ingo Molnar and a cadre of testers and coders." He went on to note, "of course, the timer stuff was just the most painful and core part (and thus the one that I remember most): there's a lot of changes all over. The appended changelog is just for the fixes since -rc7, so that doesn't look very impressive, the full changes since 2.6.20 are obviously a *lot* bigger (and you're better off reading the individual -rc changelogs)." Linus finished with a running joke about the many debates centered around current CPU scheduler efforts [story], quipping, "we now return you to your regular scheduler discussions".

Announcing 2.6.21-rc2, Linus Torvalds noted, "I'm not very proud of this, because quite frankly, -rc2 has way more changes than I really like." The current Linux kernel development model is that the bulk of changes in a new kernel should happen during the -rc1 phase, with the rest of the -rc kernels being primarily bug fixes. Linus explains, "it's largely my fault, because I simply missed a V4L/DVB merge that came in before the merge window closed, but since I didn't notice it didn't make -rc1, and as such it got merged late and is in -rc2 instead." With typical humor he added, "but because I'll flail around wildly and rather blame anything else than my own incompetence, I'll just claim that all the other kernel developers have been irresponsible, and caused -rc2 to be bigger than needed. In some areas (you know who you are) it may even be true.."

Summarizing other changes in the new release candidate, Linus said, "apart from the V4L/DVB merge, we've got a late PARISC update, and a number of driver updates (ata, networking, usb) changes. Along with the normal smattering of random stuff (core networking, selinux, infiniband, agp, mips, arm)." He then pointed to Adrian's regression lists [story] noting that some are fixed but there are more to go. The latest kernel can be downloaded from your nearest kernel.org mirror. You can browse through all the changes using the gitweb interface. Kernel Newbiews maintains a useful summary of all the changes going into the latest version of the Linux kernel.

Adrian Bunk posted a couple lists of known regressions that found their way into the 2.6.21-rc1 kernel [story] since the release of the 2.6.20 kernel [story]. Adrian notes that his lists only include bugs that are not yet fixed in Linus' -git tree. In an updated version of his lists he included 19 known regressions, including links to bugzilla or the appropriate mailing list discussion thread. The lists track who submitted the bug, who is currently handling it, who caused it if known, a link to a patch that fixes the problem if any, and the current status.

Linus Torvalds announced the first release candidate for the upcoming 2.6.21 kernel, ending the two-week merge window [story], "there's a lot of changes, as is usual for an -rc1 thing, but at least so far it would seem that 2.6.20 has been a good base, and I don't think we have anything *really* scary here." Linus noted that the tickless kernel patch [story] was finally merged into the mainline kernel, "the most interesting core change may be the dyntick/nohz one, where timer ticks will only happen when needed. It's been brewing for a _loong_ time, but it's in the standard kernel now as an option." Thomas Gleixner explained a year ago how this could result in cooler CPUs and power savings, "the tickless kernel feature (CONFIG_NO_HZ) enables 'on-demand' timer interrupts: if there is no timer to be expired for say 1.5 seconds when the system goes idle, then the system will stay totally idle for 1.5 seconds."

As for the rest of the changes, Linus added, "there's a ton of architecture updates (arm, mips, powerpc, x86, you name it), ACPI updates, and lots of driver work. And just a lot of cleanups." Release candidate kernels can be downloaded from your nearest kernel.org mirror. You can browse through all the changes using the gitweb interface. Kernel Newbiews maintains a useful summary of all the changes going into the latest version of the Linux kernel.

Following the release of the 2.6.20 kernel [story] Andrew Morton [interview] posted a list of patches in his -mm kernel, summarizing for each his plans as to whether or not they will be pushed upstream for inclusion in the upcoming 2.6.21 kernel. Andrew commented, "I'm getting fed up of holding onto hundreds of patches against subsystem trees, sending them over and over again and seeing nothing happen. I sent 242 patches out to subsystem maintainers on Monday and look at what's still here." In response to some confusion as to what happens to these patches, he went on explain, "once a subsystem has a subsystem tree (git or quilt) I basically never merge anything which belongs to that tree. It's always originator->mm->subsystemtree->Linus".