In an announcement for the 2.6.25.10 stable kernel, Greg KH noted, "it contains a number of assorted bugfixes all over the tree. And once again, any users of the 2.6.25 kernel series are STRONGLY encouraged to upgrade to this release." The emphasis on the word strongly led to a lengthy discussion about how security fixes are handled in the Linux Kernel. Linus Torvalds replied, "I personally consider security bugs to be just 'normal bugs'. I don't cover them up, but I also don't have any reason what-so-ever to think it's a good idea to track them and announce them as something special." Later in the thread he went on to explain, "one reason I refuse to bother with the whole security circus is that I think it glorifies - and thus encourages - the wrong behavior. It makes 'heroes' out of security people, as if the people who don't just fix normal bugs aren't as important. In fact, all the boring normal bugs are _way_ more important, just because there's a lot more of them. I don't think some spectacular security hole should be glorified or cared about as being any more 'special' than a random spectacular crash due to bad locking."

Theodore T'so pointed out that other developers had different beliefs about disclosure than Linus and referred to mailing lists such as the private security@ list described in the SecurityBugs documentation, originally created in early 2005. He then described Linus' stance, "if Linus finds out about a security bug, he will fix it and check it into the public git repository right away. But he's very honest in telling you that is what he will do --- so you can choose whether or not to include him in any disclosures that you might choose to make." Regarding whether Full Disclosure is the best policy, Ted highlighted the fact that the debate has been going on for several decades, "it is clear that we're not going settle this debate now, and certainly not on the Linux Kernel Mailing List." Later in the discussion, Linus offered a succinct summary of his viewpoint, "my responsibility is to do a good job. And not pander to the people who want to turn security into a media circus."

"It's been long promised, but there it is now," began Linux creator Linus Torvalds, announcing the 2.6.25 Linux kernel. He continued, "special thanks to Ingo who found and fixed a nasty-looking regression that turned out to not be a regression at all, but an old bug that just had not been triggering as reliably before. That said, that was just the last particular regression fix I was holding things up for, and it's not like there weren't a lot of other fixes too, they just didn't end up being the final things that triggered my particular worries." Linus added:

"The full changelog from 2.6.24 is 7.5M, with a 12MB compressed patch. Tons and tons has changed, but if you've been following the -rc releases, you'll already know about the big things. The changes from the last rc (-rc9) are fairly small and mostly pretty trivial, and the shortlog is appended. So it's mostly one-liners, with some updates to drivers (net and usb) and to networking that are a bit larger (although a number of the driver updates are things like just new ID's etc)."

More information about the latest release can be found on the KernelNewbies Linux 2.6.25 wiki page.

"Finally found it ... the patch below solves the sparsemem crash and the test system boots up fine now," announced Ingo Molnar. He described the patch as fixing a "memory corruption and crash on 32-bit x86 systems. If a !PAE x86 kernel is booted on a 32-bit system with more than 4GB of RAM, then we call memory_present() with a start/end that goes outside the scope of MAX_PHYSMEM_BITS." He included a source snippet with the loop that could corrupt memory, "depending on what that memory is, we might crash, misbehave or just not notice the bug." Ingo went on to note that the bug was first introduced with sparsemem support in the 2.6.16 kernel:

"I believe this was the reason why my many bisection attempts were unsuccessful: the bug pattern was not stable and seemingly working kernels had the memory corruption too. It was pure luck that v2.6.24 'worked' and v2.6.25-rc9 broke visibly."

Linux creator Linus Torvalds replied, "good job. I've pushed this out, and will let this simmer at least overnight to see if there are any brown-paper-bag issues (either with this or with some last changes from Andrew), but I'm happy, and I think I'll do the real 2.6.25 tomorrow."

"I really don't want to do this, and I was actually hoping to release 2.6.25 last weekend (which is why -rc9 is a few days late - just me hoping to not do another -rc at all), but I've done an -rc9," Linus Torvalds said, announcing the 2.6.25-rc9 kernel. "The changes in -rc9 are pretty small (shortlog appended)," he continued, "and 60% of them are m68k updates - mostly defconfigs. And some doc updates. But there's some network driver updates (tg3 and wireless hostap stand out), some late XFS patches and a mvsas driver update (the mvsas driver is new in 2.6.25, so that's not going to regress anything ;). The rest is mostly one-liners, with a few reverts going on." Linus then explained why he was putting out another release candidate:

"The reason for not doing a 2.6.25 is that some people are making noises about slab/page-alloc setup issues, and I wanted something out this week, but didn't feel comfy doing a final release.

"That said, I think I'll have to do 2.6.25 early next week regardless, because we can't just keep holding things back forever. At some point it will have to turn into a 2.6.25.x issue, and the developers with stuff pending for the next version need to be able to start merging."

"No cute April 1st shenanigans, just a regular -rc release that happened to come up today because I was waiting for the input layer oops-fixes to be ready and tested," began Linus Torvalds, announcing the 2.6.25-rc8 kernel on April 1st. He continued, "the bulk of the fixes are the usual random one-liners. [...] A lot of the one-liners are some sparse cleanups, which is probably unnecessary noise at this point, but when Al sends me a series I just tend to apply it because his patches tend to be rather careful and basically always correct." Linus added:

"The big thing that is actually *noticeable* to most people is that this should fix the two top regressions: we've had some suspend-resume regressions due to the stupid ACPI _PTS ordering issues, and while the cleanups were left, the ordering changes were reverted. So that should fix issues for some people (of course, the people who had it fixed are unhappy, but regressions are worse). The other thing that bit a number of people and is now fixed (and that also probably often showed up as a suspend/resume regression) was some 'struct device' lifetime changes that broke the input layer. Thanks to people who debugged that one."

"So this hopefully continues closing various regressions, and most of the changes are pretty small (ie diffstat shows a lot of oneliners). The biggest patches are the trivial powerpc defconfig updates which show up pretty clearly in the dirstat, ie if it weren't for those, the arch/ updates would hardly show up at all," began Linus Torvalds, announcing the 2.6.25-rc7 Linux kernel. He noted that the ps2esdi driver was removed after being marked broken for years, and a new metronomefb.c driver was added for the E-Ink Metronome controller. Linus continued:

"Apart from those, most of the changes really are fairly small and spread out. The scheduler got some tweaking, the memstick driver got some TLC, and cifs and reiserfs had some fixes. The shortlog has more details, but it boils down to some reverts, some docbook fixes, some sparse annotation fixups, a number of trivial patches, and a healthy sprinkling of small fixups."

In summary, Linus suggested, "give it a good testing, because we're hopefully now well on our way towards that eventual real 2.6.25 release!"

"I lost a day-and-a-half this week due to a disk that decided to get read errors due to an unfortunate power outage, and had to spend too much time regenerating my normal setup," began Linus Torvalds, announcing the 2.6.25-rc6 kernel, "but I don't think I lost any emails, and things seemed to have calmed down a bit, so here's to hoping that -rc6 is starting to look better." He then summarized the changes:

"The dirstat shows the usual pattern of most changes being in drivers and architecture updates, although this time it's a bit skewed by the parisc and powerpc updates (hopefully closing the parisc compile regression among other things), which means that arch is about half, and drivers are just under a third of the patch (it seems to be usually the other way around)."

"It's a few days late, but I was waiting for some updates for some of the most annoying regressions until releasing it, so the end result is hopefully more useful as a result," Linus Torvalds began, announcing the 2.6.25-rc4 kernel. He offered a dirstat summary, noting, "the dirstat shows that (as usual) most of the changes are in drivers and arch (~51% and ~17% respectively), with about half the driver updates being in network drivers." Linus continued:

"In particular, the block layer changes should hopefully have sorted themselves out, and CD burning etc hopefully works for people again. Same goes for the the scheduler regressions, and a number of annoying boot-time problems. [...] It's really a fair amount of small changes spread all over, with most of the changes being quite small (604 commits, most of them small, with the BNX2X network driver and the new fsldma driver the only ones that got some bigger changes)."

"Ok, it's out there, ready for your enjoyment," Linus Torvalds said, announcing the 2.6.25-rc3 kernel. He summarized the changes:

"As usual, most of the updates are in architecture and drivers, with the dirstat showing about 37% in arch (and that's with rename detection: there's some file movement in arch/xtensa that would bring it up to 43% if you looked at it as a traditional diff) and almost 50% in drivers. Much of the include file stuff is also architecture-related updates. The driver updates are mostly fairly spread out, but some of it comes from a couple of new drivers: the mvsas SCSI driver, a new adt7473 driver, and a couple of new watchdog drivers."

Linus continued, "if you ignore the architecture-specific stuff and drivers, the rest is mostly in networking, some Documentation updates, and a few filesystem updates (mainly efs and xfs). Anyway, the upshot of it all? Quite frankly, it's all over the place. The changes in -rc3 are bigger than -rc2, probably mostly because we had some more time (-rc2 was a couple of days early because of the long weekend in the US), but hopefully also because people have started to find regressions." Among the bug fixes, he highlighted, "we had a nasty SLUB corruption issue in -rc2 that is fixed (not that very many people probably saw it), and we've hopfully fixed a number of regressions in networking and suspend/resume."

"Ok, this kernel is a winner," began Linux creator Linus Torvalds, playfully announcing the 2.6.25-rc2 kernel which gained the name "Funky Weasel is Jiggy wit it". He continued:

"Just to show how _much_ of a winner it is, it's been awarded a coveted 'weasel' series name, which should tell you just how good it's going to be. It's a name revered in Linux kernel history, and as such this brings back the good old days where if you find a bug, you're almost certainly simply mistaken, and you probably just did something wrong. But hey, you can try to prove me wrong. I dare you."

Linus went on to describe some of the changes using 'git dirstat', "in particular, it shows that almost exactly half of the updates are to drivers, with network drivers alone being a third of the whole patch. And of the remaining half, about half was architecture updates, notably to SH." He then noted, "I'm optimistic that this release cycle won't be anywhere near the pain of what 24 was, which is why I'm just going to go off for the long weekend and stay at the beach."

"Ok, it's a bloody large -rc (as was 24-rc1, for that matter), probably because the 2.6.24 release cycle dragged out, so people had a lot of things pending," noted Linus Torvalds, announcing the 2.6.25-rc1 kernel. He added, "the full diff is something like 11MB and 1.4M lines of diffs, with the bulk of the stuff being in architecture updates and drivers." Linus continued:

"Just to have some fun, I did trivial statistics, and of the 1.4M lines of diffs, about 38% - 530k lines - were in architecture files (400k+ lines of diffs in arch/, 100k+ lines of diffs in include/asm-*), and another big chunk is in drivers (including sound) at about 44% - 610k lines - of changes. The rest comes in much smaller, but still noticeable is networking (8% - 110k lines), with filesystems at 4%, and documentation at about 2%. The remaining crumbles being spread out mostly over block layer, crypto, kernel core, and security layer updates (ie SElinux and smack)."

Linus highlighted a few of the changes, including, "the Intel graphics driver is starting to do suspend/resume natively (ie even without X support), which is a welcome sign of the times and may help some people; lots of cleanups from the x86 merge (making more and more use of common files), but also the big page attribute stuff is in and caused a fair amount of churn, and while most of the issues should have been very obvious and all got fixed, this is definitely one of those things that we want a lot of very wide testing of to make sure nothing regressed; fair number of changes to things like the legacy IDE drivers too, and a totally new driver for the very common PCIE version of the Intel e1000 network card etc; and I've probably totally forgotten about tons of other stuff I should have mentioned, but the point is that not only do we have lots of new core, we do have a fair amout of changes to basic stuff that can actually affect perfectly bog-standard hardware setups. So give it all a good testing."

"While this is probably one of the last days of the merge window, please still consider pulling the 'kgdb light' git tree," began Ingo Molnar, explaining:

"This is a slimmed-down and cleaned up version of KGDB that i've created out of the original patches that we submitted two weeks ago. I went over the kgdb patches with Thomas and we cut out everything that we did not like, and cleaned up the result. KGDB is still just as functional as it was before (i tested it on 32-bit and 64-bit x86) - and any desired extra capability or complexity should be added as a delta improvement, not in this initial merge."

Ingo noted that the previous merge request modified 41 files, while this new merge request modifies only 22 files. Among the changes, he highlighted, "removed _all_ critical path impact, even if KGDB is enabled and active; removed all the lowlevel serial drivers; added a redesigned and cleaned up version of the 'KGDB over polled consoles' approach; removed the longjump code; removed the module symbol hacks; removed the GTOD/clocksource hacks; removed the softlockup hacks; removed the toplevel Makefile changes; removed the might_sleep scheduler hack; and did lots of other cleanups and rewrites as well." Ingo summarized, "as a result, this kgdb series has _obviously_ zero impact on the kernel, because it just does not touch any dangerous codepath. From this point on KGDB can evolve in small, well-controlled baby steps, as all other kernel code as well. And the resulting kgdb is still very functional: it can still break into a kernel (via SysRq-G), can catch crashes, can single-step, etc. It's already a quite usable first step."

It was recently pointed out that most of the x86 architecture patches had been merged into the mainline 2.6.25 kernel, except for the kgdb patches. Linus Torvalds replied, "I won't even consider pulling it unless it's offered as a separate tree, not mixed up with other things. At that point I can give a look." He continued:

"That said, I explained to Ingo why I'm not particularly interested in it. I don't think that 'developer-centric' debugging is really even remotely our problem, and that I'm personally a lot more interested in infrastructure that helps normal users give better bug-reports. And kgdb isn't even _remotely_ it.

"So I'd merge a patch that puts oops information (or the whole console printout) in the Intel management stuff in a heartbeat. That code is likely much grottier than any kgdb thing will ever be (Intel really screwed up the interface and made it some insane XML thing), but it's also fundamentally more important - if it means that normal users can give oops reports after they happened in X (or, these days, probably more commonly during suspend/resume) and the machine just died."

Ingo Molnar summarized his pull request for changes to the x86 architecture bound for mainline inclusion in 2.6.25 noting, "it's not a small merge, it consists of 908 commits from 96 individual arch/x86 developers (!)". He continued, "a number of core files are changed as well: most notably percpu, debugging details, timers, the firewire remote debugging patch and ... the KGDB remote debugging stub in kernel/kgdb.c." He went on to detail the extent of the testing this tree has received, "in the past few weeks tens of thousands of random x86.git bzImages were successfully built and booted on a number of (commodity) 32-bit and
64-bit testsystems - and there has been a fair amount of test exposure on -mm as well." Regarding the remote kernel debugger, Ingo explained:

"We tested KGDB to be merge-worthy within the x86 architecture (the only supported architecture for now) and it's better to have kernel/kgdb.c than arch/x86/kernel/kgdb.c. The code is reasonably clean and the user-space exposure is small - the only real exposure is the decades-old remote GDB protocol. We are happy to fix up any further cleanliness comments that people might have - but we really wanted to start somewhere and get this thing moving. As an added bonus: finally a kernel debugger that can be read without puking too much ;-) [anyone remember KDB?]"

Avi Kivity summarized the kvm patches bound for the 2.6.25 kernel:

"Changes include performance and scalability improvements, completion of the portability work (though no new architectures are supported with this submission), support for new hardware features, using general userspace memory for kvm (which enables swapping guest memory as well as sharing memory among guests), as well as the usual cleanups and incremental fixes."

The Kernel-based Virtual Machine project, kvm, was started in mid-2006, and has been part of the Linux kernel since the 2.6.20 release in February of 2007. The recent changes can be browsed with gitweb.