"So this merge window was somewhat rocky in the sense that there was a lot of arguments about it, but at the same time I at least personally think that from a technical angle, we had somewhat less scary stuff going on than has been almost the rule lately," noted Linux creator Linus Torvalds, announcing the 2.6.26-rc1 kernel. He continued:

"Lots of changes, but nothing that really feels all that fragile to me. Famous last words. I expect that the x86 PAT support (which has been long in the making) has the potential to have some issues, but the obvious problems were hashed out long ago, and while the merge window already showed one bug, that one was fairly benign and quickly fixed."

Linus highlighted, "another feature that is notable not for its size, but because people have tried to get me to merge it for some long is kgdb support. Which really turned out pretty small and clean, once people started putting their effort into making it so." He concluded, "so go out and test it. The diffstat and shortlogs are too big to post here (7500+ commits and the compressed full patch is 8.5MB in size), but one interesting tidbit I found was that during this *one* merge window, we had almost 800 different authors."

"While this is probably one of the last days of the merge window, please still consider pulling the 'kgdb light' git tree," began Ingo Molnar, explaining:

"This is a slimmed-down and cleaned up version of KGDB that i've created out of the original patches that we submitted two weeks ago. I went over the kgdb patches with Thomas and we cut out everything that we did not like, and cleaned up the result. KGDB is still just as functional as it was before (i tested it on 32-bit and 64-bit x86) - and any desired extra capability or complexity should be added as a delta improvement, not in this initial merge."

Ingo noted that the previous merge request modified 41 files, while this new merge request modifies only 22 files. Among the changes, he highlighted, "removed _all_ critical path impact, even if KGDB is enabled and active; removed all the lowlevel serial drivers; added a redesigned and cleaned up version of the 'KGDB over polled consoles' approach; removed the longjump code; removed the module symbol hacks; removed the GTOD/clocksource hacks; removed the softlockup hacks; removed the toplevel Makefile changes; removed the might_sleep scheduler hack; and did lots of other cleanups and rewrites as well." Ingo summarized, "as a result, this kgdb series has _obviously_ zero impact on the kernel, because it just does not touch any dangerous codepath. From this point on KGDB can evolve in small, well-controlled baby steps, as all other kernel code as well. And the resulting kgdb is still very functional: it can still break into a kernel (via SysRq-G), can catch crashes, can single-step, etc. It's already a quite usable first step."

It was recently pointed out that most of the x86 architecture patches had been merged into the mainline 2.6.25 kernel, except for the kgdb patches. Linus Torvalds replied, "I won't even consider pulling it unless it's offered as a separate tree, not mixed up with other things. At that point I can give a look." He continued:

"That said, I explained to Ingo why I'm not particularly interested in it. I don't think that 'developer-centric' debugging is really even remotely our problem, and that I'm personally a lot more interested in infrastructure that helps normal users give better bug-reports. And kgdb isn't even _remotely_ it.

"So I'd merge a patch that puts oops information (or the whole console printout) in the Intel management stuff in a heartbeat. That code is likely much grottier than any kgdb thing will ever be (Intel really screwed up the interface and made it some insane XML thing), but it's also fundamentally more important - if it means that normal users can give oops reports after they happened in X (or, these days, probably more commonly during suspend/resume) and the machine just died."

Ingo Molnar summarized his pull request for changes to the x86 architecture bound for mainline inclusion in 2.6.25 noting, "it's not a small merge, it consists of 908 commits from 96 individual arch/x86 developers (!)". He continued, "a number of core files are changed as well: most notably percpu, debugging details, timers, the firewire remote debugging patch and ... the KGDB remote debugging stub in kernel/kgdb.c." He went on to detail the extent of the testing this tree has received, "in the past few weeks tens of thousands of random x86.git bzImages were successfully built and booted on a number of (commodity) 32-bit and
64-bit testsystems - and there has been a fair amount of test exposure on -mm as well." Regarding the remote kernel debugger, Ingo explained:

"We tested KGDB to be merge-worthy within the x86 architecture (the only supported architecture for now) and it's better to have kernel/kgdb.c than arch/x86/kernel/kgdb.c. The code is reasonably clean and the user-space exposure is small - the only real exposure is the decades-old remote GDB protocol. We are happy to fix up any further cleanliness comments that people might have - but we really wanted to start somewhere and get this thing moving. As an added bonus: finally a kernel debugger that can be read without puking too much ;-) [anyone remember KDB?]"

The final 2.6.24 Linux kernel is expected any day now, so the various subsystem maintainers have begun summarizing what changes are expected to be merged into the mainline kernel during the 2.6.25 merge window. Ingo Molnar spoke to changes for the x86 architecture, "there are 763 commits in x86.git so far, from more than 90 contributors, so it would be difficult to mention and credit every contribution in this mail." Along with a lengthy list of other changes, he included:

"Continued, intense arch/x86 unification and cleanup work by lots of people; FIFO ticket spinlocks for better spinlock scalability; 'regset' generalizations - the most important step towards utrace support (==next-gen ptrace); support for more than 255 CPUs [up to 4096 - in theory up to 65535]; almost complete 64-bit paravirt guest support; KGDB support on x86, finally!"

"This is a request to merge KGDB into the mainline kernel," Jason Wessel announced, posting a series of patches aiming toward that goal. He continued, "as of right now KGDB is comprised of 21 different patches adding in the core api and docs first and then working up to add drivers and arch specific support to KGDB. The patches were broken down into logical pieces for review and comments." He went on to explain:

"The intent of the KGDB patches is to unify the KGDB support across all the architectures that elect to implement the KGDB functionality by providing a common core and an arch specific stub. For quite some time there has been different features and uses of KGDB across the most popular architectures. Having a common core that takes care of protocol parsing and the typical use case of software breakpoints should eliminate the inconsistencies across the archs as well as making it easier to add KGDB support to a new arch."

Andrew Morton, who has been supportive of getting a kernel debugger into the mainline kernel, explained that it was too late in the 2.6.24 review cycle to merge KGDB, meaning it would have to wait for 2.6.25 at the earliest, "this won't work very well. There's a lot of review work to be done here, and a lot of it by busy architecture maintainers. Expecting people to do all this review and test work late in the merge window when they're all madly scrambling to get their bugs^Wpatches into mainline is not reasonable. This should all have started a month ago. So we're looking at a 2.6.25 merge for this work."

"Is anyone testing the kgdb code in here?" Andrew Morton asked in his release announcement for the 2.6.23-rc1-mm2 patchset. Mike Frysinger asked, "does kgdb actually have a chance to get merged? With the history of it, i just assumed it was never going in". In the past, Linus Torvalds has resisted merging kernel debuggers and famously said, "I don't like debuggers. Never have, probably never will," going on to explain why he didn't want it to be too easy to hack the Linux kernel. An earlier push to get kgdb merged in 2004 didn't succeed, though some architectures already have versions of the debugger. The current kgdb patchset in Andrew's tree includes code for the i386, x86_64, ppc, mips, sh and arm architectures.

Andrew replied to Mike's question, "I was hoping for a 2.6.24 merge. But I haven't actually looked at it yet. Hopefully Jason is planning to get it all out for review soonish." He went on to add, "runtime testing isn't actually the most important thing at this time - if is doesn't work, well hey, we fix it, easy - we always have bugs. The main emphasis right now should be on higher-level design/review/integration stuff." Jason Wessel noted, "the KGDB tree is broken up into incremental units each layer adding more functionality and or arch specific pieces."

In a continuing effort to get kgdb merged into the mainline 2.6 Linux kernel [story], Amit Kale announced a feature freeze on his core-lite and i386-lite kgdb patches. At this point, he intends to only perform bug-fixes and code cleanup to these patches until they are merged, aiming for 2.6 inclusion by the end of this month. This first merge will only add kgdb support with a minimal featureset to the i386 architecture, allowing remote kernel debugging through a serial connection.

Recently added to the patches was some kgdb documentation, which begins:

"kgdb is a source level debugger for [the] linux kernel. It is used along with gdb to debug a linux kernel. Kernel developers can debug a kernel similar to application programs with use of kgdb. It makes it possible to place breakpoints in kernel code, step through the code and observe variables."

Rusty Russell's new module loader was recently merged into Linus' 2.5 kernel tree [story]. This new implementation aims to cleanup and reduce the amount of code in the kernel and user space required to load a kernel module. Additionally, it now removes the requirement that kernel and user space code for modutils have to be in sync.