"[The] text below is mostly for the benefit of newbies - it's more along the lines of 'how to get from [a] bug report to the source of [the] bug', with more details than normal," began Al Viro, offering a full review of another Linux kernel oops in an effort to educate more people on how this is done. Al's walk through included a patch to fix the bug that caused the oops. He noted:

"This might be worth doing on [a] more or less regular basis, especially if more people join the fun; everyone [has] their own set of tricks in [this] area and making it easier to gather might help a lot of people. It's not just about oops-tracing per se, of course - Arjan's site gives a nice collection of those, so that makes an obvious starting point."

"I'll just take this opportunity to ask people that when they send bug-fixes, please try to make the subject line and message make sense for a *reader*, not for yourself (or even to me, although if it's readable to some generic person, it's hopefully readable to me too!)," Linus Torvalds explained in response to a recent bugfix. He went on to provide some general rules:

"1)if it's not fairly generic, specify the area (architecture, subsystem, driver) that the fix is for in the subject line. [...] 2) don't use commit names in the subject line - and while it's great to use them in the body of the explanation, even there you don't want to assume that people read it from within git. [...] 3) write the commit message for an outsider, and use whitespace. The third-most common fixup I end up doing (after the above two) is to split things up into shorter paragraphs, after somebody wrote a good changelog entry, but made it one large unreadable blob of text."

Linus added, "I end up editing just about half of all the commit messages of stuff I get in email (except for Andrew's stuff, since Andrew largely does the same kinds of cleanups anyway, so I only need to edit up a small percentage of the patches he forwards). I'd like it to be *much* less than that, so I thought I should speak up since I had an example of this."

When asked how to best refer to kernels between official releases and release candidates, Linus Torvalds pointed to his automated git snapshots. "I still call them 'nightly snapshots', but they do in fact happen twice a day if there have been changes, so that's not technically correct," he noted. The latest snapshot is 2.6.23-git15, "this is an exact name, because you can go to kernel.org and look up the exact commit ID that was used to generate it (there's an 'ID' file associated with each snapshot there)." For git users, he suggested using the "git describe" command to get the git name, with the current head being named v2.6.23-6562-g8add244. He went on to explain that the name "tells you three things: (a) it's based on 2.6.23 (b) there's been 6562 commits since 2.6.23 and (c) the top-of-tree abbreviated commit is '8add244'."

When asked about the previously discussed usage of "-rc0" and other similar proposed naming conventions, Linus replied:

"Please don't use those names. They don't actually tell anything about where in the cycle it is, and as you can see above, there's been 6500+ commits since 2.6.23, so saying '2.6.23-rc0' or similar really isn't very helpful if anybody actually cares about just where in the release cycle you are."

In response to a recent merge request, Linus Torvalds explained how he preferred the request to be formatted, "please don't hide the branch name in the free-flowing text". He noted that he wanted the git URL indented and on it's own line, "so that I don't miss the right branch-name even by mistake." He went on to explain:

"I try to be careful, and I think I missed the branch-name just once (and noticed it when the diffstat didn't match), but this is something I want to teach every git user to know very intimately: make it impossible to make mistakes, by always keeping the whole git information together, and never mixed up with any free-flowing explanation."

Two new documentation directories were merged into the upcoming 2.6.23 mainline kernel, containing translations of the HOWTO and stable_api_nonsense.txt documents in Japanese and Chinese. Greg KH explained, "here are some patches that add some translations of some procedural documentation files to the Documentation/ tree." Regarding some of the concerns that were expressed with merging translated documentation into the mainline kernel tarball, Greg noted, "these files change _very_ slowly over time, and are quite easy to keep up to date by the translators." He added:

"I know that kernel development is in English, but translations of a small subset of documentation files that go over procedures and how to get involved in the community is something that I feel is important and will bring in more developers in the end. Having these files in the kernel tree is a good way to keep a central location that all can see and easily find, instead of hiding them away on different web sites that might be harder to update by anyone who needs to do so."

In response to a recent merge request, Linus Torvalds explained a best practice when moving and changing code, "when doing renames it is generally *much* nicer to do a 100% rename (perhaps with just _trivial_ changes to make it compile - the include statements etc change, and maybe you want to change the name in the comment header too)." He went on to explain, "doing 'move the code and change it at the same time' is considered bad form. Movement diffs are much harder to read anyway (a traditional diff will show it as a new-file + delete, of course), so the general rule is: move code around _without_ modifying it, so that code movement (whether it's a whole file, or just a set of functions between files) doesn't really introduce any real changes, and is easier to look through the changes; do the actual changes to the code as a separate thing." He went on to note why this is especially important during Linux development, "where patches are the main way people communicate.":

"And when using git, the whole 'keep code movement separate from changes' has an even more fundamental reason: git can track code movement (again, whether moving a whole file or just a function between files), and doing a 'git blame -C' will actually follow code movement between files. It does that by similarity analysis, but it does mean that if you both move the code *and* change it at the same time, git cannot see that 'oh, that function came originally from that other file', and now you get worse annotations about where code actually originated."

Following up to a bug report against the 2.6.22 kernel, Andrew Morton and Linus Torvalds offered some tips on how to debug kernel problems. Andrew first pointed to netconsole.txt for instructions on setting up a netconsole, "when the machine has stalled, see if you can get a task trace with ALT-SYSRQ-t. This will require CONFIG_MAGIC_SYSRQ=y and possibly setting ignore_loglevel on the kernel boot command line."

Linus Torvalds suggested "git bisect" as an alternative, "[it] will take some time, but is really a lot easier" He explains, "there's almost 7000 commits in between 2.6.21 and 22, but that still means that in about fourteen recompiles/reboots, "git bisect" should tell us where your problem starts, which will hopefully make it obvious what the problem is (or at least pinpoint it a *lot*)." He goes on to detail how to install git, obtain the latest kernel, and run "git bisect", "doing a git bisect isn't really that hard, but fourteen compiles/reboots will take some time (well, the compiles will, the reboots aren't that bad). But even if you're not a git user, it really is very simple". Specifically, he notes, "start the 'git bisect' with 'git bisect good v2.6.21', 'git bisect bad v2.6.22', and it will pick a kernel version about half-way between the two points, and you can now start testing. For each kernel you try, if it boots fine, do 'git bisect good', otherwise boot into a working kernel, and then do 'git bisect bad'. Git will then pick the next 'halfway' kernel for that case."

The translation of a some kernel documentation into Japanese led to a discussion as to whether or not it was appropriate to include translated documentation with the kernel source code. One concern that was expressed was that as the number of included translations grows, so would the size of the kernel. Another concern was the liklihood that as time passes the various translations might become out of date. Jesper Juhl suggested one workaround, "since the common language of most kernel contributors is english I personally feel that we should stick to just that one language in the tree and then perhaps keep translations on a website somewhere. So the authoritative docs stay in the tree, in english, so that as many contributors as possible can read and update them."

Greg KH noted that there were a number of files in the kernel that change infrequently and that he would like to see included, "I really do want to see a translated copy of the HOWTO, stable-api-nonsense.txt, and possibly a few other files in the main kernel tree (SubmittingPatches, CodingStyle, and SubmittingDrivers might all be good canidates for this.) These files change relatively infrequently (the HOWTO file has had only 7 changes in 1 and 1/2 years, and they were very minor ones) and should be easy for the translators to keep up with."

The git directory content manager used to manage the Linux kernel source tree [story] continues to develop at a rapid pace [story]. Keeping up with the latest changes, Jeff Garzik released an updated version of his Kernel Hacker's Guide To Git. He explains, "several changes in git-core have made working with git a lot easier, so be sure to re-familiarize yourself with the development process."

Jeff's short guide is broken into four major sections, 'getting started' which talks about installing the software and getting a copy of the linux kernel source tree, 'basic tasks' which offers examples of keeping up to date with the latest code and merging in your own changes, 'branches' offering examples of creating, using and merging branches, and 'miscellaneous debris' which mentions applying patches from an mbox file and syncronizing tags. Further documentation on the various git commands can be found in the git man pages.

Greg KH offered a short "kernel maintainer's HOWTO for quilt and -mm", offering instructions on how one can utilize quilt to create patchsets intended to be merged into Andrew Morton [interview]'s -mm tree [story]. He begins:

"So, You're a kernel maintainer faced with the fact that you are having people send you loads of patches, but don't know how to stage them in a fashion that others can see what you have and have not accepted. You also want to have them show up in the -mm releases and need to provide some hint as to the order in which they should be applied. This small document and script will provide one solution to this."

Rusty Russell [interview] recently posted an updated version of his "Unreliable Guide To Locking". The introduction begins:

"Welcome, to Rusty's Remarkably Unreliable Guide to Kernel Locking issues. This document describes the locking systems in the Linux Kernel in 2.6. With the wide availability of HyperThreading, and preemption in the Linux Kernel, everyone hacking on the kernel needs to know the fundamentals of concurrency and locking for SMP. "

Rusty's excellent guide is quite informative, helping the reader to grasp concurrency, explaining the common types of locks, providing useful examples, listing common problems, discussing locking speed, and much more. All in all, it's an essential reference.

In a couple of earlier articles, we walked through the process of upgrading to the 2.6.0-test4 kernel [story], and then using a small patch to upgrade to the 2.6.0-test5 kernel [story]. Today we'll continue our patching efforts to upgrade to an even faster feeling and more stable kernel with Andrew Morton's [interview] -mm patchset [forum].

Andrew Morton began releasing his -mm kernel patches a little over a year ago, in the summer of 2002. The -mm tree began as a 90k patch against the 2.5.17 development kernel, merging in the remote kernel debugger, kgdb. By the release of 2.5.18, the -mm patchset had grown to nearly 238k, merging in a wide assortment of fixes and new functionality. As of this writing, the current -mm patchset is 2.6.0-test5-mm3, weighing in at nearly 5 megabytes. Andrew's -mm tree has evolved from a testing ground for numerous new technologies, to a comprehensive patchset that is usually more stable than the mainline 2.6.0-test kernel itself. This bodes well for the future of the 2.6 kernel, as Andrew Morton will soon be the official 2.6 kernel maintainer.

There are numerous reasons you may desire trying Andrew's -mm kernel tree. Stability alone is a good incentive, and scanning the lengthy changelog you'll find a significant number of bug fixes that have been applied. I asked Andrew how the stability of his kernel compares to that of the mainline 2.6.0-test kernel, and he replied that though occasionally new bugs creep in, due to having the latest fixes the -mm tree is generally more stable and up-to-date.

Linux creator Linus Torvalds has released the linux 2.6.0-test5 kernel, with the following comments:

"Lots of small stuff, as usual. I think the biggest "core" change is the Futex changes by Jamie and Hugh, and the dev_t preparations by Al Viro. But there are ARM and ppc updates here too, and a few drivers have bigger fixes (tg3 driver and the USB gadget interface stand out on diffstat). Watchdog driver updates etc. And Russell King fixed more PCMCIA issues."

Read on for the full changelog.

Additionally, if you followed my recent upgrade howto [story], are running a 2.6.0-test kernel, and are interested in upgrading to 2.6.0-test5, read on for a few simple tips on upgrading with incremental patches.

Anyone who's been following Linux kernel development for the past several months has heard about one exciting feature after another being merged into the still un-released 2.6 kernel. New features that noticeably affect user experience include Robert Love's [interview] preemptible kernel work [story], Ingo Molnar's [interview] O(1) Scheduler [story], Rik Van Riel's [interview] reverse mapping VM [story], Nick Piggins' [interview] Anticipatory I/O scheduler [story], and much, much more...

Having some spare time a few nights ago, I decided to give the latest kernel, 2.6.0-test4, a trial run on my aging 550Mhz PIII desktop computer, and the result was nothing short of spectacular. As the final 2.6.0 release approaches, it is important that an increasing number of users (aka testers) give this kernel a try, especially as currently it's still a sexy task for developers to track down kernel bugs and stabalize their work. Once work starts on the 2.7 development tree, inevitably much talent will again be focusing on new features.

The purpose of this document is to provide some helpful tips to readers that currently compile their own 2.4 kernels, but haven't yet made the leap to 2.6. This is still a development kernel, so you may run into problems, but overall stability and performance is quite impressive and I can't recommend enough that you try it today.