"It's been two weeks rather than the usual one, because we've been hunting a really annoying VM regression that not a lot of people seem to have seen, but I didn't want to release an -rc4 with it," began Linux creator Linus Torvalds, announcing the 2.6.34-rc4 Linux kernel. He explained, "we had the choice of either reverting all the anon-vma scalability improvements, or finding out exactly what caused the regression and fixing it. And we got pretty close to the point where I was going to just revert it all." Linus continued:
"Absolutely _huge_ kudos to Borislav Petkov who reported the problem and was able to not just reliably reproduce it, but also test new patches to try to narrow things down at a moments notice. The thing took ten days of emails flying back and forth, and Borislav was there all the time, day and night, through several patches that tried to fix it (several real bugs, but not the one he hit) and lots of patches to just add instrumentation to get us nearer to the cause of the problem. And finally, today, confirmation that we actually nailed the problem. So if anybody has been seeing a oops (or sometimes a GP fault) in page_referenced(), that should be gone now."
As for the rest of the changes, Linus noted, "the bulk of the changes come from drivers - a new network driver (cxgb4), but also updates to the radeon and nouveau drivers. And then there is the random updates everywhere." Read on for the full changelog.
"So yet another week, another -rc," began Linux creator, Linus Torvalds, announcing the 2.6.27-rc8 Linux kernel. He continued, "this one should be the last one: we're certainly not running out of regressions, but at the same time, at some point I just have to pick some point, and on the whole the regressions don't look _too_ scary. And -rc8 obviously does fix more of them." Linus went on to note that most of the changes since -rc7 are small, "and there aren't even a whole lot of them."
Jiri Kosina cautioned that there is still an unknown bug affecting the e1000e driver currently in the 2.6.27 kernel, "rendering the cards unusable for most of the i-am-not-a-hacker users (and remember, even Dave Airlie bricked his laptop completely to death, when trying to restore eeprom contents)" When asked how to duplicate the bug, Jiri noted that the inability to reliably reproduce the bug added to the difficulty in debugging the problem, "apparently it is some kind of race, as it usually takes multiple cycles to trigger".
"The patches most people hopefully care about tend to be small details," noted Linus Torvalds, announcing the 2.6.27-rc6 kernel. He continued, "and so more regressions should hopefully be closed now, some by just reverting the commits that caused breakage. I don't think anything special merits explicit comment, but you can get a flavor for things by scanning the appended shortlog." Earlier in the announcement email, Linus did note some specifics about which drivers caused the bulk of the patch:
"Same old deal - except it's been almost two weeks since -rc5. That said, the diff is actually about the same size, so I guess that means things are calming down. Most of the diff (bulk-wise) is updates to the new gspca (standard USB webcam) driver, although some of it is also removal of the dead miropcm20* driver."
Linus Torvalds announced the 2.6.27-rc5 Linux Kernel, noting that his "weekly releases" tend to happen every eight days, adding, "the bulk of it is all config updates, and with arm and powerpc leading the pack." Linus continued:
"While the config updates amount to about three quarters of the diff, and if you don't use a rename-aware diff the blackfin include file movement pretty much accounts for the rest, hidden behind all those trivial (but bulky) changes are a lot of small changes that hopefully fix a number of regressions.
"The most exciting (well, for me personally - my life is apparently too boring for words) was how we had some stack overflows that totally corrupted some basic thread data structures. That's exciting because we haven't had those in a long time. The cause turned out to be a somewhat overly optimistic increase in the maximum NR_CPUS value, but it also caused some introspection about our stack usage in general. Including things like a patch to gcc to fix insane stack usage for vararg functions on x86-64. But that one would only hit anybody who was a bit too adventurous and selected the big 4096 CPU configuration. The rest of the regressions fixed are a bit more pedestrian."
"Another week, another -rc," began Linus Torvalds, announcing the 2.6.27-rc4 Linux kernel, continuing, "this time the diffstat is almost totally dominated by the addition of the musb driver that drives the MUSB and TUSB controllers integrated into omap2430 and davinci. That, together with the removal of the auerswald USB driver (replaced by libusb version) is more than half of the bulk of the patch, and obviously most users won't ever notice." Linus added:
"Apart from those bulky USB updates, there's some arch updates (blackfin and ia64), network and input driver updates, and an XFS and UBIFS update. The rest is mostly random stuff all over, probably best described by the appended shortlog. A number of regressions should be off the table, but more remain..."
"So it's been a week since -rc1, and -rc2 is out there," began Linux creator Linus Torvalds, announcing the 2.6.27-rc2 Linux kernel. He noted, "there's a lot of random changes in there, and I'm hoping we're starting to calm down, but one particular _kind_ of random change is probably worth pointing out explicitly due to the things it can result in: the fact that a number of architectures ended up using the 'lull' after -rc1 (hah!) to do the 'include/asm-xyz' => 'arch/xyz/include/asm' renames." Linus explained that for people actively developing and merging code with git, "be aware that we've recently had more renames than the rename detection limit in git defaults to, and as a result, if you have a rename<->data change conflict, you may want to increase the default limit." Linus noted that developers with sufficient ram can set "renamelimit=0" to completely disable the limit, and others can set it to a high value such as 5,000, "the default limit is pretty low just to not cause problems for people who have less memory in their machines than kernel developers tend to have..."
Linus continued, "the dirstat (with rename detection on, so as to not show the movement as huge changes) is fairly usual, with most of the changes in drivers, along with an ext4 and xfs update making 'fs' show up pretty high too". He added:
"The shortlog is still a tad too big to make it on the list (again, as usual - normally I end up posting shortlogs for -rc3 and later when they become more manageable) but let me just say that it isn't really all that interesting. Theres' a lot of small changes here, but nothing that makes you go 'Wow!'. Not that there _should_ be anything like that in -rc2, of course, so I'm not complaining."
"It's two weeks (and one day), and the merge window is over," began Linus Torvalds, announcing the 2.6.27-rc1 kernel. He continued, "finally. I don't know why, but this one really did feel pretty dang busy. And the size of the -rc1 patch bears that out - at 12MB, it's about 50% bigger than 26-rc1 (but not that much bigger than 24/25-rc1, so it's not like it's anything unheard of)." He reflected, "the pure size of the -rc's _is_ making me a bit nervous, though. Sure, it means that we are good at merging it all, but I have to say that I sometimes wonder if we don't merge too much in one go, and even our current (fairly short) release cycle is actually too big." As for the actual changes, Linus explained:
"Much of -rc1 was in linux-next, but certainly not everything. We'll see how that whole thing ends up evolving - it certainly didn't solve all problems, and there was some bickering about things that weren't there (and some things that mostly were ;), but maybe it helped. There's a ton of new stuff in there, but at least personally the interesting things are the BKL pushdown and perhaps the introduction of the lockless get_user_pages_fast(). The build system also got updated to allow moving the architecture include files ('include/asm-xyz') into the architecture subdirectories ('arch/xyz/include/asm'), and sparc seems to have taken advantage of that already."
Other changes Linus highlighted included merging the UBI filesystem, as well as, "tracing, firmware loading, continued x86 arch merging, and moving more code to generic support (unified generic IPI handling, coherent dma memory allocation, show_mem etc). Bootmem rewrites. [And] some support for further scalability (ie 4k cpu cores)."
"Ok, the last -rc obviously wasn't the last one after all, since here's a new one," noted Linus Torvalds, announcing the 2.6.26-rc9 kernel. He continued, "enough changes that we needed another -rc, and the regression list isn't emptying fast enough either (probably because a number of people, including reporters, are vacationing)." He went on to summarize:
"The actual bulk of this all is a new UVC video driver for the standard USB Video Class specification. It's a new driver, so shouldn't cause any regressions, but it's fairly sizable [...] ie 78% is just that one new driver, and almost 92% is driver updates in general (although some of them are reverts, so they show up as diffs against -rc8, but they actually cause the _total_ diff against 2.6.25 to shrink a bit). The fs updates are partly some minor updates to 9p, ecryptfs, proc and udf, but partly some delayed cleanup patches that went through Al. Bad Al. But when Al sends me patches, I apply them. I worry what would happen if I didn't. The rest is mainly small fixes (one-liners and 'few-liners') all over the place, many of them merged from Andrew's -mm queue."
"It hasn't been a week, I know, and this is a pretty small set of changes since -rc7, but I'm going to be mostly incommunicado for the next week or so, so I just released what will hopefully be the last -rc," began Linux creator Linus Torvalds, announcing the 2.6.26-rc8 kernel. He added, "or maybe not. It depends on how good you all are while I'm not looking." Regarding the latest release candidate, Linus explained:
"Most of the bulk of the changes here are to Xen and to KVM in particular, which shows up as a rather unusual dirstat: 65% is in arch/x86 (counting the asm-x86 changes too). The rest is mostly random stuff, the appended ShortLog gives a reasonable idea. Several bugzilla entries are hopefully now closed."
"Another week, another -rc," began Linux creator Linus Torvalds, announcing the 2.6.26-rc7 Linux kernel, "and as usual, it's mainly drivers and arch updates - over 90% of changes are in one or the other." He continued:
"A big part of it (about two thirds of the driver update, in fact) is a late-dropping AGP/DRM update that adds support for some new Intel and ATI graphics cards. And a big part of the arch update is the inevitable def_config updates, of course. I'm not all that happy about the timing of the support for the new cards, but at the same time I also hate delaying new drivers. Obviously the hope is that it can't cause any regressions, since the added code is almost entirely for stuff that simply wasn't supported at all before."
Linus concluded, "if you ignore the driver and arch updates, the rest is pretty minor. About half is in networking, and half of the remaining is filesystems updates (mainly ocfs2). And random smatterings elsewhere, like some scheduler updates."
"I'd like to say that the diffs are shrinking and things are calming down, but I'd be lying," began Linux creator Linus Torvalds, announcing the 2.6.26-rc6 kernel. He noted, "another week, another -rc, and another 350 commits. Yes, the diff is smaller than the one from rc4 to rc5 (despite having more commits), so we're on the right trajectory, but I was hoping for less churn at this stage." Linus continued:
"As usual, most of the changes are to drivers (with arch updates a strong second). The DVB updates are the biggest chunk of that, but on the whole it's quite spread out. As mentioned, the diffs are smaller and there are more commits, and yes, most of the commits are really rather small and trivial fixes.
"Give it a try, we should have a few less regressions once more,"
"Another week, another batch of mostly pretty small fixes. Hopefully the regression list is shrinking, and we've fixed at least a couple of the oopses on Arjan's list," said Linux creator Linus Torvalds, announcing the 2.6.26-rc5 kernel. He added, "as usual, the bulk of the changes are in drivers and arch code - together they are about 70% of the diffstat. And the arch stats are bloated by some new/updated SH and avr defconfig files, which is also fairly common at this stage." Linus concluded:
"Perhaps unusually, 13% is in kernel/, almost all of it fixing up some scheduler issues - with the bulk of it by far being a couple of reverts due to performance regressions. But there's a few other fixes too. And then there is networking and some ocfs2 updates. Along with various one-liners sprinkled all around.
"The shortlog (appended) gives a reasonable view of it all. Nothing hugely exciting sticks to my mind, but then I don't think we've had any really hugely exciting problem spots either.."
"You know the drill by now: another week, another -rc," began Linux creator, Linus Torvalds, announcing the 2.6.26-rc4 kernel. "There's a lot of small stuff in here", he continued, "most people won't even notice. The most noticeable thing is for all you 32-bit x86 people who use PAE (enabled by the HIGHMEM64G config option) due to having too much memory in your machine - mprotect() was broken due to some of the PAT fix/cleanup patches, causing the NX bit to be not set correctly." Linus described the fixed bug:
"If you had PAE enabled _and_ a recent enough CPU to have NX, but not recent enough to be 64-bit (or you were just perverse and wanted to run a 32-bit kernel despite having a chip that could do 64-bit and enough memory that you _really_ should have used a 64-bit kernel), you'd get various random program failures with SIGSEGV. It ranged from X not starting up to apparently OpenOffice not working if it did."
He went on to note, "most of the changes, as usual, are in drivers, at 60%, with some DRI changes leading the way (fixing a number of other regressions, mainly by reverting the under-cooked vblank update). Network, MMC, USB, watchdog and IDE drivers also got updates. We had CIFS and NFS updates, and some arch updates as usual." Linus concluded, "nothing really hugely exciting, I think we're doing pretty ok in the release cycle, and I'm getting the feeling that things are calming down."
"This time around, we have 60+% of the changes in drivers, notably drives/video and drivers/media, with some infiniband, networking and usb lovin' to fill things out," began Linux creator Linus Torvalds, announcing the 2.6.26-rc3 kernel. "The rest is (as usual) mostly arch updates," he continued, "this time mostly mips, m68k and uml." Linus noticed that Linux kernel development has been managed with git now as long as it was managed with BitKeeper, a little over three years for both tools. He explained, "the most striking difference has nothing to do with git or BK (the switch-over timing was just the reason I decided to take a look), but with the fact that we're not just continuing to develop, but we're developing faster and with more people," adding:
"So during the three years 2002->2005, we had 63,428 commits, attributed to 1,560 different authors (caveat: misspellings etc will mean that some people get counted more than once). During the last three years, we've had 96,885 attributed to 4,068 distinct authors (with the same caveat, obviously).
"I didn't do a lot of per-commit statistics yet, but from the little I've done it also seems like we've gotten increasingly better at doing small commits (which is probably one of the reasons we have a larger number of them, but also why we have more authors - small commits is how people get into doing kernel development)."
"About 45% architecture updates (counting the include files too), about 30% drivers, and about 25% odds-and-ends. The odds-and-ends are mainly Documentation, filesystems (mostly cifs) and core kernel (scheduler updates etc)," said Linux creator Linus Torvalds, announcing the 2.6.26-rc2 kernel. He added, "if you read the shortlog and get the feeling that most of it is pretty boring small details, you'd be right. There is little exciting there." He continued:
"A fairly small part of it, but quite possibly the most noticeable one, is how the semaphore changes impacted the BKL (the old 'big kernel lock' that is still used for some legacy code, for you non-core people out there), which in the past had different versions ('regular', 'preemptable'). A few months ago we dropped the regular BKL version, but in 2.6.25-rc1 we then had performance (and then correctness) issues with the interaction between the semaphore implementation and the preemptable BKL, so we're back to the old regular version for now."