BSDCan 2008 officially started this morning at 9AM with an opening talk by the event's organizer, Dan Langille. However, in reality the event has already been running for two days, with the FreeBSD tutorials having started on the 14'th. After arriving in Ottawa yesterday afternoon and finding my room in a 20 story University of Ottawa residence, I wandered down to the Royal Oak Pub for early registration, meeting several dozen BSD hackers from all over the world.

This morning's opening talk was well attended, filling up first with clusters of laptop users around the power outlets along both walls. By 15 minutes after the hour, the room was completely full, and Dan started with a humorous slideshow of example letters he's been receiving ever since posting the words "letter of invitation" somewhere on the BSDCan website two year back. Coming primarily from Nigeria, the letter's authors often claim to represent large groups of developers, yet always coming from "disposable" email addresses. After some laughs, he launched into his opening keynote.

"As some of the latency junkies on lkml already know, commit 8e3e076 in v2.6.26-rc2 removed the preemptible BKL feature and made the Big Kernel Lock a spinlock and thus turned it into non-preemptible code again. This commit returned the BKL code to the 2.6.7 state of affairs in essence," began Ingo Molnar. He noted that this had a very negative effect on the real time kernel efforts, adding that Linux creator Linus Torvalds indicated the only acceptable way forward was to completely remove the BKL. Ingo explained:

"This task is not easy at all. 12 years after Linux has been converted to an SMP OS we still have 1300+ legacy BKL using sites. There are 400+ lock_kernel() critical sections and 800+ ioctls. They are spread out across rather difficult areas of often legacy code that few people understand and few people dare to touch. It takes top people like Alan Cox to map the semantics and to remove BKL code, and even for Alan (who is doing this for the TTY code) it is a long and difficult task."

Ingo went on to describe how the BKL works, how it differs from other locking mechanisms, and why this complicates removing it permanently from the kernel. He noted that the various dependencies of the lock are lost in the haze of 15 years of code changes, "all this has built up to a kind of Fear, Uncertainty and Doubt about the BKL: nobody really knows it, nobody really dares to touch it and code can break silently and subtly if BKL locking is wrong." He then suggested "changing the rules of the game", creating a "kill-the-BKL" branch which "turns the BKL into an ordinary albeit somewhat big mutex, with a quirky lock/unlock interface called 'lock_kernel()' and 'unlock_kernel()'."

KernelTrap is excited to be able to offer live coverage of this year's BSDCan 2008 in Ottawa, Canada on May 16th and 17th. The two day conference takes place at the University of Ottawa, and was organized for the fifth consecutive year by Dan Langille who has also made it possible for me to attend and cover the event on KernelTrap. I spoke with Dan to get some background information on the conference, and learn about some of the upcoming highlights.

The event's webpage explains:

"BSDCan, a BSD conference held in Ottawa, Canada, has quickly established itself as the technical conference for people working on and with 4.4BSD based operating systems and related projects. The organizers have found a fantastic formula that appeals to a wide range of people from extreme novices to advanced developers."

"I'm please to announce [the] POHMEL high performance network filesystem. POHMELFS stands for Parallel Optimized Host Message Exchange Layered File System," began Evgeniy Polyakov, explaining:

"This is a high performance network filesystem with local coherent cache of data and metadata. Its main goal is distributed parallel processing of data. Network filesystem is a client transport. POHMELFS protocol was proven to be superior to NFS in lots (if not all, then it is in a roadmap) operations."

This latest release prompted Jeff Garzik to reply, "this continues to be a neat and interesting project :)" New features include fast transactions, round-robin failover, and near-wire limit performance. This adds to existing features which include a local coherent data and metadata cache, async processing of most events, and a fast and scalable multi threaded user space server. Planned features include a server extension to allow mirroring data across multiple devices, strong authentication, and possible data encryption when transferring data over the network. Evgeniy linked to several benchmarks in his blog.

Matthew Dillon sent out a series of updates about his developing HAMMER filesystem, noting that he is currently focusing on the reblocking and pruning code, tracking down a number of bugs resulting in B-Tree corruption. He also noted that previously HAMMER was comprised of three components: B-Tree nodes, records, and data. In his latest cleanups, he has entirely removed the record structure, "this will seriously improve the performance of directory and inode access." This change did require an on-media format change, "I know I have said this before, but there's a very good chance that no more on-media changes will be made after this point. The official freeze of the on-media format will not occur until the 2.0 release, however."

Matt added, "HAMMER is stable enough now that I am able to run it on my LAN backup box. I'm using it to test that the snapshots work as expected as well as to test the long term effects of reblocking and pruning." He then cautioned:

"Please note that HAMMER is not ready for production use yet, there is still the filesystem-full handling to implement and much more serious testing of the reblocking and pruning code is required, not to mention the crash recovery code. I expect to find a few more bugs, but I'm really happy with the results so far."

"About 45% architecture updates (counting the include files too), about 30% drivers, and about 25% odds-and-ends. The odds-and-ends are mainly Documentation, filesystems (mostly cifs) and core kernel (scheduler updates etc)," said Linux creator Linus Torvalds, announcing the 2.6.26-rc2 kernel. He added, "if you read the shortlog and get the feeling that most of it is pretty boring small details, you'd be right. There is little exciting there." He continued:

"A fairly small part of it, but quite possibly the most noticeable one, is how the semaphore changes impacted the BKL (the old 'big kernel lock' that is still used for some legacy code, for you non-core people out there), which in the past had different versions ('regular', 'preemptable'). A few months ago we dropped the regular BKL version, but in 2.6.25-rc1 we then had performance (and then correctness) issues with the interaction between the semaphore implementation and the preemptable BKL, so we're back to the old regular version for now."

"So this merge window was somewhat rocky in the sense that there was a lot of arguments about it, but at the same time I at least personally think that from a technical angle, we had somewhat less scary stuff going on than has been almost the rule lately," noted Linux creator Linus Torvalds, announcing the 2.6.26-rc1 kernel. He continued:

"Lots of changes, but nothing that really feels all that fragile to me. Famous last words. I expect that the x86 PAT support (which has been long in the making) has the potential to have some issues, but the obvious problems were hashed out long ago, and while the merge window already showed one bug, that one was fairly benign and quickly fixed."

Linus highlighted, "another feature that is notable not for its size, but because people have tried to get me to merge it for some long is kgdb support. Which really turned out pretty small and clean, once people started putting their effort into making it so." He concluded, "so go out and test it. The diffstat and shortlogs are too big to post here (7500+ commits and the compressed full patch is 8.5MB in size), but one interesting tidbit I found was that during this *one* merge window, we had almost 800 different authors."

"This is starting to get beyond frustrating for me," complained David Miller of the latest merge window, launching what turned into a very lengthy and ongoing discussion about the Linux kernel development process. The concept of a regular "merge window" was first discussed in July of 2005 with the release of the 2.6.14-rc4 kernel, following the 2005 Developers' Summit. From 2.6.14 on, the release of each official 2.6.y kernel has been followed by a two week period during which major changes are merged into the kernel, followed by a 2.6.y-rc1 release. David complained that this particular merge window has been more painful than others, "the tree breaks every day, and it's becoming an extremely non-fun environment to work in. We need to slow down the merging, we need to review things more, we need people to test their [...] changes!"

During the lengthy discussion, Linux creator Linus Torvalds explained:

"The notion that we should even _try_ to aim to slow things down, that one I find unlikely to be true, and I don't even understand why anybody would find it a logical goal? Of course, you will have fewer new bugs if you have fewer changes. But that's not a goal, that's a tautology and totally uninteresting. A small program is likely to have fewer bugs, but that doesn't make something small 'better' than something large that does more. Similarly, a stagnant development community will introduce new bugs more seldom. But does that make a stagnant one better than a vibrant one? Hell no. So what I'm arguing against here is not that we should aim for worse quality, but I'm arguing against the false dichotomy of believing that quality is incompatible with lots of change."