In April, 2.4 kernel maintainer Willy Tarreau queried the Linux kernel mailing list regarding how the 2.4 kernel is still being used. He followed up summarizing the responses, suggesting that about 5% of 2.4 users run the kernel on old recycled laptops at home or on PDA's and thin clients, running whatever works with no real need to upgrade. Another 5% of the users are on desktop PCs and monitoring stations, not upgrading because "it works". From there, about 50% of the users run the 2.4 kernel on general purpose servers and update regularly, still running the older kernel due to lack of need for new features and lack of time, and possibly due to failed earlier attempts to upgrade. Another 20% use the 2.4 kernel on application specific servers where reliability is of the highest importance. 10% of the users run the older stable kernel on routers, firewalls, and intrusion detections systems, with 1 to 2 year uptimes and limited updates in a business setting, and shorter uptimes and frequent updates in a personal setting. The final 10% or so use the kernel in embedded systems where stability is again very important, and the build tree may be highly modified, causing at least one major network equipment manufacturer to still be shipping devices with the 2.4.2 kernel. Willy continued:

"Based on that and on the workflow people took the time to explain, I realize that the distinction between -pre and -rc is useless (ding! Linus if you read this, don't beat me). In fact, either people want absolute reliability and they pick one kernel from the stable 2.4.X.Y branch, or they want more recent updates and they simply do their shopping in the -master branch, which is fairly easy thanks to the Gitweb interface. But there are almost no testers in 2.4, just users. That means that I don't have to expect immediate feedback when posting a pre-release. And it has happened several times that I got a build error report several weeks after the release.

"Also, since most people do not update more than 1 - 2 times a year, it's not very useful to have more than 1 - 2 new versions a year, especially since we have the stable release. For this reason, I think I will issue stable releases a bit more often for users to quickly get their fixes, but progressively increase the delay between major releases. Those ones will only be issued with new PCI IDs, major driver updates, compiler support, etc..."

"New year, new kernel: Linux 2.4.36 is finally ready and has been checked long enough to be released. Quite a bunch of bugs, build errors and security issues have been fixed since 2.4.35, but all of those fixes were merged into 2.4.35-stable," 2.4 maintainer Willy Tarreau stated, announcing the latest 2.4 stable Linux kernel. He noted, "I should say that I'm quite satisfied of this dual-branch release model which proves to be very successful at separating quick fixes from changes which require more thorough testing." Willy went on to add:

"Concerning future versions, I have nothing pending in the queue anymore. I will then go on with 2.4.36.X when bug fixes come in, and only open 2.4.37 when I get something which I do not consider suitable for 2.4.36.X."

The previous 2.4.35 stable kernel was released in July of 2007. Source level changes can be viewed through the linux-2.4 gitweb interface.

"I have ported [the] adutux driver for [the] ADU series device list from 2.6 to 2.4," Vitaliy Ivanov announced on the Linux USB development mailing list. 2.4 maintainer Willy Tarreau explained that while the backport looked good, as a rule it was best to not merge new drivers into the stable 2.4 branch of the Linux kernel, "2.4 is currently used by people who already are in 2.4 and cannot/do not want to switch, and by people who are looking for close to zero maintenance. Drivers are often a reason to switch away from 2.4, but not to stay in 2.4."

One of the arguments for merging the driver into the mainline 2.4 tree was so it would then be added to the various distribution kernels. Pete Zaitcev explained that this wasn't the way it worked with enterprise kernels, "at least in case of RHEL, such backports never were automatic. In any case, RHEL 2.1 and 3 do not receive new drivers anymore. We only do bugfixes if something comes up. Realistically speaking, 2.4 kernels are just too old for anyone to use." Willy did agree to consider merging the driver if Vitaliy wanted to step up as the official maintainer for the 2.4 backport.

"I've just released Linux 2.4.36-pre1," announced 2.4 maintainer Willy Tarreau. He described a new feature found in the first pre-release:

"In private discussions, Solar Designer proposed to restrict the ability to map the NULL address to CAP_RAW_IO capable processes only. The idea behind this was to prevent 'normal' users from trying to exploit NULL dereferences in the kernel which have not been discovered yet. This is purely a preventive measure."

Willy added that a similar feature exists in the 2.6 kernel, "Chris Wright noted that 2.6 already has a somewhat similar feature brought by Eric Paris, which introduces a sysctl by which the admin can set the lower mappable address." He also noted that the feature can be disabled, "Alan Cox indicated that it was desirable to be able to dynamically disable the feature because some (very) rare programs map the NULL pointer to speed up their walk through linked lists by avoiding NULL pointer checks." Finally he asked for testers, "please report any breakage you would detect when enabling it. We're pretty confident that almost every applications will not see any difference since no normal application maps NULL. But it would be interesting to identify the "special" ones."

"After 6 months of careful integration and testing, I'm happy to announce availability of Linux 2.4.35," 2.4 maintainer Willy Tarreau announced on the lkml. This is the second stable 2.4 kernel released since Willy became the 2.4 kernel maintainer nearly a year ago in August of 2006. Source level changes can be viewed through the linux-2.4 gitweb interface. Willy added:

"I'm very conscious that 2.4 has mostly left desktop PCs and notebooks, but it's still commonly found on servers, route reflectors or firewalls. For this reason, I'm open to merge the small updates required to maintain such systems running (eg: PCI IDs and such), but I will generally refuse all patches which add support for new desktop or notebook-specific hardware, unless the people present very convincing arguments. Those people generally would better upgrade their systems to 2.6."

The 2.4 stable kernel tree has been maintained by Willy Tarreau for a year, since July of 2006 [story]. When recently asked if the tree had been abandoned, Willy replied, "no it's not abandoned at all! The difficulty with 2.4 is to get user feedback on patches. While in 2.6, there are hundreds or thousands of testers for every release, in 2.4, I have to wait longer after every release in order to start collecting problem reports, or confirmation of fixes. People using it in production generally cannot reboot a machine in the evening just to try a patch. Also, subsystem maintainers have less time to spend on it and are themselves slowed down by the slow feedback process."

As to the upcoming 2.4.35 kernel, Willy noted, "I still have a few patches to merge, and I expect to be able to work on them next week-end, then release -rc1, which I expect to be the only -rc before -final (which will probably not be true as in every -rc)."

2.4 kernel maintainer [story] Willy Tarreau ran some tests to compare Con Kolivas [interview]'s Staircase Deadline CPU scheduler [story] with Ingo Molnar [interview]'s new Completely Fair Scheduler [story]. He summarized his experiences:

"I think that CFS is based on a more promising concept but is less mature and is dangerous right now with certain workloads. SD shows some strange behaviours like not using all CPU available and a little jerkyness, but is more robust and may be the less risky solution for a first step towards a better scheduler in mainline, but it may also probably be the last O(1) scheduler, which may be replaced sometime later when CFS (or any other one) shows at the same time the smoothness of CFS and the robustness of SD."

Some debate was raised by logic added since CFS version 3 to automatically nice the X process for better GUI responsiveness. The CFS changelog comment labels the change as a usibility fix explaining, "automatic renicing of kernel threads such as keventd, OOM tasks and tasks doing privileged hardware access (such as Xorg)." Ingo posted a standalone patch demonstrating how these processes are detected and automatically niced, offering it for inclusion into Con's Staircase scheduler. Willy concurred that it was a good idea, "I think it could be a good idea since you recommend to renice X with SD. Most of the problem users are facing with renicing X is that they need to change their configs or scripts. If the kernel can reliably detect X and handle it differently, why not do it ?" Con was less convinced, "hmm well I have tried my very best to do all the changes without changing 'policy' as much as possible since that trips over so many emotive issues that noone can agree on, and I don't have a strong opinion on this as I thought it would be better for it to be a config option for X in userspace instead. Either way it needs to be turned on/off by admin and doing it by default in the kernel is... not universally accepted as good."

2.4 kernel maintainer Willy Tarreau [story] announced the release of the 2.4.34 stable Linux kernel, "2.4.34 brings the usual bunch of security fixes, bugfixes, and adds support for gcc 4 to x86, x86-64 and sparc64, thanks to Mikael Pettersson's work." Willy also released the 2.4.33.7 kernel with a security fix added in 2.4.34-rc3. He went on to note some caveats:

"One user reported regular panics with aacraid since 2.4.32, so there's no regression here. I will seek for some help to get this fixed in 2.4.35. I also get reports of people getting trapped by NIC vendors who suddenly change their ethernet chips with no big warning notice. The i82546GB chip which replaced the i82546EB in e1000 cards come to mind. It is not supported by the driver in 2.4.34 but I will try to solve this in 2.4.35 (right now, you have to download the vendor's drivers when you replace a NIC). Another driver should get some lifting : skge. I have got a few reports of problems with the vendor's sk98lin driver and I noticed the same problems at work (UDP becoming silent on NFS server)."

Willy Tarreau replaced Marcelo Tosatti [interview] as the 2.4 stable Linux kernel maintainer in August of 2006 [story]. In response to a series of compilation fixes sent to the lkml by Mariusz Kozlowski, Willy suggested that all patches would be postponed until 2.4.34 is released. He suggested that in the interum the appropriate subsystem maintainers should be contacted to determine whether or not each of the patches should be merged, "we would merge the accepted patches and those without any reply which we consider relevant early in the 35-pre cycle so that people have some time to inform us about the potential conflicts they encounter."

Willy went on to describe how to determine who maintains each of the files, "check for the maintainer in the files themselves, or in the MAINTAINERS file. As a rule of thumb, if a file has not changed in the last 3 years and its maintainer is not one of the active ones you regularly see posting on LKML, then there are great chances that the file is unmaintained." He continued, "generally, the core subsystems (network, filesystems, archs, ...) are still well maintained by people who *really want* to validate the patches before forwarding them upstream." He went on to note, "they're all busy, so make the question simple enough so that they can quickly reply with ACK/NACK/QUEUED. Keep me CCed so that you don't have to forward me the response. Generally, they will reply within one week." Willy concluded, "if there's no easily identifiable maintainer anymore, or if some maintainers don't reply, then it becomes my job."

Hans Reiser formed Namesys and began the development of Reiserfs ten years ago. The first release of the filesystem, Reiser3, is part of the mainline 2.4 and 2.6 Linux kernels. The more recent Reiser4 is a complete redesign and reimplementation of Reiserfs, aiming to soon be merged into the mainline 2.6 Linux kernel.

In this interview, Hans discusses his background and how he came to create Namesys and Reiserfs. He looks back at Reiser3, describing the advantages it had over other filesystems when it was released and its current state. He then explores the many improvements currently in Reiser4, describing the plugin architecture and its exciting potential for future semantic enhancements.

Continuing the earlier discussion about low latency and Ingo Molnar [interview]'s voluntary kernel preemption patch [story], the conversation moved onto the affect a filesystem can have on latency. Specifically, 2.6 maintainer Andrew Morton [interview] noted that ReiserFS was known to have some latency issues in both the 2.4 and 2.6 Linux kernels, "resierfs: yes, it's a problem. I 'fixed' it multiple times in 2.4, but the fixes ended up breaking the fs in subtle ways and I eventually gave up." However, he did go on to note, "actually, the 2.4 low-latency patch does still have some reiserfs fixes, so it's probably better than reiserfs in 2.6."

When asked if ext3 was a better choice for low latency work, Andrew Morton replied, "ext3 is certainly better than [reiserfs], but still has a couple of potential problem spots. ext2 is probably the best at this time." Data is continuing to be collected and reviewed by a number of kernel developers, so the more noticeable latency issues in the 2.6 kernel will likely be addressed soon.

Andrea Arcangeli is well known for having completely rewritten and stabilized the virtual memory subsystem in the 2.4 Linux kernel. Many were surprised when Linus Torvalds merged Andrea's VM into 2.4.10, but the new memory subsystem has long since proved itself. Andrea is a 27 year old Linux kernel hacker living in Italy and working for SUSE.

Marcelo Tosatti became the maintainer of the 2.4 stable kernel when he was 18 years old in November of 2001. His first kernel release was 2.4.16 on November 26'th which very quickly followed the earlier 2.4.15 to address an issue with filesystem corruption. Two years later, he has recently released 2.4.23 and plans to soon put the 2.4 stable kernel into maintenance mode, only addressing bugs and security issues.

Living in Brazil, Marcelo currently works for Cyclades Corporation. In this interview he looks at how he became the 2.4 maintainer, the challenges involved, and brings us up to date with the current status of the 2.4 kernel.

Kerneltrap has spoken with Linux guru Alan Cox. He is perhaps the second most influential Linux kernel hacker, next only to Linus. In this interview he talks about himself, his history with computers and Linux, working for Red Hat, Marcello and the 2.4 kernel, the DMCA, the future of Linux and much more.