Another thread discussed potentially merging the swap prefetch patch into the mainline Linux kernel. Con Kolivas [story] started the thread saying "I fixed all bugs I could find and improved it as much as I could last kernel cycle. Put me and the users out of our misery and merge it now or delete it forever please." Replying to an off-list message, Andrew Morton asked users of the patch, "please provide us more details on your usage and testing of that code. Amount of memory, workload, observed results, etc?"
Nick Piggin [interview] noted that he's still interested in better understanding and possibly fixing what's happening with swap and reclaim on the systems reporting a benefit from the swap-prefetch patch. He went on to note, "regarding swap prefetching. I'm not going to argue for or against it anymore because I have really stopped following where it is up to, for now. If the code and the results meet the standard that Andrew wants then I don't particularly mind if he merges it. It would be nice if some of you guys would still report and test problems with reclaim when prefetching is turned off -- I have never encountered the morning after sluggishness (although I don't doubt for a minute that it is a problem for some)." Ingo Molnar followed up to these coments acking the patch, "I have tested it and have read the code, and it looks fine to me. (i've reported my test results elsewhere already [story]) We should include this in v2.6.23."
From: Con Kolivas [email blocked] Subject: Re: -mm merge plans for 2.6.23 Date: Tue, 10 Jul 2007 20:15:43 +1000 On Tuesday 10 July 2007 18:31, Andrew Morton wrote: > When replying, please rewrite the subject suitably and try to Cc: the > appropriate developer(s). ~swap prefetch Nick's only remaining issue which I could remotely identify was to make it cpuset aware: http://marc.info/?l=linux-mm&m=117875557014098&w=2 as discussed with Paul Jackson it was cpuset aware: http://marc.info/?l=linux-mm&m=117895463120843&w=2 I fixed all bugs I could find and improved it as much as I could last kernel cycle. Put me and the users out of our misery and merge it now or delete it forever please. And if the meaningless handwaving that I 100% expect as a response begins again, then that's fine. I'll take that as a no and you can dump it. -- -ck -
From: Andrew Morton [email blocked] Subject: Re: [ck] Re: -mm merge plans for 2.6.23 Date: Tue, 10 Jul 2007 18:14:19 -0700 On Wed, 11 Jul 2007 11:02:56 +1000 "Matthew Hawkins" [email blocked] wrote: > We all know swap prefetch has been tested out the wazoo since Moses was a > little boy, is compile-time and runtime selectable, and gives an important > and quantifiable performance increase to desktop systems. Always interested. Please provide us more details on your usage and testing of that code. Amount of memory, workload, observed results, etc? > Save a Redhat > employee some time reinventing the wheel and just merge it. This wheel > already has dope 21" rims, homes ;-) ooh, kernel bling.
From: Ira Snyder [email blocked] Subject: Re: [ck] Re: -mm merge plans for 2.6.23 Date: Tue, 10 Jul 2007 20:21:23 -0600 On Tue, 10 Jul 2007 18:14:19 -0700 Andrew Morton [email blocked] wrote: > On Wed, 11 Jul 2007 11:02:56 +1000 "Matthew Hawkins" [email blocked] wrote: > > > We all know swap prefetch has been tested out the wazoo since Moses was a > > little boy, is compile-time and runtime selectable, and gives an important > > and quantifiable performance increase to desktop systems. > > Always interested. Please provide us more details on your usage and > testing of that code. Amount of memory, workload, observed results, > etc? > I often leave long compiles running overnight (I'm a gentoo user). I always have the desktop running, with quite a few applications open, usually firefox, amarok, sylpheed, and liferea at the minimum. I've recently tried using a "stock" gentoo kernel, without the swap prefetch patch, and in the morning when I get on the computer, it hits the disk pretty hard pulling my applications (especially firefox) in from swap. With swap prefetch, the system responds like I expect: quick. It doesn't hit the swap at all, at least that I can tell. Swap prefetch definitely makes a difference for me: it makes my experience MUCH better. My system is a Core Duo 1.83GHz laptop, with 1GB ram and a 5400 rpm disk. With the disk being so slow, the less I hit swap, the better. I'll cast my vote to merge swap prefetch.
From: timotheus [email blocked] Subject: Re: [ck] Re: -mm merge plans for 2.6.23 Date: Tue, 10 Jul 2007 23:37:23 -0400 Ira Snyder [email blocked] writes: >> Always interested. Please provide us more details on your usage and >> testing of that code. Amount of memory, workload, observed results, >> etc? >> > > I often leave long compiles running overnight (I'm a gentoo user). I > always have the desktop running, with quite a few applications open, > usually firefox, amarok, sylpheed, and liferea at the minimum. I've > recently tried using a "stock" gentoo kernel, without the swap > prefetch patch, and in the morning when I get on the computer, it hits > the disk pretty hard pulling my applications (especially firefox) in > from swap. With swap prefetch, the system responds like I expect: > quick. It doesn't hit the swap at all, at least that I can tell. > > Swap prefetch definitely makes a difference for me: it makes my > experience MUCH better. > > My system is a Core Duo 1.83GHz laptop, with 1GB ram and a 5400 rpm > disk. With the disk being so slow, the less I hit swap, the better. > > I'll cast my vote to merge swap prefetch. Very similar experiences. Other usage patterns that swap prefetch can cause improvements with: - Idling VMware session with large memory. Since VMware (server) can use mixed swap/RAM, the prefetch allows it swap back into RAM without having to make the application active in the foreground. - Firefox, OO Office, long from-source compilations, all of the normal. - My largest RAM capacity machine is a Core 2 Duo Laptop with 2 GB of RAM. It still benefits from the prefetch after running long compilations or backups. - Also, I have an old Pentium 4 server (1.3 GHz, original RDRAM, ...) that uses the CK patches including swap prefetch. It has only 640 MB of RAM, and runs GBytes of data backup every night. The swap is split among multiple disks, and can easily fill .5 GBytes over night. Applications that run in a VNC session, web browsers, office programs, etc., all resume much faster with the prefetch. Even the intial ssh-login appears snappier; but I think that is just CK's fine work elsewhere. :) I am curious how much of the benefit is due to prefetch, or due to CK using `vm_mapped' rather than `vm_swappiness'. Swappiness always seemed such an unbenificial hack to me... (The past 6 months I've tried weeks/months of using various kernels, -mm, -ck, vanilla, genpatches, combinations there of -- x86 and ppc.) I vote for prefetch and `vm_mapped'.
From: Grzegorz Kulewski [email blocked] Subject: Re: [ck] Re: -mm merge plans for 2.6.23 Date: Wed, 11 Jul 2007 05:59:05 +0200 (CEST) On Tue, 10 Jul 2007, Andrew Morton wrote: > On Wed, 11 Jul 2007 11:02:56 +1000 "Matthew Hawkins" [email blocked] wrote: > >> We all know swap prefetch has been tested out the wazoo since Moses was a >> little boy, is compile-time and runtime selectable, and gives an important >> and quantifiable performance increase to desktop systems. > > Always interested. Please provide us more details on your usage and > testing of that code. Amount of memory, workload, observed results, > etc? I am using swap prefetch in -ck kernels since it was introduced. My machine: Athlon XP 2000MHz, 1GB DDR 266, fast SATA disk, different swap configurations but usually heaps of swap (2GB and/or 8GB). My workload: desktop usage, KDE, software development, Firefox (HUGE memory hog), Eclipse and all that stuff (HUGE memory hog), sometimes other applications, sometimes some game such as Americas Army (that one will eat all your memory in any configuration), Konsole with heaps of tabs, usually some heavy compilations in the background. Observed result (of not broken swap prefetch versions): after closing some memory hog (for example stopping playing game and starting to write some code or reloading Firefox after it leaked enough memory to nearly bring the system down) the disk will work for some time and after that everything works as expected, no heavy swap-in when switching between applications and so on, nearly no lags in desktop usage. This is nearly unnoticable. Unless I have to run pure mainline. In that case I can notice that swap prefetch is off very quickly because after closing such memory hog and returning to some other application the system is slow for long time. Worse: after it starts to work reasonably and I try to switch to some other application or even try to use some dialog window or module of current application I have to wait, sometimes > 10s for it to swap back in (even if 70% of my RAM is free at that time, after memory hog is gone). It is painfull. I observed similar results on my laptop (Athlon 64, 512MB RAM, slow ATA disk, similar workload but reduced because hardware is weak). For me swap prefetch makes huge difference. The system lags a lot less in such circumstances. Personaly I think swap prefetch is a hack. Maybe not very dirty and ugly but still a hack. But since: * nobody proposed anything that can replace it and can be considered a no-hack, * swap prefetch is rather well tested and shouldn't cause regressions (no known regressions as far as I know, the patch does not look very invasive, was reviewed several times, ...), * Con said he won't make further -ck relases and won't port these patches to newer kernels, * there are at least several people who see the difference, * if somebody really hates it (s)he can turn it off I think it could get merged, at least temporarily, before somebody can suggest some better or extended solution. Personaly I would be very happy to see it in so people like me don't have to patch it in or (worse) port it (possibly causing bugs and filling additional bug reports and asking additional questions on these lists). I even wonder if adding the opposite of swap prefetch too wouldn't be even better for many workloads. Something like: "when system and swap-disk is idle try to copy some pages to swap so when system needs memory swap-out could be much cheaper". I suspect patch like that can reduce startup times (and other operations) of great memory hogs because disk (the slowest device) will only have to read the application and won't have to swap-out half of the RAM at the same time. I am happy to provide further info if needed. Thanks, Grzegorz Kulewski
From: "Matthew Hawkins" [email blocked] Subject: Re: [ck] Re: -mm merge plans for 2.6.23 Date: Wed, 11 Jul 2007 12:54:40 +1000 On 7/11/07, Andrew Morton [email blocked] wrote: > On Wed, 11 Jul 2007 11:02:56 +1000 "Matthew Hawkins" [email blocked] wrote: > Always interested. Please provide us more details on your usage and > testing of that code. Amount of memory, workload, observed results, > etc? My usual workstation has 1Gb of ram & 2Gb of swap (single partition - though in the past with multiple drives I would spread swap around the less-used disks & fiddle with the priority). Its acting as server for my home network too (so it has squid, cups, bind, dhcpd, apache, mysql & postgresql) but for the most part I'll have Listen playing music while I switch between Flock &/or Firefox, Thunderbird, and xvncviewer. On the odd occasion I'll fire up some game (gewled, actioncube, critical mass). Compiling these days has been mostly limited to kernels, I've been building mostly -ck and -cfs - keeping up-to-date and also doing some odd things (like patching the non-SD -ck stuff on top of CFS). Mainly just to get swap prefetch, but also not to lose skills since I'm out of the daily coding routine now. Anyhow with swap prefetch, applications that may have been sitting there idle for a while become responsive in the single-digit seconds rather than double-digit or worse. The same goes for a morning wakeup (ie after nightly cron jobs throw things out) and also after doing basically anything that wants memory, like benchmarking the various kernels I'm messing with or doing some local DB work or coding a memory leak into a web application running under apache ;) -- Matt
From: Nick Piggin [email blocked] Subject: Re: [ck] Re: -mm merge plans for 2.6.23 Date: Wed, 11 Jul 2007 15:18:51 +1000 Matthew Hawkins wrote: > On 7/11/07, Andrew Morton [email blocked] wrote: > Anyhow with swap prefetch, applications that may have been sitting > there idle for a while become responsive in the single-digit seconds > rather than double-digit or worse. The same goes for a morning wakeup > (ie after nightly cron jobs throw things out) OK that's a good data point. It would be really good to be able to do an analysis on your overnight IO patterns and the corresponding memory reclaim behaviour and see why things are getting evicted. Not that swap prefetching isn't a good solution for this situation, but the fact that things are getting swapped out for you also means that mapped files and possibly important pagecache and dentries are being flushed out, which we might be able to avoid. Thanks, Nick -- SUSE Labs, Novell Inc.
From: "Ray Lee" [email blocked] Subject: Re: [ck] Re: -mm merge plans for 2.6.23 Date: Tue, 10 Jul 2007 22:47:04 -0700 On 7/10/07, Nick Piggin [email blocked] wrote: > Matthew Hawkins wrote: > > On 7/11/07, Andrew Morton [email blocked] wrote: > > > Anyhow with swap prefetch, applications that may have been sitting > > there idle for a while become responsive in the single-digit seconds > > rather than double-digit or worse. The same goes for a morning wakeup > > (ie after nightly cron jobs throw things out) > > OK that's a good data point. It would be really good to be able to > do an analysis on your overnight IO patterns and the corresponding > memory reclaim behaviour and see why things are getting evicted. Eviction can happen for multiple reasons, as I'm sure you're painfully aware. It can happen because of poor balancing choices, or it can happen because the system is just short of RAM for the workload. As for the former, you're absolutely right, it would be good to know where those come from and see if they can be addressed. However, it's the latter that swap prefetch can help and no amount of fiddling with the aging code can address. As an honest question, what's it going to take here? If I were to write something that watched the task stats at process exit (cool feature, that), and recorded the IO wait time or some such, and showed it was lower with a kernel with the prefetch, would *that* get us some forward motion on this? I mean, from my point of view, it's a simple mental proof to show that if you're out of RAM for your working set, things that you'll eventually need again will get kicked out, and prefetch will bring those back in before normal access patterns would fault them back in under today's behavior. That seems like an obvious win. Where's the corresponding obvious loss that makes this a questionable addition to the kernel? Ray
From: Nick Piggin [email blocked] Subject: Re: [ck] Re: -mm merge plans for 2.6.23 Date: Wed, 11 Jul 2007 16:00:05 +1000 Ray Lee wrote: > As an honest question, what's it going to take here? If I were to > write something that watched the task stats at process exit (cool > feature, that), and recorded the IO wait time or some such, and showed > it was lower with a kernel with the prefetch, would *that* get us some > forward motion on this? Honest answer? Sure, why not. Numbers are good. -- SUSE Labs, Novell Inc.
From: Nick Piggin [email blocked] Subject: Re: [ck] Re: -mm merge plans for 2.6.23 Date: Wed, 11 Jul 2007 15:54:43 +1000 Ray Lee wrote: > On 7/10/07, Nick Piggin [email blocked] wrote: > >> Matthew Hawkins wrote: >> > On 7/11/07, Andrew Morton [email blocked] wrote: >> >> > Anyhow with swap prefetch, applications that may have been sitting >> > there idle for a while become responsive in the single-digit seconds >> > rather than double-digit or worse. The same goes for a morning wakeup >> > (ie after nightly cron jobs throw things out) >> >> OK that's a good data point. It would be really good to be able to >> do an analysis on your overnight IO patterns and the corresponding >> memory reclaim behaviour and see why things are getting evicted. > > > Eviction can happen for multiple reasons, as I'm sure you're painfully > aware. It can happen because of poor balancing choices, or it can s/balancing/reclaim, yes. And for the nightly cron job case, this is could quite possibly be the cause. At least updatedb should be fairly easy to apply use-once heuristics for, so if they're not working then we should hopefully be able to improve it. -- SUSE Labs, Novell Inc.
From: "Ray Lee" [email blocked] Subject: Re: [ck] Re: -mm merge plans for 2.6.23 Date: Tue, 10 Jul 2007 23:04:59 -0700 On 7/10/07, Nick Piggin [email blocked] wrote: > >> OK that's a good data point. It would be really good to be able to > >> do an analysis on your overnight IO patterns and the corresponding > >> memory reclaim behaviour and see why things are getting evicted. > > > > Eviction can happen for multiple reasons, as I'm sure you're painfully > > aware. It can happen because of poor balancing choices, or it can > > s/balancing/reclaim, yes. And for the nightly cron job case, this is > could quite possibly be the cause. At least updatedb should be fairly > easy to apply use-once heuristics for, so if they're not working then > we should hopefully be able to improve it. <nod> Sorry, I'm not so clear on the terminology, am I. So, that's one part of it: one could argue that for that bit swap prefetch is a bit of a band-aid over the issue. A useful band-aid, that works today, isn't invasive, and can be ripped out at some future time if the underlying issue is eventually solved by a proper use-once aging mechanism, but nevertheless a band-aid. The other part is when I've got evolution and a few other things open, then I run gimp on a raw photo and do some work on it, quit out of gimp, do a couple of things in a shell to upload the photo to my server, then switch back to evolution. Hang, waiting on swap in. Well, the kernel had some free time there to repopulate evolution's working set, and swap prefetch would help that, while better (or perfect!) heuristics in the reclaim *won't*. That's the real issue here.
From: Nick Piggin [email blocked] Subject: Re: [ck] Re: -mm merge plans for 2.6.23 Date: Wed, 11 Jul 2007 16:24:04 +1000 Ray Lee wrote: > On 7/10/07, Nick Piggin [email blocked] wrote: > >> >> OK that's a good data point. It would be really good to be able to >> >> do an analysis on your overnight IO patterns and the corresponding >> >> memory reclaim behaviour and see why things are getting evicted. >> > >> > Eviction can happen for multiple reasons, as I'm sure you're painfully >> > aware. It can happen because of poor balancing choices, or it can >> >> s/balancing/reclaim, yes. And for the nightly cron job case, this is >> could quite possibly be the cause. At least updatedb should be fairly >> easy to apply use-once heuristics for, so if they're not working then >> we should hopefully be able to improve it. > > > <nod> Sorry, I'm not so clear on the terminology, am I. > > So, that's one part of it: one could argue that for that bit swap > prefetch is a bit of a band-aid over the issue. A useful band-aid, > that works today, isn't invasive, and can be ripped out at some future > time if the underlying issue is eventually solved by a proper use-once > aging mechanism, but nevertheless a band-aid. I think for some workloads it is probably a bandaid, and for others the concept of prefetching likely to be used again data back in is undeniably going to be a win for others. A lot of postitive reports I have seen about this say that desktop the next morning is more responsive. So I kind of want to know what's happening here -- as far as I can tell, swap prefetching shouldn't help a huge amount to recover from a simple updatedb alone -- although if other cron stuff happened that used a bit more memory afterwards and pushing out some of updatedb's cache, perhaps that's when swap prefetching finds its niche. I don't know. However, I don't like the fact that there is _any_ swap happening on 1GB desktops after a single updatedb run. Is something else running that hogs a huge amount of memory? Maybe that explains it, but I don't know. I do know that we probably don't do very good use-once algorithms on the dentry and inode caches, so updatedb might cause them to push swap out. We could test that by winding the vfs reclaim right up. > The other part is when I've got evolution and a few other things open, > then I run gimp on a raw photo and do some work on it, quit out of > gimp, do a couple of things in a shell to upload the photo to my > server, then switch back to evolution. Hang, waiting on swap in. Well, > the kernel had some free time there to repopulate evolution's working > set, and swap prefetch would help that, while better (or perfect!) > heuristics in the reclaim *won't*. > > That's the real issue here. Yeah that's an issue, and swap prefetching has the potential to help there no doubt at all. How much is the saving? I don't think it will be like an order of magnitude because unfortunately we also get mapped pagecache being thrown out as well as swap, so for example all your evolution mailbox, libraries, executable, etc. is still going to have to be paged back in. Regarding swap prefetching. I'm not going to argue for or against it anymore because I have really stopped following where it is up to, for now. If the code and the results meet the standard that Andrew wants then I don't particularly mind if he merges it. It would be nice if some of you guys would still report and test problems with reclaim when prefetching is turned off -- I have never encountered the morning after sluggishness (although I don't doubt for a minute that it is a problem for some). -- SUSE Labs, Novell Inc.
From: Ingo Molnar [email blocked] Subject: Re: swap prefetch (Re: -mm merge plans for 2.6.23) Date: Wed, 11 Jul 2007 09:50:53 +0200 * Nick Piggin [email blocked] wrote: > Regarding swap prefetching. I'm not going to argue for or against it > anymore because I have really stopped following where it is up to, for > now. If the code and the results meet the standard that Andrew wants > then I don't particularly mind if he merges it. I have tested it and have read the code, and it looks fine to me. (i've reported my test results elsewhere already) We should include this in v2.6.23. Acked-by: Ingo Molnar [email blocked] Ingo
