I once posted a patch to handle this DoS, but, as usual, it wasn't accepted.
Here is an excerpt:
Re: [PATCH 1/1] threads_max: Simple lockout prevention patch
From: Al Boldi <email@example.com>
To: Andrew Morton <firstname.lastname@example.org>
Date: 04/24/06 02:12 pm
Andrew Morton wrote:
Quoting from the 'Resource limits' thread on lkml on 27/09/05:
Then yes, if you set a system-wide limit that is less than the sum of the
limits imposed on each accountable part of the system you can have lock out.
But thats your fault for misconfiguring the system. Don't do that.
-- end of quote