I got same problem. Even with
Server doesn't accept any new network connections. Rised limits, turned off selinux, simpified pam as much as possible. I got a feeling there might be some kind of hidden limit in a kernel setted up at compile time.
I'm using
Default precompiled kernel.
Thing that confuses me is why there is so little reports about this issue.