[Fedora-directory-users] FDS SSL performance tuning query

[Jonathan Barber]

On Wed, Aug 08, 2007 at 10:38:58AM -0500, David Bogen wrote:
> We use SSL connections (LDAPS) almost exclusively and have easily
> handled over 7000 SSL connections per minute without extensive tuning of
> FDS.  That particular server is a RHEL4 box running an AMD Opteron with
> 4GB of RAM.
> 
> Even a crusty old PIII (1.2Ghz) running RHEL3 has handled over 1000 SSL
> connections per minute from a high-performance cluster, though I suspect
> that the upper limit of that system isn't too far above that number and
> we are moving beyond it to another 64-bit system.
> 
> Our experience has shown start_tls to be noticeably slower than straight
> ssl; slow enough that the difference is noticeable to people and not
> just to measurements.  I would recommend going with straight SSL and not
> messing around with start_tls.
> 
> If your connections are limited at 1600/minute I wonder if you aren't
> perhaps hitting a limitation elsewhere in your system as our experience
> seems to indicate that FDS can handle the load you are throwing at it.

That was with just one of the NSS strsclnt clients running, with two
clients (from different hosts to a third server) I get 2000/s aggregate.
So I guess I'm reaching an upper bound on the client.

To add another data point to the mess, I now have a situation where if
the cluster libnss_ldap is talking plain LDAP, the LDAP and TLS works
fine, but I can see hangs in the LDAPS. Using the openldap ldapsearch
client with debugging turned on with function tracing (-d 1), shows
blocking at the point where the client says:
...
...
TLS trace: SSL_connect:SSLv2/v3 write client hello A

the next message being (immediately after I stop hammering the LDAP):
TLS trace: SSL_connect:SSLv3 read server hello A
...
...

Which is just weird. LDAP/TLS works fine during this period.

However, the cluster has been more heavily loaded during this period of
testing, so it's entirely possible that I'm not being able to generate
enough LDAP requests to provoke the behaviour I was seeing the other
day.

It's encouraging to hear that it is possible to get more SSL/TLS
connections though.

David Boreham is correct in the assumption that the LDAP server CPU is
pegged at 99%, when this is all going on. Running pstack hasn't got me
anything yet, as it just shows gdb running at 99% CPU and the LDAP goes
unresponsive and needs to be restarted. Given that this is a production
service - I'm somewhat adverse to leaving it in that state for too long :)

Setting up a test environment is something that's going to me a while to
get together, as I have training and planned work over the next couple
of weeks. I'll try and report back any progress when I make any.

Thanks for everyone's suggestions.

> David
> 
> -- 
> David Bogen   :: (608) 263-0168
> Unix SysAdmin :: IceCube Project
> david.bogen at icecube.wisc.edu
> 
> --
> Fedora-directory-users mailing list
> Fedora-directory-users at redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-directory-users

-- 
Jonathan Barber
High Performance Computing Analyst
Tel. +44 (0) 1382 386389