[Fedora-directory-users] FDS SSL performance tuning query

[Jonathan Barber]

On Tue, Aug 07, 2007 at 05:22:19PM +0100, Jonathan Barber wrote:
> Hello all, currently we have a FDS instance running on RHEL4 with a
> small number of entries (6,000), we also have a linux compute cluster of
> 100 nodes which uses LDAP for user account data (via libnss_ldap).
> 
> nss_ldap on the cluster is configured to use SSL, and everything is fine
> most of the time. However, occasionally, when a large job is started on
> the cluster, the number of connections increases from 100/minute to
> 1600/minute (26/sec).
> 
> This causes the server to become generally unresponsive, and FDS
> especially so (as judged by the time required to retrieve the DSE via
> TLS). Which is a right pain as it causes our samba PDC to timeout and
> everything goes wrong very quickly.
> 
> I can reproducably, impact on FDS performance by running:
> $ getent passwd | cut -d: -f 1 | while read i; do id $i; done
> 
> across the cluster. When SSL is off, the command to run fine and doesn't
> impact on other searches.

Having done a bit more digging, I discovered that whilst the cluster is
banging on port 389, searches from other hosts to to 389 and TLS on port
389 are fine, but SSL on port 686 hangs. So it looks like it could be
port specific?

Very confused...

> As a short term measure, we've disabled LDAPS on the cluster nodes,
> which is fine as users don't log into them, but we had planned to expand
> the use of LDAP to cover more hosts (Macs and Linux) that require a 
> confidential channal for authentication. So this experience is giving us
> some trepidation about moving forward with that plan.
> 
> Our system is configured following the guidance of the wiki [0], with a
> maximum of 16834 available file descriptors and 50M of cache (more than
> enough to hold the DB) - and the ratio of cache hits/misses look good
> with little paging out. Running logconv.pl on the access logs doesn't
> show any unindexed searches, so that isn't an issue.
> 
> Our server CPU is a 3Ghz Xeon with 1G of RAM, and looking at the
> performance of NSS 3.2 [1], I would expect the machine to be able to
> setup and tear down many more connections than we are currently seeing.
> Indeed, running the test described in [1] with the nss-3.11.4 binaries,
> I get over 1200 connections per second [2], so it certainly doesn't seem
> to be a problem with NSS.
> 
> This suggests to me that the problem lies in FDS somewhere. So, does
> anyone have any suggestions as to how to improve the SSL/TLS performance
> of FDS, or point me at tuning docs for the SSL side of FDS?
> 
> Cheers.
> 
> [0] http://directory.fedoraproject.org/wiki/Performance_Tuning
> [1] http://www.mozilla.org/projects/security/pki/nss/nss-3.2-performance-results
> [2] server$ ./selfserv -n "Server-Cert" -p 6000
>     client$ time ./strsclnt -p 6000 server -c 1000
>     strsclnt: -- SSL: Server Certificate Validated.
>     strsclnt: 0 cache hits; 1 cache misses, 0 cache not reusable
>     strsclnt: 999 cache hits; 1 cache misses, 0 cache not reusable
> 
>     real    0m0.605s
>     user    0m0.795s
>     sys     0m0.226s
> -- 
> Jonathan Barber
> High Performance Computing Analyst
> Tel. +44 (0) 1382 386389
> 
> --
> Fedora-directory-users mailing list
> Fedora-directory-users at redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-directory-users

-- 
Jonathan Barber
High Performance Computing Analyst
Tel. +44 (0) 1382 386389