[Fedora-directory-users] FDS SSL performance tuning query

[Gordon Messmer]

Jonathan Barber wrote:
> Hello all, currently we have a FDS instance running on RHEL4 with a
> small number of entries (6,000), we also have a linux compute cluster of
> 100 nodes which uses LDAP for user account data (via libnss_ldap).
> 
> nss_ldap on the cluster is configured to use SSL, and everything is fine
> most of the time. However, occasionally, when a large job is started on
> the cluster, the number of connections increases from 100/minute to
> 1600/minute (26/sec).

Use nscd.  I know you said that you'd rather avoid it, but the 
performance penalty of LDAP without NSCD is significant, to say the 
least.  Even plain LDAP is terribly expensive, since each process that 
needs NSS info will go through the connection process for its lookups. 
At best that means that you'll see a high rate of connections on your 
directory server, which will drive its load up.  Worse, you're likely to 
see the connection rate kept very high *and* a large number of open 
connections on all of your hosts (since each process that does a lookup 
will keep its connection open).

Using nscd, most lookups will be done locally, and connections will be 
pooled.  Lookup latency will be reduced on your clients, and both 
connection rate and the number of open connections on your server will 
diminish greatly.

The cost of nscd is that data may not update on your clients 
immediately, but how often are you going to change a user's uidNumber or 
homeDirectory?  I've never seen the cost of nscd outweigh the tremendous 
benefits.

> I can reproducably, impact on FDS performance by running:
> $ getent passwd | cut -d: -f 1 | while read i; do id $i; done

Well, yes.  If you're doing roughly 6000 connections over SSL on 100 
machines concurrently, it's going to impact performance in a bad way.

If you only get 30 SSL connections per second (as seems likely given 
that 1000 connections takes ~30 seconds, per a later message in this 
thread), there may be a flaw in FDS.  I haven't tested its SSL 
connection rate personally, so I have nothing against which to compare. 
   (Since I use Kerberos for authentication, I don't use SSL.) In any 
case, I'd expect that job to take more than 30 minutes.

If you want to increase SSL connections per second, you can use an SSL 
accelerator to proxy the SSL connections.

> Our system is configured following the guidance of the wiki [0]
 > [0] http://directory.fedoraproject.org/wiki/Performance_Tuning

That document *sucks*.  First, it sucks because it is terribly 
incomplete, leaving out some rather critical tuning parameters.  Second, 
it sucks because it mixes OS tuning and directory server tuning with 
little discussion of what each change accomplishes, and how to correlate 
that with expected use.  Third, it sucks because the data is outdated 
and may actually affect performance *adversely* on Linux if followed 
completely.  Fourth, it sucks because many of the settings recommended 
(specifically, the limits.conf and profile modifications) won't affect 
the directory server at all.  Fifth, it sucks because better 
documentation exists, and this document distracts users from it:
http://www.redhat.com/docs/manuals/dir-server/ag/7.1/dsmanage.html

Personally, I'd delete the wiki document entirely, and link directly to 
this chapter of the Admin Guide.  (But I'd note that Sun's documentation 
is much better at discussing exactly how to size the caches)

> , with a
> maximum of 16834 available file descriptors and 50M of cache (more than
> enough to hold the DB) - and the ratio of cache hits/misses look good
> with little paging out.

That's good, but since performance problems only show up with SSL 
connections, your database and entry caches probably aren't the issue. 
(Another set of items the wiki doesn't cover -- the difference between 
the database cache and the entry cache, and how to size each).