[libvirt] [PATCH] Increase default file handle limits for daemons

[Daniel P. Berrange]

On Wed, Mar 15, 2017 at 07:34:04PM +0100, Guido Günther wrote:
> Hi,
> On Wed, Mar 15, 2017 at 04:55:04PM +0000, Daniel P. Berrange wrote:
> > Linux still defaults to a 1024 open file handle limit. This causes
> > scalability problems for libvirtd / virtlockd / virtlogd on large
> > hosts which might want > 1024 guest to be running. In fact if each
> > guest needs > 1 FD, we can't even get to 500 guests. This is not
> > good enough when we see machines with 100's of physical cores and
> > TBs of RAM.
> > 
> > In comparison to other memory requirements of libvirtd & related
> > daemons, the resource usage associated with open file handles
> > is essentially line noise. It is thus reasonable to increase the
> > limits unconditionally for all installs.
> > 
> > Signed-off-by: Daniel P. Berrange <berrange at redhat.com>
> > ---
> >  daemon/libvirtd.service.in       | 7 +++++--
> >  src/locking/virtlockd.service.in | 4 ++++
> >  src/logging/virtlogd.service.in  | 5 +++++
> >  3 files changed, 14 insertions(+), 2 deletions(-)
> > 
> > diff --git a/daemon/libvirtd.service.in b/daemon/libvirtd.service.in
> > index c72dde5..22fc156 100644
> > --- a/daemon/libvirtd.service.in
> > +++ b/daemon/libvirtd.service.in
> > @@ -24,8 +24,11 @@ ExecStart=@sbindir@/libvirtd $LIBVIRTD_ARGS
> >  ExecReload=/bin/kill -HUP $MAINPID
> >  KillMode=process
> >  Restart=on-failure
> > -# Override the maximum number of opened files
> > -#LimitNOFILE=2048
> > +# At least 1 FD per guest, often 2 (eg qemu monitor + qemu agent).
> > +# If we want to support 2048 guests, we'll typically need 4096 FDs
> 
> 4096 FDs here…

Sigh, double both these numbers

> > +# If changing this, also consider virtlogd.service & virtlockd.service
> > +# limits which are also related to number of guests
> > +LimitNOFILE=8192
> 
> …but 8192 here. So we're looking at 4096 rather than 2048 guests (2 fds per guest)?
> 
> >  
> >  [Install]
> >  WantedBy=multi-user.target
> > diff --git a/src/locking/virtlockd.service.in b/src/locking/virtlockd.service.in
> > index 69b568f..c369591 100644
> > --- a/src/locking/virtlockd.service.in
> > +++ b/src/locking/virtlockd.service.in
> > @@ -13,6 +13,10 @@ ExecReload=/bin/kill -USR1 $MAINPID
> >  # cause the machine to be fenced (rebooted), so make
> >  # sure we discourage OOM killer
> >  OOMScoreAdjust=-900
> > +# Needs to allow for max guests * average disks per guest
> > +# libvirtd.service written to expect 4096 guests, so if we
> > +# allow for 4 disks per guest, we get:
> > +LimitNOFILE=16384
> 
> Correct if we're looking at 4095 guests above.
> 
> >  
> >  [Install]
> >  Also=virtlockd.socket
> > diff --git a/src/logging/virtlogd.service.in b/src/logging/virtlogd.service.in
> > index 09e0740..be039c6 100644
> > --- a/src/logging/virtlogd.service.in
> > +++ b/src/logging/virtlogd.service.in
> > @@ -13,6 +13,11 @@ ExecReload=/bin/kill -USR1 $MAINPID
> >  # cause the machine to be fenced (rebooted), so make
> >  # sure we discourage OOM killer
> >  OOMScoreAdjust=-900
> > +# Need to have at least one file open per guest (eg QEMU
> > +# stdio log), but might be more (eg serial console logs)
> > +# libvirtd.service written to expect 2048 guests, so if we
> 
> Rather 4096 as above?

Yes.

> 
> > +# guess at 2 log files per guest here (stdio + 1 serial):
> > +LimitNOFILE=8192
> >  
> >  [Install]
> >  Also=virtlogd.socket

I'll push with the comments fixed

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|