[PATCH] libvirt-guests.sh: fix log output with new systemd

[Christian Ehrhardt]

On Thu, Apr 30, 2020 at 7:23 PM Daniel P. Berrangé <berrange at redhat.com> wrote:
>
> On Thu, Apr 30, 2020 at 06:28:08PM +0200, Christian Ehrhardt wrote:
> > On Thu, Apr 30, 2020 at 5:10 PM Daniel P. Berrangé <berrange at redhat.com> wrote:
> > >
> > > On Thu, Apr 30, 2020 at 04:58:25PM +0200, Christian Ehrhardt wrote:
> > > > On Thu, Apr 30, 2020 at 2:51 PM Daniel P. Berrangé <berrange at redhat.com> wrote:
> > > > Well it seems I have a reproducible symptom and a fix, but not the
> > > > explanation why the latter fixes the former.
> > > > I'll need to dive into some debug & analysis myself to explain it better.
> > > > I'll be back here once I got time to do that in depth check.
> > > >
> > > > Until then whoever is affected (should be everyone) can give it a
> > > > thought as well.
> > > > Repro is as easy as
> > > > One console:
> > > > $ journalctl -f -u libvirt-guests
> > > > Other console:
> > > > $ systemctl stop libvirt-guests
> > > > $ systemctl start libvirt-guests
> > > >
> > > > I see it with 245.4-4ubuntu3 (18.04) I'm not seeing it on
> > > > 237-3ubuntu10.39 (20.04).
> > > > Maybe it is a systemd bug after all?
> > > > I'd be interested to hear if that is/isn't clobbering output for
> > > > anyone else and what your systemd versions are?
> > >
> > > FWIW, it works correctly on Fedora 31 with systemd 243.
> >
> > Eoan with 242-7ubuntu3.7 is good as well.
> > I might need to try to get some interim versions from somewhere.
>
> I've reproduced on Fedora 33 rawhide with systemd 245 - the first
> place where it lists running guests is screwed up slightly:
>
> Apr 30 17:16:13 libvirt-fedora-rawhide systemd[1]: Stopping Suspend/Resume Running libvirt Guests...
> Apr 30 17:16:13 libvirt-fedora-rawhide libvirt-guests.sh[69903]: Running guests on default URI:
> Apr 30 17:16:13 libvirt-fedora-rawhide libvirt-guests.sh[69892]: Runningcore2
> Apr 30 17:16:13 libvirt-fedora-rawhide libvirt-guests.sh[69934]: Suspending guests on default URI...
> Apr 30 17:16:13 libvirt-fedora-rawhide libvirt-guests.sh[69892]: SSuspending core1: ...
> Apr 30 17:16:14 libvirt-fedora-rawhide libvirt-guests.sh[69892]: Suspending core1: done
> Apr 30 17:16:14 libvirt-fedora-rawhide libvirt-guests.sh[69892]: Suspending core2: ...
> Apr 30 17:16:15 libvirt-fedora-rawhide libvirt-guests.sh[69892]: Suspending core2: done
> Apr 30 17:16:15 libvirt-fedora-rawhide systemd[1]: libvirt-guests.service: Succeeded.
> Apr 30 17:16:15 libvirt-fedora-rawhide systemd[1]: Stopped Suspend/Resume Running libvirt Guests.
>
> On resume it is even worse
>
> Apr 30 17:19:40 libvirt-fedora-rawhide systemd[1]: Starting Suspend/Resume Running libvirt Guests...
> Apr 30 17:19:40 libvirt-fedora-rawhide libvirt-guests.sh[70041]: Resuming guests on default URI...
> Apr 30 17:19:40 libvirt-fedora-rawhide libvirt-guests.sh[70030]: R
> Apr 30 17:19:40 libvirt-fedora-rawhide libvirt-guests.sh[70048]: R
> Apr 30 17:19:41 libvirt-fedora-rawhide libvirt-guests.sh[70048]: esuming guest core1:
> Apr 30 17:19:41 libvirt-fedora-rawhide libvirt-guests.sh[70079]: esum
> Apr 30 17:19:41 libvirt-fedora-rawhide libvirt-guests.sh[70030]: e
> Apr 30 17:19:41 libvirt-fedora-rawhide libvirt-guests.sh[70086]: e
> Apr 30 17:19:42 libvirt-fedora-rawhide libvirt-guests.sh[70086]: esuming guest core2:
> Apr 30 17:19:42 libvirt-fedora-rawhide libvirt-guests.sh[70119]: esum
> Apr 30 17:19:42 libvirt-fedora-rawhide libvirt-guests.sh[70030]: e
> Apr 30 17:19:42 libvirt-fedora-rawhide systemd[1]: Finished Suspend/Resume Running libvirt Guests.
>
>
> It is possible it isn't systemd related - could be other packages that
> are co-incidentally affecting it.

Since launchpad holds all former builds still ready for download I
could easily test a few versions.

I up/downgraded just the following packages on an otherwise unmodified system:
- libnss-systemd
- libpam-systemd
- libsystemd0
- systemd
- systemd-container
- systemd-sysv

Those versions I did check:

245.4-4ubuntu3 bad
245.2-1ubuntu1 bad
244.3-1ubuntu1 bad
244.1-0ubuntu3 bad (bad on retry)
243-3ubuntu1 good (good (on retry)
242-7ubuntu3 good

Retry means that I went and installed from good -> bad -> good and the
behavior was the same.
So none of the installs fixed it to then be good forever. It was
consistently good <244 and bad >=244.

Since I only installed the mentioned systemd packages and no others
I'd say it is systemd.
I can't break this up into which package, since they inter-depend each other.
I guess it might be time to file a systemd bug for this - if not to
fix then to understand what is going on so that we can make a better
decision.

>
> Regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>


-- 
Christian Ehrhardt
Staff Engineer, Ubuntu Server
Canonical Ltd