<div dir="ltr">We also looked at udev settle call points in the logs and it seems that it is called a lot of times before.<br><br>The bug I mentioned is <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=616689">https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=616689</a> and they also discuss that maybe udev settle is not working as intended.<br><br>The kernel version of the appliance (as can be seen in the log) is 4.19<br><br>> Collecting the full logs is the right approach to diagnosing this.<br><br>I added the full log for the first failure logs, I think we can see from there that udev settle is called but the file does not exist yet.<br><br>We thought that maybe if we explicitly add the following logic right after g.launch() it might help:<div><br></div><div>1. For each device returned by: lvm 'lvs' '-o' 'vg_name,lv_name' '-S' 'lv_role=public && lv_skip_activation!=yes' '--noheadings' '--separator' '/'</div><div>1.1. stat the device /dev/vg/lv</div><div>1.2. if stat fails on device does not exist - wait</div><div>1.3. Go back to 1</div><div><br></div><div>If we wait for too long, relaunch guestfs.</div><div><br></div><div>However it would be nicer to maybe implement this inside guestfs.launch() itself</div><div><br></div><div>Sam</div></div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 5, 2021 at 3:45 PM Richard W.M. Jones <<a href="mailto:rjones@redhat.com">rjones@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, Apr 05, 2021 at 02:47:51PM +0300, Sam Eiderman wrote:<br> > Hi,<br> > <br> > We are experiencing very rare LVM failures - 2 failures so far, in<br> > different OSs, in different libguestfs functions.<br> > <br> > The first failure is inspect_os() not finding the root operating<br> > system on rhel7.4.<br> > LVM volumes are returned by lvm command but files under /dev do not exist (yet?)<br> > <br> > Second failure is in is_lv() - is_lv() successfully enumerates all lvm<br> > volumes but then internal stat() command fails again on /dev file<br> > since it does not exist (yet?) (rhel8.0)<br> > <br> > All of our tests run in parallel, 1 guestfs instance per core on a 32<br> > core machine and they run on GCP (nested virtualization).<br> > <br> > What we think that is happening here is that libguestfs' appliance is<br> > booting very somewhat slower than usual and that the links to some<br> > devices do not appear yet (even after multiple seconds).<br> > We found this old issue that might be connected to this behavior (in<br> > some way): <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=616689" rel="noreferrer" target="_blank">https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=616689</a><br> <br> I wonder if "udevadm settle" is not working? The daemon will use this<br> command at various times in order to ensure that all preceeding udev<br> messages have been processed and all /dev changes have been made.<br> <br> It is called once at appliance boot:<br> <br> <a href="https://github.com/libguestfs/libguestfs/blob/b18ac489db76a700f2168ae6eb64e9d450613a27/appliance/init#L109" rel="noreferrer" target="_blank">https://github.com/libguestfs/libguestfs/blob/b18ac489db76a700f2168ae6eb64e9d450613a27/appliance/init#L109</a><br> <br> And throughout the daemon code:<br> <br> <a href="https://github.com/libguestfs/libguestfs/blob/b18ac489db76a700f2168ae6eb64e9d450613a27/daemon/utils.c#L732" rel="noreferrer" target="_blank">https://github.com/libguestfs/libguestfs/blob/b18ac489db76a700f2168ae6eb64e9d450613a27/daemon/utils.c#L732</a><br> <br> $ git grep 'udev_settle ()' -- daemon<br> daemon/blockdev.c: udev_settle ();<br> daemon/<a href="http://cryptsetup.ml" rel="noreferrer" target="_blank">cryptsetup.ml</a>: udev_settle ()<br> daemon/<a href="http://cryptsetup.ml" rel="noreferrer" target="_blank">cryptsetup.ml</a>: udev_settle ()<br> daemon/file.c: udev_settle ();<br> daemon/guestfsd.c: udev_settle ();<br> daemon/hotplug.c: udev_settle ();<br> [etc etc]<br> <br> It could be that udev_settle is not being called at the right points,<br> or is not working in the way we understand.<br> <br> ...<br> > Short second failure logs (is_lv() only) - notice that is_lv() is<br> > invoked on /dev/vg_myvg/lv_var but it fails due to a problem in<br> > /dev/rhel/swap not existing)<br> > <br> > 2021-03-07 10:58:53 T libguestfs - 0 - enter - is_lv<br> > 2021-03-07 10:58:53 T libguestfs - 0 - appliance - guestfsd: =><br> > aug_get (0x13) took 0.00 secs<br> > 2021-03-07 10:58:53 T libguestfs - 0 - appliance - guestfsd: <= is_lv<br> > (0x108) request length 64 bytes<br> > 2021-03-07 10:58:53 T libguestfs - 0 - appliance - commandrvf:<br> > stdout=n stderr=y flags=0x0<br> > 2021-03-07 10:58:53 T libguestfs - 0 - appliance - commandrvf: udevadm<br> > --debug settle -E /dev/vg_myvg/lv_var<br> > 2021-03-07 10:58:53 T libguestfs - 0 - appliance - command: lvm 'lvs'<br> > '-o' 'vg_name,lv_name' '-S' 'lv_role=public &&<br> > lv_skip_activation!=yes' '--noheadings' '--separator' '/'<br> > 2021-03-07 10:58:53 T libguestfs - 0 - appliance - command: lvm returned 0<br> > 2021-03-07 10:58:53 T libguestfs - 0 - appliance - command: lvm: stdout:<br> > 2021-03-07 10:58:53 T libguestfs - 0 - appliance - rhel/root<br> > 2021-03-07 10:58:53 T libguestfs - 0 - appliance - rhel/swap<br> > 2021-03-07 10:58:53 T libguestfs - 0 - appliance - vg_myvg/lv_var<br> > 2021-03-07 10:58:53 T libguestfs - 0 - appliance - guestfsd: error:<br> > stat: /dev/rhel/swap: No such file or directory<br> <br> You might want to look earlier in this log to see if udevadm settle<br> was called between the LVs being activated and this API function. If<br> it was not being called then possibly we need to insert a call after<br> activation. If it was being called then perhaps udev settle is not<br> working the way we understand it.<br> <br> Collecting the full logs is the right approach to diagnosing this.<br> <br> The only other issue I can think of is the change in kernel PCI device<br> enumeration code (starting in Linux 5.6,<br> <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1804207" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1804207</a>). I suppose in<br> theory the underlying devices might not be ready at all before we run<br> udev settle in the appliance. However I have not seen this actually<br> happen.<br> <br> Rich.<br> <br> -- <br> Richard Jones, Virtualization Group, Red Hat <a href="http://people.redhat.com/~rjones" rel="noreferrer" target="_blank">http://people.redhat.com/~rjones</a><br> Read my programming and virtualization blog: <a href="http://rwmj.wordpress.com" rel="noreferrer" target="_blank">http://rwmj.wordpress.com</a><br> libguestfs lets you edit virtual machines. Supports shell scripting,<br> bindings from many languages. <a href="http://libguestfs.org" rel="noreferrer" target="_blank">http://libguestfs.org</a><br> <br> </blockquote></div>