[libvirt] Mystery of qemu being unable to open NBD Unix domain socket in current directory

Wed Jun 6 17:29:21 UTC 2018

I'm chasing down a very frustrating bug which only happens on i686 &
Koji during the nbdkit tests and seemingly nowhere else.  Anyway this
is what I've been able to put together:

The libguestfs appliance (guest) is created with this XML snippet:

  <disk device="disk" type="network">
    <source protocol="nbd">
      <host transport="unix" socket="cow.sock"/>
    </source>
    <target dev="sda" bus="scsi"/>
    <driver name="qemu" type="raw" cache="writeback"/>
    <address type="drive" controller="0" bus="0" target="0" unit="0"/>
  </disk>

It fails to start with the error:

  internal error: process exited while connecting to monitor: 2018-06-06T17:02:54.450507Z qemu-system-i386: -drive file=nbd+unix://?socket=cow.sock,format=raw,if=none,id=drive-scsi0-0-0-0,cache=writeback: Failed to connect socket cow.sock: No such file or directory [code=1 int1=-1]

The socket definitely exists in the directory of the program running
libvirt.  I verified that by adding ls -l commands to the build:

  srwxr-xr-x.  1 mockbuild mockbuild          0 Jun  6 17:02 cow.sock

Permissions on the containing directories are fine (the whole lot is
running as mockbuild):

  + pwd
  /builddir/build/BUILD/nbdkit-1.3.1/tests
  + ls -ld /builddir
  drwx------. 5 mockbuild mockbuild 4096 Jun  6 17:00 /builddir
  + ls -ld /builddir/build
  drwxr-xr-x. 9 mockbuild mockbuild 4096 Jun  6 16:57 /builddir/build
  + ls -ld /builddir/build/BUILD
  drwxrwxr-x. 3 mockbuild 1000 4096 Jun  6 16:59 /builddir/build/BUILD
  + ls -ld /builddir/build/BUILD/nbdkit-1.3.1
  drwxr-xr-x. 11 mockbuild mockbuild 4096 Jun  6 17:00 /builddir/build/BUILD/nbdkit-1.3.1
  + ls -ld /builddir/build/BUILD/nbdkit-1.3.1/tests
  drwxr-xr-x. 4 mockbuild mockbuild 4096 Jun  6 17:02 /builddir/build/BUILD/nbdkit-1.3.1/tests

The qemu log doesn't give any clues:

  2018-06-06 17:02:54.412+0000: starting up libvirt version: 4.4.0, package: 1.fc29 (Fedora Project, 2018-06-05-11:53:38, buildvm-12.phx2.fedoraproject.org), qemu version: 2.12.0qemu-2.12.0-2.fc29, kernel: 4.16.8-300.fc28.x86_64, hostname: buildhw-10.phx2.fedoraproject.org
  LC_ALL=C PATH=/builddir/build/BUILD/nbdkit-1.3.1:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin HOME=/builddir USER=mockbuild LOGNAME=mockbuild QEMU_AUDIO_DRV=none TMPDIR=/var/tmp /usr/bin/qemu-system-i386 -name guest=guestfs-62ak9dzrjxfoctua,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/builddir/.config/libvirt/qemu/lib/domain-3-guestfs-62ak9dzrjxfo/master-key.aes -machine pc-i440fx-2.12,accel=tcg,usb=off,dump-guest-core=off -m 500 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 6094da49-2def-4256-b901-e4fe5f1409c8 -display none -no-user-config -nodefaults -device sga -chardev socket,id=charmonitor,path=/builddir/.config/libvirt/qemu/lib/domain-3-guestfs-62ak9dzrjxfo/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -no-acpi -boot strict=on -kernel /var/tmp/.guestfs-1000/appliance.d/kernel -initrd /var/tmp/.guestfs-1000/appliance.d/initrd -append 'panic=1 noapic console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=vt100' -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive 'file=nbd+unix://?socket=cow.sock,format=raw,if=none,id=drive-scsi0-0-0-0,cache=writeback' -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1,write-cache=on -drive file=/tmp/libguestfsprC4lc/overlay1.qcow2,format=qcow2,if=none,id=drive-scsi0-0-1-0,cache=unsafe -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0,write-cache=on -chardev socket,id=charserial0,path=/tmp/libguestfsj35K6W/console.sock -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/tmp/libguestfsj35K6W/guestfsd.sock -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.libguestfs.channel.0 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x4 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
  2018-06-06 17:02:54.412+0000: Domain id=3 is tainted: custom-argv
  2018-06-06T17:02:54.450507Z qemu-system-i386: -drive file=nbd+unix://?socket=cow.sock,format=raw,if=none,id=drive-scsi0-0-0-0,cache=writeback: Failed to connect socket cow.sock: No such file or directory
  2018-06-06 17:02:54.453+0000: shutting down, reason=failed

At this point I don't really have any ideas.  Does libvirt now run
qemu in a different directory?  Does the error message mean something
else apart from the file not existing?

Also the nbd+unix syntax doesn't appear to be documented in qemu.
Is this a new thing?  Normally we use nbd:unix:...

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v