The issue about adding multipath device's targets into qemu-pr-helper's namespace

[Lin Ma]

Hi all,

I have a namespace question about passthrough disk(multipath device).
In case of enabling namespace and cgroups in qemu.conf, The target(s) of 
the
multipath device won't be added into qemu-pr-helper's namespace under 
certain
situation, It causes the PERSISTENT RESERVE command failure in guest.

While user starts a vm,
To build namespace, The qemuDomainSetupDisk() will be invoked via 
threadA(this
thread id will be the qemu's pid),
To build cgroup, The qemuSetupImagePathCgroup() will be invoked via 
threadB.

Both of the functions invoke the virDevMapperGetTargets() trying to 
parse a
multipath device to target paths string, Then fill the targetPaths[].

The issue I experienced is:
After libvirtd started, Everything works well for the first booted vm 
which has
the passthrough multipath device.
But If I shut it down & start it again, OR keep it running & start 
another vm
which has other passthrough multipath device, Then the target(s) of the 
fresh
started vm won't be added into the related qemu-pr-helper's namespace 
and it
causes PERSISTENT RESERVE command failure in the corresponding guest.
I digged into code, In this situation, The targetPaths[] in 
qemuDomainSetupDisk()
won't be filled, it keeps NULL after virDevMapperGetTargets() returns.
The virDevMapperGetTargets doesn't fill targetPaths[] because the 
dm_task_run()
of libdevmapper returns 0 with errno 9(Bad file descriptor).
So far, I don't understand why the dm_task_run() return 0 in this 
situation.
BTW, The virDevMapperGetTargets() can always successfully fill the 
targetPaths[]
in qemuSetupImagePathCgroup().

Please refer to the following 2 tests:
The multipath configuration on host:
host:~ # multipath -l
vm1-data (3600140582d9024bc13f4b8db5ff12de0) dm-11 FreeNAS,lv68
size=6.0G features='0' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=0 status=active
   `- 2:0:0:2 sdd 8:48 active undef running
vm2-data (36001405fc5f29ace3ec4fb8acd32aae5) dm-8 FreeNAS,lv46
size=4.0G features='0' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=0 status=active
   `- 2:0:0:1 sde 8:64 active undef running

===================================================================
Test A:
host:~ # systemctl restart libvirtd
host:~ # virsh list
  Id   Name   State
--------------------

host:~ #
host:~ # virsh domblklist vm1
  Target   Source
------------------------------------------
  vda      /opt/vms/vm1/disk0.qcow2
  sda      /dev/mapper/vm1-data

host:~ #
host:~ # virsh start vm1
Domain vm1 started

host:~ # virsh list
  Id   Name        State
---------------------------
  1    vm1        running

host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
host:~ # ls -l /dev/sd*
brw-rw---- 1 root disk 8, 48 Jul 14 16:30 /dev/sdd
host:~ # exit
exit
host:~ #

vm1:~ # lsscsi
[0:0:0:0]    disk    FreeNAS  lv68             0123   /dev/sda
vm1:~ #
vm1:~ # sg_persist --in -k /dev/sda
   FreeNAS   lv68              0123
   Peripheral device type: disk
   PR generation=0x0, there are NO registered reservation keys
vm1:~ #

host:~ # virsh shutdown vm1
Domain vm1 is being shutdown

host:~ # virsh list
  Id   Name   State
--------------------

host:~ #
host:~ # virsh start vm1
Domain vm1 started

host:~ # virsh list
  Id   Name        State
---------------------------
  2    vm1        running

host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
host:~ # ls -l /dev/sd*
ls: cannot access '/dev/sd*': No such file or directory
host:~ # exit
exit
host:~ #

vm1:~ # sg_persist --in -k /dev/sda
   FreeNAS   lv68              0123
   Peripheral device type: disk
PR in (Read keys): Aborted command
Aborted command
vm1:~ #
===================================================================
Test B:
host:~ # systemctl restart libvirtd
host:~ # virsh list
  Id   Name   State
--------------------

host:~ #
host:~ # virsh domblklist vm1
  Target   Source
------------------------------------------
  vda      /opt/vms/vm1/disk0.qcow2
  sda      /dev/mapper/vm1-data

host:~ #
host:~ # virsh start vm1
Domain vm1 started

host:~ # virsh list
  Id   Name        State
---------------------------
  1    vm1        running

host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
host:~ # ls -l /dev/sd*
brw-rw---- 1 root disk 8, 48 Jul 14 17:28 /dev/sdd
host:~ # exit
exit
host:~ #

vm1:~ # lsscsi
[2:0:0:0]    disk    FreeNAS  lv68             0123   /dev/sda
vm1:~ #
vm1:~ # sg_persist --in -k /dev/sda
   FreeNAS   lv68              0123
   Peripheral device type: disk
   PR generation=0x0, there are NO registered reservation keys
vm1:~ #

host:~ # virsh list
  Id   Name        State
---------------------------
  1    vm1        running

host:~ #
host:~ # virsh domblklist vm2
  Target   Source
------------------------------------------
  vda      /opt/vms/vm2/disk0.qcow2
  sda      /dev/mapper/vm2-data

host:~ #
host:~ # virsh start vm2
Domain vm2 started

host:~ # virsh list
  Id   Name        State
---------------------------
  1    vm1        running
  2    vm2        running

host:~ # nsenter -t $(qemu-pr-helper pid of vm2) -a bash
host:~ # ls -l /dev/sd*
ls: cannot access '/dev/sd*': No such file or directory
host:~ # exit
exit
host:~ #

vm2:~ # lsscsi
[0:0:0:0]    disk    FreeNAS  lv46             0123   /dev/sda
vm2:~ #
vm2:~ # sg_persist --in -k /dev/sda
   FreeNAS   lv46              0123
   Peripheral device type: disk
PR in (Read keys): Aborted command
Aborted command
vm2:~ #
===================================================================

Any comments will be much appreciated.

Thanks in advance,
Lin