[linux-lvm] new multipath device mistakenly replaced another PV in existing volume group

[Neutron Sharc]

I'm seeing a strange problem (iscsi LUNs + multipath + lvm) that I
will walk through an example.

I have an iscsi target machine that exposes many iscsi LUNs. Iscsi
initiator logs in 4 iscsi LUNs (vol1_[0-4]),  creates a multipath
device for each LUN (/dev/mapper/vol1_[0-4]), and combines the 4
multipath devices into a volume group and LV (vol1/vol1_lv).

Then I log in another 4 iscsi LUNs (vol3_[0-3]), create a multipath
device for each new LUN (/dev/mapper/vol3_[0-3]).  Now there is a
strange thing:
some new multipath devices (/dev/mapper/vol3_0, /dev/mapper/vol3_2)
replaced existing PVs in vol1.  As a result, these new multipath
devices have open-count > 0, so I cannot pvcreate on them:

$ sudo dmsetup ls --tree
vol1-vol1_lv (252:4)
 ├─vol3_0 (252:9)   <==  fresh multipath device, should NOT be in vol1
 │  └─ (65:128)
 ├─vol3_2 (252:10)  <==  fresh multipath device, should NOT be in vol1
 │  └─ (65:144)
 ├─vol1_1 (252:1)
 │  └─ (65:48)
 └─vol1_0 (252:0)
    └─ (65:32)
vol1_3 (252:3)  <== was in vol1-vol1_lv, but was knocked out
 └─ (65:16)
vol1_2 (252:2)  <== was in vol1-vol1_lv, but was knocked out
 └─ (65:64)
vol3_3 (252:11)
 └─ (65:160)
vol3_1 (252:12)
 └─ (65:176)


Please note that all these vol3_[0-3] are fresh, without any LVM
metadata on them, as shown by pvscan::
sudo pvscan --cache /dev/mapper/vol3_0
  Incorrect metadata area header checksum on /dev/mapper/vol3_0 at offset 4096
  Incorrect metadata area header checksum on /dev/mapper/vol3_0 at offset 4096
  Incorrect metadata area header checksum on /dev/mapper/vol3_0 at offset 4096


$ sudo pvcreate /dev/mapper/vol3_0   <== this multipath device was
mistakenly included into vol1

  Found duplicate PV 9F6vU9NVBfEq1w3e04T5UreO6fDVPJNy: using
/dev/mapper/vol1_3 not /dev/mapper/vol3_0
  Using duplicate PV /dev/mapper/vol1_3 without holders, replacing
/dev/mapper/vol3_0
  Can't open /dev/mapper/vol3_0 exclusively.  Mounted filesystem?


========== Configs I used:
BTW,  I have enabled lvmetad,  my lvm.conf has this:
filter = [ "a|/dev/mapper/.*|", "r|.*|" ]
global_filter = [ "a|/dev/mapper/.*|", "r|.*|" ]

My /etc/multipath.conf is:
defaults {
  user_friendly_names     yes
  path_grouping_policy    failover
  polling_interval        10
  path_selector           "round-robin 0"
  find_multipaths         yes
  features       "1 queue_if_no_path"
}
blacklist {
  devnode  "^sda[1-9]"
}
multipaths {
  multipath {
    wwid 360000000758757de9fb289cbde12abab
    alias vol1_0
  }
  // more devices
}

iscsi initiator is on centos 6.5, with pkgs version:
lvm2-2.02.143-12.el6.x86_64
device-mapper-multipath-0.4.9-100.el6.x86_64

iscsi target is tgtd on another Ubuntu machine.



Comments are appreciated.


-Shawn