[libvirt-users] for a guest accessing host "full disk", how to prevent host vgscan

[David Mansfield]

On 12/22/2011 02:00 PM, David Mansfield wrote:
>
>
> On 12/22/2011 12:44 PM, Daniel P. Berrange wrote:
>> On Thu, Dec 22, 2011 at 09:20:01AM -0500, David Mansfield wrote:
>>>
>>> On 12/21/2011 05:41 PM, Daniel P. Berrange wrote:
>>>> On Wed, Dec 21, 2011 at 05:23:33PM -0500, David Mansfield wrote:
>>>>> Hi All.
>>>>>
>>>>> I have a dell system with a H700 raid. Within the hardware RAID
>>>>> config I've created a "virtual disk" which I have assigned to one of
>>>>> my guests. On the host the device is "/dev/sdb", on the guest it's
>>>>> "/dev/vdb".
>>>>>
>>>>> This works fine.
>>>>>
>>>>> Within the guest, we have created lvm PV on /dev/vdb (using the
>>>>> whole disk - no partitions) and created a volume group. The guest's
>>>>> hostname is "argo" and the vg is called "vg_argo_bkup".
>>>>>
>>>>> When I reboot the host, it does a vgscan and finds the volume group
>>>>> and activates it in the _host_, which I need to prevent (I think??).
>>>>>
>>>>> I have successfully done this by filtering "/dev/sdb" in
>>>>> /etc/lvm/lvm.conf (which does NOT work as advertised BTW), but
>>>>> referencing the extremely volatile SCSI "sd*" names seems a terrible
>>>>> way to do this. If I fiddle around in the HW raid config, the
>>>>> /dev/sd? may change.
>>>>>
>>>>> I plan on creating about 10 more VM's spread over a number of
>>>>> machines over the next weeks with a very similar setup, and the
>>>>> admin overhead seems like it'll be onerous and error-prone.
>>>>>
>>>>> I'd love to be able to filter the volume groups by VG name instead
>>>>> of pv device node. The host's hostname is "narnia" and I'd love to
>>>>> say, 'vgscan --include-regex "vg_narnia.*"' or something similar, if
>>>>> you get my drift.
>>>>>
>>>>> Does anyone have a best practice for this? I'm sure iSCSI
>>>>> enthusiasts must have the exact same issue all the time.
>>>> The recommended approach is not to assign the entire disk to the
>>>> guest. Partition the host disk, to contain 1 single partition
>>>> consuming all space, then assign the partition to the guest. Worst
>>>> case is you loose a few KB of space due to partition alignment, but
>>>> this is a small price to pay to avoid the LVM problems you describe
>>>> all to well.
>>> I don't really understand. The host still scans the partitions,
>>> right? And the partition "dev" names change dynamically if the
>>> whole-disk changes it's "dev" name. Won't I still have to list
>>> specific volatile names in the /etc/lvm/lvm.conf on the host?
>> The host will see '/dev/sda' and '/dev/sda1', you'll assign
>> ' /dev/sda1' to the guest, and it will appear as /dev/vda.
>> In the guest you'll create '/dev/vda1' and format it as the
>> PV. So while the host will see /dev/sda1, it won't see the
>> nested partition table, and thus won't see the PV
>
> Ahhh. Brilliant. Thanks. Now the only problem is getting the underlying
> stripe alignment right ;-)
>

In case anyone googles this (or is on this list and curious ;-) and gets 
to the last part, about aligning the partitions, I thought I'd mention 
the completely bizarre behavior I'm seeing.

First of all, with a 3TB "disk", you have to use the GPT label, 
available in 'parted' and not in 'fdisk'.  Anyway, my particular "disk" 
is a RAID5 (on my H700 raid controller) of 4 disks, with a 64k stripe 
element.  Therefore each full-stripe is 192k.  This is 384 512-byte 
sectors.  So I used this line in parted:

mkpart primary 384s  5857345502s

And the performance on /dev/sda1 sucked compared to /dev/sda.  I tried 
many different alignments, then finally tried:

mkpart primary 383s  5857345502s

And voila!

I can't explain this, because I can verify the partition start sector is 
indeed 383 according to /sys/block/sda1/start, so it's not a 'parted' 
bug or anything.

The difference in the host was significant, but I went further and 
assigned the partition, with different alignments, to the VM and tested 
there and the difference was magnified.  When aligned, the VM hit very 
close to bare metal speed, but when unaligned it suffered a perf hit 
(about 50% of when aligned) vs bare metal (which was slower anyway to 
the misalignment).

So thanks to Daniel who gave me the trick, and a reverse-thanks to the 
gods of the computers for coming up with this bizarre unexpected result.

David Mansfield
Cobite, INC.