[libvirt] Limit autostarting of domains to a few at a time.

[Ruben Kerkhof]

On Mon, Nov 5, 2012 at 2:39 PM, Peter Krempa <pkrempa at redhat.com> wrote:
> On 11/05/12 13:32, Ruben Kerkhof wrote:
>>
>> Hi list,
>
>
> Hi Ruben,

Hi Peter,

Thanks for your reply.
>
>>
>> I have about a hundred kvm vm's on a physical host, all marked as
>> autostart.
>> As soon as libvirtd starts up, all vms get started and they start
>> fighting for disk I/O.
>
>
>> Would it be possible to add an option to wait a while before starting
>> the next vm?
>> Batch-starting them in parallel, let's say 10 at a time, would be great
>> too.
>
>
> The problem with this is how to detect if the machine is actually started
> up. For libvirt the guests are black boxes and we don't have insight if the
> guest has already started up. There are a few options how we could detect
> this, but none of them are ideal:
> 1) wait until I/O load calms down - this is impossible with I/O heavy guests

Correct. We use the cpu and blkio cgroups which help somewhat in this case.

>
> 2) use guest agent - no way to know if it has actually started/is installed
>
> 3) wait a certain amount of time - some guests take longer to start

I would be happy with that last option as a configuration setting.
It's far from perfect, but with a bit of experimentation I should be
able to come up with a reasonable wait time.

>
>
>>
>> I know I could use the libvirt-guest init script to do this, but we
>> rely on the autostart property for several purposes.
>
>
> The libvirt-guests script isn't 100% ideal for this purpose. It has the
> ability to re-start guests that were shut down by it, but can't be
> configured to start arbitrary guests.
>
> Unfortunately autostarting of guest can't be disabled in the daemon. If it
> was possible you could "virsh list --uuid --inactive --autostart" guests
> that need to be autostarted and use the function from libvirt-guests that is
> used to start the machines to do so (instead of reading them from the
> save-file).

That would be a reasonable solution.

> This would still use the non-ideal approach (3) to rate-limit the starting.
> With the knowledge of the guest, this could be improved.
>
> As of the ability to disable autostarting, I'll have a look if it's hard to
> implement. I think this would make sense to have it as a config option. On
> the other hand, there's no ideal way to implement the detection of the
> end-of-boot event on the guest, so it'll probably be better to leave piece
> of functionality to be implemented by the users who know their
> infrastructure.
>
> Until then, you'll have either to start them via libvirt-autostart without
> the ability to rate-limit starting or note the autostart state into some
> other element (eg. the <metadata> element) and have a custom script that
> starts the guests.

I didn't know that one, but that looks like a nice workaround for now, thanks.
>
>>
>> Kind regards,
>>
>> Ruben Kerkhof
>>
>
> Peter

Thanks again,

Ruben