[libvirt-users] high memory guest issues - virsh start and QEMU_JOB_WAIT_TIME

[Michal Privoznik]

On 02/14/2017 08:13 AM, Blair Bethwaite wrote:
> Hi all,
> 
> In IRC last night Dan helpfully confirmed my analysis of an issue we are
> seeing attempting to launch high memory KVM guests backed by hugepages...
> 
> In this case the guests have 240GB of memory allocated from two host NUMA
> nodes to two guest NUMA nodes. The trouble is that allocating the hugepage
> backed qemu process seems to take longer than the 30s QEMU_JOB_WAIT_TIME
> and so libvirt then most unhelpfully kills the barely spawned guest. Dan
> said there was currently no workaround available so I'm now looking at
> building a custom libvirt which sets QEMU_JOB_WAIT_TIME=60s.

I don't think I understand this. Who is running the other job? I mean,
I'd expect qemu fail to create the socket and thus hitting 30s timeout
in qemuMonitorOpenUnix().

> 
> I have two related questions:
> 1) will this change have any untoward side-effects?

Since this timeout is shared with other jobs, you might have to wait a
bit longer for an API to return with error if a domain is stuck and
unresponsive.

> 2) if not, then is there any reason not to change it in master until a
> better solution comes along (or possibly better, alter
> qemuDomainObjBeginJobInternal
> to give a domain start job a little longer compared to other jobs)?

It's a trade off between "responsiveness" of a libvirt API and being
able to talk to qemu which is under heavy load. From libvirt's POV we
are unable to tell whether qemu is doing something or is stuck (e.g.
looping endlessly). So far, we felt like 30 seconds is a good choice.
But I don't mind being proven wrong.

Michal