[libvirt] Ongoing work on lock contention in qemu driver?

[Peter Feiner]

On Wed, May 22, 2013 at 7:31 PM, Peter Feiner <peter at gridcentric.ca> wrote:
> Since some security driver operations are costly, I think it's
> worthwhile to reduce the scope of the security manager lock or
> increase the granularity by introducing more locks. After a cursory
> look, the security manager lock seems to have a much broader scope
> than necessary. The system / library calls underlying the security
> drivers are all thread safe (e.g., defining apparmor security profiles
> or chowning disk files), so a global lock isn't strictly necessary.
> Moreover, since most virSecurity calls are made whilst a virDomainObj
> lock is held and the security calls are generally domain specific,
> *most* of the security calls are probably thread safe in the absence
> of the global security manager lock. Obviously some work will have to
> be done to see where the security lock actually matters and some
> finer-grained locks will have to be introduced to handle these
> situations.

To verify that this is worthwhile, I disabled the apparmor driver
entirely. My 20 VM creation test ran about 10s faster (down from 35s
to 25s).

After giving this approach a little more thought, I think an
incremental series of patches is a good way to go. The responsibility
of locking could be pushed down into the security drivers. At first,
all of the drivers would lock where their managers' locked. Then each
driver could be updated to do more fine-grained locking. I'm going to
work on a patch to push the locking down into the drivers, then I'm
going to work on a patch for better locking in the apparmor driver.

> I also think it's worthwhile to eliminate locking from the the
> virDomainObjList lookups and traversals. Since virDomainObjLists are
> accessed in a bunch of places, I think it's a good defensive idea to
> decouple the performance of these accesses from virDomainObj locks,
> which are held during potentially long-running operations like domain
> creation. An easy way to divorce virDomainObjListSearchName from the
> virDomainObj lock would be to keep a copy of the domain names in the
> virDomainObjList and protect that list with the virDomainObjList lock.

After removing the security driver contention, this was still a
substantial bottleneck: virConnectDefineXML could still take a few
seconds. I removed the contention by keeping a copy of the domain
definition's name in the domain object. Since the name is immutable
and the domain object is protected by the list lock, the list
traversal can read the name without taking any additional locks. This
patch reduced virConnectDefineXML to tens of milliseconds.