[vfio-users] VFIO and KSM (and maybe hugepages)

[Colin Godsey]

I was able to reproduce the issue again by enabling KSM and THP, with
aggressive settings for KSM. This time it appeared on one VM when the other
hibernated itself (has also happened with them both running previously).
The issue manifested itself first as two driver crashes (very uncommon for
me, but ignored for this test). Final issue was the triangle-tearing I was
seeing before. Was about the 1hr mark which is rather fast. There were ~300
shared pages at the time, so assuming it happened upon unsharing the pages.

I’ve confirmed that THP + no KSM works fine, as it should in v4.4. The last
test I need to do is KSM with no THP. Considering these are both remapping
modules using slightly different methods, I wouldn’t be surprised if there
was a possibility of some sort of ref-counting issue when used together
(combined with the long-held get_user_page for VFIO). KSM and THPs ref
counting in 4.4 is really hard to follow, and they were both entirely
redone in 4.5.

Looking over the KSM code, THP, VFIO and various other modules… it does
appear that get_user_pages for long-held pins is rather brittle, and the
THP and KSM code doesnt appear to strengthen any of its guarantees.

KSM does appear to have some really messy count-checking logic around its
write_protection function, including special post-race checks looking for
O_DIRECT.

I… think I found all the /proc calls I need to catch any mapping that might
be occurring (smaps, maps). Is there a better way, or any more details proc
calls? Also, all IOMMU DMA regions should show in the map under vfio
correct (for the DMAR)?

System background-

ubuntu ‘lowlatency’  kernel 4.4.0-22-lowlatency (voluntary kernel preempt,
irq threads, etc). Confirmed artifacts happen on the generic kernel also.
skylake with updated microcode (hopefully fixing the p-state and hang
issues)
Single dye, single NUMA region.

On Tue, May 31, 2016 at 4:54 PM Colin Godsey <crgodsey at gmail.com> wrote:

> Hmm It would have been a v4.4 build. Could very well just be coincidence I
> haven’t seen it again.
>
> Either way, I appreciate the info! I’ve been wary of trying THP/KSM for a
> while because of that, but this renews my faith. The clarification about
> THP is also a relief. There’s still so many articles/posts that list these
> ‘gotchas’ regarding VFIO (that have been mostly fixed), it is pretty easy
> to go on unneeded witch hunts =\
>
> Regarding KSM though… as far as i can tell (in the 4.4 kernel) KSM doesn’t
> use the normal GUP style references (uses get_user_pages_fast
> internally): https://github.com/torvalds/linux/blob/v4.4/mm/ksm.c#L887
>
> I’m not horribly familiar with KSM itself, but from what I gathered of its
> history, this may be a mechanism it uses to allow shared pages to still go
> to swap etc.
>
> Also afaik KSM, THP and kswapd (@ v4.4) all manipulate the mm differently,
> with KSM and THP having their own unique problems dealing with each other.
> This could also have been some kind of perfect storm, using basically all
> of the memory-mapping technologies modern linux has, at the same time…
>
> I’ll need to step through the entire pre and post 4.5 TLB changes and see
> if I anything looks familiar to the current KSM mapping/ref-tracking.
>
> Unfortunately I don’t have any other info at the time- I may try to run
> some more isolated tests, but it was an intermittent issue that required a
> few hours of gaming to flush out… so might take a bit there =\
>
> On Tue, May 31, 2016 at 3:34 PM Alex Williamson <
> alex.williamson at redhat.com> wrote:
>
>> On Tue, 31 May 2016 20:20:58 +0000
>> Colin Godsey <crgodsey at gmail.com> wrote:
>>
>> > I had a few questions regarding general ‘page management’ and VFIO,
>> mostly
>> > related to kernel shared pages.
>> >
>> > I have a host running 2 virtual ‘gaming rigs’ with a single dedicated
>> GPU
>> > each. I had an intermittent problem where when gaming (on the same game)
>> > with both rigs, one would receive graphic artifacts. Specifically I
>> would
>> > see triangle/geometry artifacts which usually indicate corrupt GPU RAM.
>> >
>> > Both cards are so completely different, and different generation, one is
>> > really new, I didn’t believe it was bad VRAM. Graphics drivers to swap
>> > various buffers from system RAM to VRAM so I figured it could also be
>> > something related to system RAM.
>> >
>> > I disabled any kind of… alternative page management I could- swap, KSM,
>> > huge pages etc. and it did fix it. Because the issue only would effect
>> one
>> > machine, and I only observed it when the same game was running on both,
>> I
>> > assumed maybe it was related to KSM.
>> >
>> > *Is there any possible way KSM could interfere with the DMAR in some way
>> > where it tries to share/alter DMA regions?* And broader: what prevents
>> > systems like khugepage, kswap, and ksm from interfering with these
>> regions
>> > in the first place? I’ve read that transparent hugepages can interfere
>> with
>> > VFIO, is it safe to assume that other DMA issues could arise with other
>> > types of page management?
>>
>> What kernel were you running where you saw this?  vfio uses
>> get_user_pages to increase the reference count on pages mapped through
>> the iommu.  This should prevent both ksm and transparent hugepages from
>> being able to operate on the pages.  Kernel v4.5 had a bug (now fixed
>> in v4.5.5) that did not honor the reference, allowing thp (maybe ksm
>> too) to still operate on those pages.  So as long as you're not running
>> v4.5.0 through v4.5.4 (or a v4.5-rc), I'm not aware of any issues with
>> page pinning.  Thanks,
>>
>> Alex
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160606/aad1bd71/attachment.htm>