[Virtio-fs] xfstest generic/503 hangs

[Liu Bo]

On Mon, Mar 23, 2020 at 07:18:57PM +0100, Max Reitz wrote:
> Hi,
> 
> I have this bug report here:
> https://bugzilla.redhat.com/show_bug.cgi?id=1813885
> 
> And I’m afraid I’m not really making progress on debugging it, so I was
> wondering whether any of you might have some insights.
> 
> The problem is that the generic/503 xfstest hangs on virtio-fs.  Now, I
> don’t know how the reporter got that test to run in the first place,
> because for me, it requires fcollapse and fzero, which as far as I can
> tell are currently not supported for virtio-fs.
> 
> So I first had to disable those requirements, and then let the helper
> program (src/t_mmap_collision.c) not test those operations.
> 
> Then, the test hangs.  What I could find out so far is that the hang
> occurs in src/t_mmap_collision.c’s truncate_down_fn() (run through
> run_test(&truncate_down_fn), namely in one of the pread()s.  I can also
> see that some of the pread()s before fail with EFAULT.
> 
> A bit more context: t_mmap_collision.c opens a test file twice (I think
> the idea is that you open it once on an FS with DAX, and once without,
> but AFAIU it should work either way).  For the relevant test, it mmap()s
> the DAX FD, truncates it, then fallocates it to increase the size again.
>  Then it reads from the non-DAX FD.
> 
> It does all of that in two threads simultaneously for a second.
> 
> The EFAULT seems to come from the guest kernel.  I don’t see virtiofsd
> returning an error anywhere.  I don’t know where it comes from exactly,
> only that when I replace all occurrences of “EFAULT” by e.g. “EBADSLT”
> in mm/, the test crashes instead of hanging, so I take that to mean that
> the error comes from something in mm/ (which I suppose isn’t too
> unexpected).
> 
> The test passes if running the test function in a single thread instead
> of two, or if you use a separate TEST_DEV and SCRATCH_DEV – but in the
> latter case, you really have two separate files, so the test becomes
> rather moot (AFAIU).
> 
> The fact that truncate_down_fn() uses fallocate() seems irrelevant.
> When you replace it by ftruncate() (i.e. the dax_fd is just first
> truncated to 0, and then truncated back to @file_size), the test fails
> in the same way.  So maybe there is some interaction between the
> ftruncate() and a concurrent pread()?  But where does the EFAULT come from?
> 
> Does anyone have any spontaneous ideas? :/
>

When it comes to "hang", I guess it's a similar problem of guest dax
reading while host truncates the file, this can be verify by the hang
stack, did it hang at dax_iomap_actor() -> dax_copy_to_iter()?

I think we did use fuse_break_layout() to avoid pinned blocks being
truncated underneath in this kind of mmap problems, but seems it
doesn't work as expected.

thanks,
-liubo