<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 23, 2021 at 6:47 AM Vivek Goyal <<a href="mailto:vgoyal@redhat.com">vgoyal@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">On Tue, Mar 23, 2021 at 12:55:26PM +0100, Sergio Lopez wrote:<br>
> On Mon, Mar 22, 2021 at 12:47:04PM -0400, Vivek Goyal wrote:<br>
> > On Mon, Mar 22, 2021 at 05:09:32PM +0100, Miklos Szeredi wrote:<br>
> > > On Mon, Mar 22, 2021 at 6:52 AM Eric Ernst <<a href="mailto:eric_ernst@apple.com" target="_blank">eric_ernst@apple.com</a>> wrote:<br>
> > > ><br>
> > > > Hey ya’ll,<br>
> > > ><br>
> > > > One challenge I’ve been looking at is how to setup an appropriate memory cgroup limit for workloads that are leveraging virtiofs (ie, running pods with Kata Containers). I noticed that memory usage of the daemon itself can grow considerably depending on the workload; though much more than I’d expect.<br>
> > > ><br>
> > > > I’m running workload that simply runs a build on kernel sources with -j3. In doing this, the source of the linux kernel are shared via virtiofs (no DAX), so as the build goes on, there are a lot of files opened, closed, as well as created. The rss memory of virtiofsd grows into several hundreds of MBs.<br>
> > > ><br>
> > > > When taking a look, I’m suspecting that virtiofsd is carrying out the opens, but never actually closing fds. In the guest, I’m seeing fd’s on the order of 10-40 for all the container processes as it runs, whereas I see the number of fds for virtiofsd continually increasing, reaching over 80,000 fds. I’m guessing this isn’t expected?<br>
> > > <br>
> > > The reason could be that guest is keeping a ref on the inodes<br>
> > > (dcache->dentry->inode) and current implementation of server keeps an<br>
> > > O_PATH fd open for each inode referenced by the client.<br>
> > > <br>
> > > One way to avoid this is to use the "cache=none" option, which forces<br>
> > > the client to drop dentries immediately from the cache if not in use.<br>
> > > This is not desirable if cache is actually in use.<br>
> > > <br>
> > > The memory use of the server should still be limited by the memory use<br>
> > > of the guest: if there's memory pressure in the guest kernel, then it<br>
> > > will clean out caches, which results in the memory use decreasing in<br>
> > > the server as well. If the server memory use looks unbounded, that<br>
> > > might be indicative of too much memory used for dcache in the guest<br>
> > > (cat /proc/slabinfo | grep ^dentry). Can you verify?<br>
> > <br>
> > Hi Miklos,<br>
> > <br>
> > Apart from above, we identified one more issue on IRC. I asked Eric<br>
> > to drop caches manually in guest. (echo 3 > /proc/sys/vm/drop_caches)<br>
> > and while it reduced the fds open it did not seem to free up significant<br>
> > amount of memory.<br>
> > <br>
> > So question remains where is that memory. One possibility is that we<br>
> > have memory allocated for mapping arrays (inode and fd). These arrays<br>
> > only grow and never shrink. So they can lock down some memory.<br>
> > <br>
> > But still, lot of lo_inode memory should have been freed when<br>
> > echo 3 > /proc/sys/vm/drop_caches was done. Why all that did not<br>
> > show up in virtiofsd RSS usage, that's kind of little confusing.<br>
> <br>
> Are you including "RssShmem" in "RSS usage"? If so, that could be<br>
> misleading. When virtiofsd[-rs] touches pages that reside in the<br>
> memory mapping that's shared with QEMU, those pages are accounted<br>
> in the virtiofsd[-rs] process's RssShmem too.<br>
> <br>
> In other words, the RSS value of the virtiofsd[-rs] process may be<br>
> overinflated because it includes pages that are actually shared<br>
> (there's no a second copy of them) with the QEMU process.<br>
> <br>
> This can be observed using a tool like "smem". Here's an example<br>
> <br>
> - This virtiofsd-rs process appears to have a RSS of ~633 MiB<br>
> <br>
> root 13879 46.1 7.9 8467492 649132 pts/1 Sl+ 11:33 0:52 ./target/debug/virtiofsd-rs<br>
> root 13947 69.3 13.4 5638580 1093876 pts/0 Sl+ 11:33 1:14 qemu-system-x86_64<br>
> <br>
> - In /proc/13879/status we can observe most of that memory is<br>
> actually RssShmem:<br>
> <br>
> RssAnon: 9624 kB<br>
> RssFile: 5136 kB<br>
> RssShmem: 634372 kB<br>
<br>
Hi Sergio,<br>
<br>
Thanks for this observation about RssShmem. I also ran virtiofsd and<br>
observed memory usage just now and it indeed looks like that only<br>
RssShmem usage is very high.<br>
<br>
RssAnon: 4884 kB<br>
RssFile: 1900 kB<br>
RssShmem: 1050244 kB<br>
<br>
And as you point out that this memory is being shared with QEMU. So<br>
looks like from cgroup point of view, we should put virtiofsd and<br>
qemu in same cgroup and have a combined memory limit so that this<br>
shared memory is accounting looks proper.<br>
<br>
Eric, does this sound reasonable.<br></blockquote><div><br></div><div>Sergio, Vivek --</div><div><br></div><div>Today QEMU/virtiofsd do live within the same memory cgroup, and are bound by that same overhead I need to introduce. Good to know regarding the sharing (this restores some sanity to my observations, thank you!), but the real crux of the problem is two items:</div><div> 1) the FDs are held long after the application in guest is done with them because of dentry cache in the guest (when cache=auto for virtiofsd).</div><div> 2) virtiofsd/QEMU is holding on to the memory after the fds are released</div><div><br></div><div>--Eric</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<br>
Thanks<br>
Vivek<br>
<br>
> <br>
> - In "smem", we can see a similar amount of RSS, but the PSS is<br>
> roughly half the size because "smem" is splitting it up between<br>
> virtiofsd-rs and QEMU:<br>
> <br>
> [root@localhost ~]# smem -P virtiofsd-rs -P qemu<br>
> PID User Command Swap USS PSS RSS <br>
> 13879 root ./target/debug/virtiofsd-rs 0 13412 337019 662392 <br>
> 13947 root qemu-system-x86_64 -enable- 0 434224 760096 1094392 <br>
> <br>
> - If we terminate the virtiofsd-rs process, the output of "smem" now<br>
> shows that QEMU's PSS has grown to account for the PSS that was<br>
> previously assigned to virtiofsd-rs too, so we can confirm that was<br>
> memory shared between both processes.<br>
> <br>
> PID User Command Swap USS PSS RSS <br>
> 13947 root qemu-system-x86_64 -enable- 0 1082656 1084966 1095692 <br>
> <br>
> Just to be 100% sure, I've also run "heaptrack" on a virtiofsd-rs<br>
> instance, and can confirm that the actual heap usage of the process<br>
> was around 5-6 MiB.<br>
> <br>
> Sergio.<br>
<br>
<br>
_______________________________________________<br>
Virtio-fs mailing list<br>
<a href="mailto:Virtio-fs@redhat.com" target="_blank">Virtio-fs@redhat.com</a><br>
<a href="https://listman.redhat.com/mailman/listinfo/virtio-fs" rel="noreferrer" target="_blank">https://listman.redhat.com/mailman/listinfo/virtio-fs</a><br>
</blockquote></div></div>