<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 19, 2019 at 12:27 AM Alex Williamson <<a href="mailto:alex.williamson@redhat.com">alex.williamson@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Tue, 18 Jun 2019 14:48:11 +0200<br>
Sylvain Bauza <<a href="mailto:sbauza@redhat.com" target="_blank">sbauza@redhat.com</a>> wrote:<br>
<br>
> On Tue, Jun 18, 2019 at 1:01 PM Cornelia Huck <<a href="mailto:cohuck@redhat.com" target="_blank">cohuck@redhat.com</a>> wrote:<br>
> <br>
> > On Mon, 17 Jun 2019 11:05:17 -0600<br>
> > Alex Williamson <<a href="mailto:alex.williamson@redhat.com" target="_blank">alex.williamson@redhat.com</a>> wrote:<br>
> > <br>
> > > On Mon, 17 Jun 2019 16:10:30 +0100<br>
> > > Daniel P. Berrangé <<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>> wrote:<br>
> > > <br>
> > > > On Mon, Jun 17, 2019 at 08:54:38AM -0600, Alex Williamson wrote: <br>
> > > > > On Mon, 17 Jun 2019 15:00:00 +0100<br>
> > > > > Daniel P. Berrangé <<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>> wrote:<br>
> > > > > <br>
> > > > > > On Thu, May 23, 2019 at 05:20:01PM -0600, Alex Williamson wrote: <br>
> > <br>
> > > > > > > Hi,<br>
> > > > > > ><br>
> > > > > > > Currently mediated device management, much like SR-IOV VF <br>
> > management, <br>
> > > > > > > is largely left as an exercise for the user. This is an attempt <br>
> > to <br>
> > > > > > > provide something and see where it goes. I doubt we'll solve<br>
> > > > > > > everyone's needs on the first pass, but maybe we'll solve enough <br>
> > and <br>
> > > > > > > provide helpers for the rest. Without further ado, I'll point <br>
> > to what <br>
> > > > > > > I have so far:<br>
> > > > > > ><br>
> > > > > > > <a href="https://github.com/awilliam/mdevctl" rel="noreferrer" target="_blank">https://github.com/awilliam/mdevctl</a><br>
> > > > > > ><br>
> > > > > > > This is inspired by driverctl, which is also a bash utility. <br>
> > mdevctl <br>
> > > > > > > uses udev and systemd to record and recreate mdev devices for<br>
> > > > > > > persistence and provides a command line utility for querying, <br>
> > listing, <br>
> > > > > > > starting, stopping, adding, and removing mdev devices. <br>
> > Currently, for <br>
> > > > > > > better or worse, it considers anything created to be <br>
> > persistent. I can <br>
> > > > > > > imagine a global configuration option that might disable this and<br>
> > > > > > > perhaps an autostart flag per mdev device, such that mdevctl <br>
> > might <br>
> > > > > > > simply "know" about some mdevs but not attempt to create them<br>
> > > > > > > automatically. Clearly command line usage help, man pages, and<br>
> > > > > > > packaging are lacking as well, release early, release often, <br>
> > plus this <br>
> > > > > > > is a discussion starter to see if perhaps this is sufficient to <br>
> > meet <br>
> > > > > > > some needs. <br>
> > > > > ><br>
> > > > > > I think from libvirt's POV, we would *not* want devices to be made<br>
> > > > > > unconditionally persistent. We usually wish to expose a choice to<br>
> > > > > > applications whether to have resources be transient or persistent.<br>
> > > > > ><br>
> > > > > > So from that POV, a global config option to turn off persistence<br>
> > > > > > is not workable either. We would want control per-device, with<br>
> > > > > > autostart control per device too. <br>
> > > > ><br>
> > > > > The code has progressed somewhat in the past 3+ weeks, we still <br>
> > persist <br>
> > > > > all devices, but the start-up mode can be selected per device or <br>
> > with a <br>
> > > > > global default mode. Devices configured with 'auto' start-up<br>
> > > > > automatically while 'manual' devices are simply known and available <br>
> > to <br>
> > > > > be started. I imagine we could add a 'transient' mode where we purge<br>
> > > > > the information about the device when it is removed or the next time<br>
> > > > > the parent device is added. <br>
> > > ><br>
> > > > Having a pesistent config written out & then purged later is still<br>
> > > > problematic. If the host crashes, nothing will purge the config file,<br>
> > > > so it will become a persistent device. Also when listing devices we<br>
> > > > want to be able to report whether it is persistent or transient. The<br>
> > > > obvious way todo that is to simply look if a config file exists or<br>
> > > > not. <br>
> > ><br>
> > > I was thinking that the config file would identify the device as<br>
> > > transient, therefore if the system crashed we'd have the opportunity to<br>
> > > purge those entries on the next boot as we're processing the entries<br>
> > > for that parent device. Clearly it has yet to be implemented, but I<br>
> > > expect there are some advantages to tracking devices via a transient<br>
> > > config entry or else we're constantly re-discovering foreign mdevs. <br>
> ><br>
> > I think we need to reach consensus about the actual scope of the<br>
> > mdevctl tool.<br>
> ><br>
> > <br>
> Thanks Cornelia, my thoughts:<br>
> <br>
> - Is it supposed to be responsible for managing *all* mdev devices in<br>
> > the system, or is it more supposed to be a convenience helper for<br>
> > users/software wanting to manage mdevs?<br>
> > <br>
> <br>
> The latter. If an operator (or some software) wants to create mdevs by not<br>
> using mdevctl (and rather directly calling the sysfs), I think it's OK.<br>
> That said, mdevs created by mdevctl would be supported by systemctl, while<br>
> the others not but I think it's okay.<br>
<br>
I agree (sort of), and I'm hearing that we should drop any sort of<br>
automatic persistence of mdevs created outside of mdevctl. The problem<br>
comes when we try to draw the line between unmanaged and manged<br>
devices. For instance, if we have a command to list mdevs it would<br>
feel incomplete if it didn't list all mdevs both those managed by<br>
mdevctl and those created elsewhere. For managed devices, I expect<br>
we'll also have commands that allow the mode of the device to be<br>
switched between transient, saved, and persistent. Should a user then<br>
be allowed to promote an unmanaged device to one of these modes via the<br>
same command? Should they be allowed to stop an unmanaged device<br>
through driverctl? Through systemctl? These all seem like reasonable<br>
things to do, so what then is the difference between transient and<br>
unmanaged mdev and is mdevctl therefore managing all mdevs, not just<br>
those it has created?<br>
<br></blockquote><div><br></div><div>Well, IMHO, mdevs created by mdevctl could all be persisted or transient just by adding an option when calling mdevctl, like :</div><div>"mdevctl create-mdev [--transient] <uuid> <pci_id> <type>" where default would be persisting the mdev.<br></div><div></div><div><br></div><div>For mdevs *not* created by mdevctl, then a usecase could be "I'd like to ask mdevctl to manage mdevs I already created" and if so, a mdevctl command like :</div><div>"mdevctl manage-mdev [--transient] <mdev_uuid>"</div><div><br></div><div>Of course, that would mean that when you list mdevs by "mdev list-all" you wouldn't get mdevs managed by mdevctl.</div><div>Thoughts ?</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> - Do we want mdevctl to manage config files for individual mdevs, or<br>
> > are they supposed to be in a common format that can also be managed<br>
> > by e.g. libvirt?<br>
> > <br>
> <br>
> Unless I misunderstand, I think mdevctl just helps to create mdevs for<br>
> being used by guests created either by libvirt or QEMU or even others.<br>
> How a guest would allocate a mdev (ie. saying "I'll use this specific mdev<br>
> UUID") is IMHO not something for mdevctl.<br>
<br>
Right, mdevctl isn't concerned with how a specific mdev is used, but I<br>
think what Connie is after is more the proposal from Daniel where<br>
libvirt can essentially manage mdevctl config files itself and then<br>
only invoke mdevctl for the dirty work of creating and deleting<br>
devices. In fact, assuming systemd, libvirt could avoid direct<br>
interaction with mdevctl entirely, instead using systemctl device units<br>
to start and stop the mdevs. Maybe where that proposal takes a turn is<br>
when we again consider non-systemd hosts, where maybe mdevctl needs to<br>
write out an init script per mdev and libvirt injecting itself into<br>
manipulation of the config files would either need to perform the same<br>
or fall back to mdevctl. Unfortunately there seems to be an ultimatum<br>
to either condone external config file manipulation or expand the scope<br>
of the project into becoming a library.<br>
<br></blockquote><div><br></div><div>Well, like I said, I think it's maybe another user case : just using libvirt when you want a guest having vGPUs and then libvirt would create mdevs (so users wouln't need to know at that).</div><div>That said, for the moment, I think we don't really need it so maybe a new RFE once we at least have mdevctl packaged and supported by RHEL ?</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> - Should mdevctl be a stand-alone tool, provide library functions, or<br>
> > both? Related: should it keep any internal state that is not written<br>
> > to disk? (I think that also plays into the transient vs. persistent<br>
> > question.)<br>
<br>
I don't think we want an mdevctld, if that's what you mean by internal<br>
state not written to disk. I think we ideally want all state in the<br>
mdev config files or discerned through sysfs. How we handle<br>
non-systemd hosts may throw a wrench in that though since currently the<br>
systemd integration relies on a template to support arbitrary mdevs and<br>
I'm not sure how to replicate that in other init services. If we need<br>
to dynamically manage per mdev init files in addition to config files,<br>
we're not so self contained.<br>
<br>
> FWIW, I'd love using mdevctl for OpenStack (Nova) just at least for<br>
> creating persisted mdevs (ie. mdevs that would be recreated after rebooting<br>
> using systemctl). That's the real use case I need.<br>
> Whether libvirt would internally support mdevctl would be nice but that's<br>
> not really something Nova needs, so I leave others providing their own<br>
> thoughts.<br>
> <br>
> <br>
> My personal opinion is that mdevctl should be able to tolerate mdevs<br>
> > being configured by other means, but probably should not try to impose<br>
> > its own configuration if it detects that (unless explicitly asked to do<br>
> > so). Not sure how feasible that goal is.<br>
> ><br>
> > That's what I misunderstand : in order to have a guest using a vGPU, you <br>
> need to do two things :<br>
> 1/ create the mdev<br>
> 2/ allocate this created dev to a specific guest config<br>
> <br>
> Of course, we could imagine a way to have both steps to be done directly by<br>
> libvirt, but from my opinion, mdevctl is really helping 1/ and not 2/.<br>
<br>
Yep, we also don't want to presume libvirt is the only consumer here.<br>
mdevctl should also support other VM management tools, users who write<br>
their own management scripts, and even non-VM related use cases.<br>
<br></blockquote><div><br></div><div>Oh yes, please don't premuse mdevctl is only needed by libvirt.</div><div>FWIW, once mdevctl is supported by RHEL, I'd love to use it for OpenStack Nova at least because I want to persist the mdevs.</div><div>At the moment, Nova just creates mdevs directly by sysfs and look the existing onces up by sysfs, but upstream community in Nova thinks the mission statement is not about managing mdevs so we don't really want to add in Nova some kind of DB persistence just for mdevs.</div><div></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> > A well-defined config file format is probably a win, even if it only<br>
> > ends up being used by mdevctl itself.<br>
<br>
Yes, regardless of whether others touch them, conversion scripts on<br>
upgrade should be avoided. Do we need something beyond a key=value<br>
file? So far we're only storing the mdev type and startup mode, but<br>
vfio-ap clearly needs more, apparently key=value1,value2,... type<br>
representation. Still, I think I'd prefer simple over jumping to xml<br>
or json or yaml. Thanks,<br>
<br></blockquote><div><br></div><div>Heh, in OpenStack discussing about a file format is possibly one of the longest arguments we already have, so I leave others to say their own opinions but FWIW, as we use Python we tend to prefer YAML files. I don't care about the format tho, just take the most convenient for libvirt I'd say.</div><div><br></div><div>-Sylvain</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Alex<br>
</blockquote></div></div>