[vfio-users] Passthru of one GPU with a PC with 2 identical GPUs installed

[Alex Williamson]

On Wed, 22 May 2019 21:47:20 +0100
James Courtier-Dutton <james.dutton at gmail.com> wrote:

> On Wed, 22 May 2019 at 00:11, Alex Williamson <alex.williamson at redhat.com>
> wrote:
> 
> >
> > I think a better approach would be to extend the pci= kernel command
> > line option to include driver_override support, perhaps something like:
> >
> > pci=...,driver_overrides=<pci_dev1>=<driver1>;<pci_dev2>=<driver2>,
> >
> >
> > OK, lets assume for a moment, that I will try to implement the above.  
> Mindful of the fact that if there is a bug in this code, it might upset a
> far wider amount of people than my previous suggestion that would only
> affect vfio-pci users.
> I understand that rather than the driver selectively ignoring a device
> probe for a specific device, this method would restrict the probe call
> itself, to only call the mentioned driver.
> 
> Use Case 1: I can understand how the pci=...  line can go in the kernel
> boot command line and kind of force the driver_binding of pci_dev and its
> driver.
> Use Case 2: What I am not so sure about is how this behavior can be
> modified after boot. Say, after boot, someone wished to un-bind it from one
> driver, and then bind it to a different one, how would we undo the
> driver_overrides so that it could then call a device probe on a different
> driver?

PCI devices expose driver_override in sysfs, this mechanism would
simply provide an initial value for that rather than null.  The device
can be returned to the normal driver by unbinding, writing an empty
string to the driver_override for the device, and forcing a driver
probe.  The driverctl utility already facilitates this, see 'driverctl
unset-override'.  What cannot be undone afaik is should the device be
removed and re-added, the override would be put back in place.  Seems
like a narrow use case though for that though.

> Use Case 3: A pci_dev is set on the command line, but it is not yet present
> in the system, but we wish the setting to be there for when that device is
> hot-plugged.

Should work.  The override should be part of the device discovery
process before any driver would have a chance to claim it, it shouldn't
matter whether the device exists from boot or is added later.
 
> Are there any other use cases I should consider?
> Might we wish to consider an option so the user can, for a specific device,
> unbind it from one driver and bind it to a different one.

Already exists, this is exactly why driver_override was added to the
kernel.

> I have also noticed this interface:
> https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci
> Where 0000:00:19.0 format is used for the bind/unbind methods, and not your
> device path suggestion.
> For consistency, would we need to add support for device path to the
> bind/unbind methods?

It could be added, but I don't think it's necessary or a prerequisite.
The bind/unbind interfaces are not persistent, the device has already
been given this address in the current environment, that's not the case
at boot time.

This has been something on my todo list for a while, so I appreciate
you taking a swing at it.  I'm willing to help or maybe take over if
it becomes too much.  Thanks,

Alex