[vfio-users] Convenience script for binding to vfio-pci

[Alex Williamson]

On Mon, 2015-09-14 at 17:24 +0200, Andre Richter wrote:
> Hi Alex,
> 
> thanks for the quick feedback!
> I'll incorporate your changes tomorrow.
> 
> 2015-09-14 16:51 GMT+02:00 Alex Williamson <alex.williamson at redhat.com>:
> > Hi Andre,
> >
> > On Mon, 2015-09-14 at 14:23 +0000, Andre Richter wrote:
> >> Hi,
> >>
> >> just in case anybody is interested, I want to share a little convenience
> >> script of mine:
> >>
> >> https://github.com/andre-richter/vfio-pci-bind
> >>
> >> This script takes a Bus:Device:Function string of the form "00:00.0"
> >> as command line argument and:
> >>
> >> 1. Unbinds all devices that are in the same iommu group as the supplied
> >> device from their current driver (except PCIe bridges).
> >>
> >> 2. Binds to vfio-pci:
> >>   1. The supplied device.
> >>   2. All unbound devices that have the same device_id and vendor_id as the
> >> supplied device.
> >>   3. All devices that are in the same iommu group.
> >>
> >> 3. Transfers ownership of the respective iommu groups inside /dev/vfio to
> >> $SUDO_USER
> >>
> >> Script must be executed via sudo!
> >
> >
> > This looks fragile:
> >
> > unless File.exist?('/sys/bus/pci/devices/0000:00:00.0/iommu/')
> >   puts 'No signs of an IOMMU. Check your hardware and/or linux cmdline parameters.'
> >   puts 'Use intel_iommu=on or iommu=pt iommu=1'
> >   exit
> > end
> >
> > Why not check the iommu link for the device to be attached, or look
> > in /sys/class/iommu rather than assume device 0000:00:00.0?
> >
> > TARGET_DEV_SYSFS_PATH = "/sys/bus/pci/devices/0000:#{bdf}/"
> >
> > You're assuming domain 0000.
> 
> I have never seen a machine with a domain greater than 0000, so I got
> lazy there, but ofc you're right, will fix.
> 
> >
> > # Do not care about bridges
> > dev_sysfs_paths.delete_if { |dsp| File.read(dsp + '/class')[0..5] == '0x0604' }
> >
> > Testing for HEADER_TYPE 0 would be more compatible and match the patch I
> > proposed for whitelisting all bridges, not just those w/o a driver or
> > bound to pcieport.  Ex
> >
> >
> > if [ $(( 0x$(setpci -s 0000:00:1c.0 0e.b) & 0x7f )) -ne 0 ]; then
> >     echo bridge
> > fi
> >
> >
> >   # Add id to vfio-pci driver
> >   File.write('/sys/bus/pci/drivers/vfio-pci/new_id', devid)
> >
> > If you used /sys/bus/pci/devices/ssss:bb:dd.f/driver_override you could
> > avoid the "feature" of binding all devices with the same vendor:device
> > ID.
> 
> Actually I was wondering what a reasonable approach is when freeing an
> IOMMU group with multiple devices.
> Leave the other devices unbound? Bind them to pci-stub? What do you suggest?

I'd just do something like this:

To bind:

echo vfio-pci > .../devices/ssss:bb:dd.f/driver_override
echo ssss:bb:dd.f > .../drivers/vfio-pci/bind
echo > .../devices/ssss:bb:dd.f/driver_override

To unbind:

echo ssss:bb:dd.f > .../drivers/vfio-pci/unbind
echo ssss:bb:dd.f > /sys/bus/pci/drivers_probe

The new_id, remove_id interface is racy, so if you don't touch it,
drivers_probe should always pickup what the kernel would have had the
device bound to.

> > # Adjust ownership
> > ids = ENV['SUDO_UID'] + ':' + ENV['SUDO_GID']
> > Dir.glob('/dev/vfio/[0-9]*').each { |f| `chown #{ids} #{f}` }
> >
> > Is this changing the ownership of *all* the groups to the current user?
> > Why not just the group affected?  Thanks,
> >
> > Alex
> >
> 
> Cheers,
> Andre