[PATCH 5/5] node_device: mdev vfio-ccw support

[Erik Skultety]

On Wed, Sep 09, 2020 at 05:29:20PM +0200, Cornelia Huck wrote:
> On Wed, 9 Sep 2020 14:40:33 +0200
> Erik Skultety <eskultet at redhat.com> wrote:
>
> > On Wed, Sep 09, 2020 at 08:23:39AM +0200, Bjoern Walk wrote:
> > > Erik Skultety <eskultet at redhat.com> [2020-09-08, 06:01PM +0200]:
> > > > On Mon, Aug 24, 2020 at 01:59:15PM +0200, Bjoern Walk wrote:
> > > > > From: Boris Fiuczynski <fiuczy at linux.ibm.com>
> > > > > +        case VIR_NODE_DEV_CAP_SYSTEM:
> > > > > +        case VIR_NODE_DEV_CAP_USB_DEV:
> > > > > +        case VIR_NODE_DEV_CAP_USB_INTERFACE:
> > > > > +        case VIR_NODE_DEV_CAP_NET:
> > > > > +        case VIR_NODE_DEV_CAP_SCSI_HOST:
> > > > > +        case VIR_NODE_DEV_CAP_SCSI_TARGET:
> > > > > +        case VIR_NODE_DEV_CAP_SCSI:
> > > > > +        case VIR_NODE_DEV_CAP_STORAGE:
> > > > > +        case VIR_NODE_DEV_CAP_FC_HOST:
> > > > > +        case VIR_NODE_DEV_CAP_VPORTS:
> > > > > +        case VIR_NODE_DEV_CAP_SCSI_GENERIC:
> > > > > +        case VIR_NODE_DEV_CAP_DRM:
> > > > > +        case VIR_NODE_DEV_CAP_MDEV_TYPES:
> > > > > +        case VIR_NODE_DEV_CAP_MDEV:
> > > > > +        case VIR_NODE_DEV_CAP_CCW_DEV:
> > > >
> > > > What about ccw mdevs? I would think that it's CCW end point device that
> > > > we would want to assign concurrently, but we're only "slicing" the root channel
> > > > subsystem device. If you have and CSS mdev, are the CCW subchannels separate
> > > > for each VM or does it behave kind of like NPIV with its vHBAs? OR I'm
> > > > completely off here?
> > >
> > > You are correct. This is an unfortunate architectural decision on our
> > > platform. For vfio-ccw, we instatiate the mdevs on the CSS layer, which
> > > has a 1-to-1 relation to the underlying CCW device. That's why we need
> > > the information about the CSS devices in libvirt, to give the user a
> > > chance to easily find this relation when he wants to do passthrough with
> > > a DASD for example. It's a bit unintuitive and even silly for the
> > > end user, but it's architecturally correct and hence it was implemented
> > > like this in the vfio-ccw kernel driver. We don't even get the benefits
> > > of something like NPIV because for every subchannel there is only one
> > > CCW device (at least to my knowledge).
> >
> > Having read both Connie's blog on channels and the I/O chapter in the free
> > architectural publication you linked in the other reply, I now have a very
> > vague understating of the architecture. One thing that still
> > puzzles me though is that if there's a 1:1 subchannel mapping between CSS and an
> > CCW device which then communicates via a control unit (which there may be more
> > than 1) with CSS over up to 8 channel paths (this at least to me resembles
> > NPIV), what does the mdev really solve, can't you assign one of the channel
> > paths of one of the several control units the device is connected to to
> > plug into the VM? Well, clearly not, I'm just being curious and trying to
> > understand the use case here.
>
> I think it's easier to leave control units mostly out of the picture,
> as they are not really relevant for the modelling here. And yes, there
> is always exactly one subchannel for an I/O device.
>
> Subchannels (mostly a concept describing how to interact with a device)
> and channel paths (mostly corresponding to the way data flows to/from
> the device) are probably best understood as different ways to describe
> how devices are accessed. You don't really want to pass channel paths
> (as they are shared between many devices); the subchannel is a more
> natural unit to pass. The operating system also has way more control
> over subchannels than over individual channel paths.

Gotcha.

>
> Regarding mdev, the idea here is less to be able to slice up a device
> as you do with GPUs, but more to be able to add a vendor driver that
> does the actual interesting work (channel program translations etc.)
>
> As to why we use the subchannel instead of the device, that's mostly
> following from how the low-level channel I/O layer in the Linux kernel
> is organized: The default I/O subchannel driver does a lot of things
> that a normal device driver should not need to deal with (path
> grouping, verification, status accumulation, etc.), but which would get
> into the way if you wanted to assign the device via vfio. Therefore,
> vfio-ccw implements an alternate I/O subchannel driver.

Okay, now it's much clearer what the mdev use case is.

Thanks Connie.

Erik