[libvirt] [PATCH v2 RESEND 00/12] PCI passthrough support on s390

[Boris Fiuczynski]

On 07/24/2018 10:20 AM, Andrea Bolognani wrote:
> On Mon, 2018-07-23 at 13:37 +0200, Ján Tomko wrote:
>> On Tue, Jul 10, 2018 at 04:02:14PM +0800, Yi Min Zhao wrote:
>>> Abstract
>>> ========
>>> The PCI representation in QEMU has recently been extended for S390
>>> allowing configuration of zPCI attributes like uid (user-defined
>>> identifier) and fid (PCI function identifier).
>>> The details can be found here:
>>> https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg07262.html
>>>
>>> To support the new zPCI feature of the S390 platform, two new XML
>>> attributes, @uid and @fid, are introduced for device addresses of type
>>> 'pci', i.e.:
>>>   <hostdev mode='subsystem' type='pci'>
>>>     <driver name='vfio'/>
>>>     <source>
>>>       <address domain='0x0001' bus='0x00' slot='0x00' function='0x0'/>
>>>     </source>
>>>     <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'
>>>       uid='0x0003' fid='0x00000027'/>
>>>   </hostdev>
>>>
>>> uid and fid are optional attributes. If they are defined by the user,
>>> unique values within the guest domain must be used. If they are not
>>> specified and the architecture requires them, they are automatically
>>> generated with non-conflicting values.
>>>
>>> Current implementation is the most seamless one for the user as it
>>> unites the address specific data of a PCI device on one XML element.
>>> It could accommodate both specifying our special parameters (uid and fid)
>>> and re-using standard statements (domain, bus, slot and function) for
>>> PCI devices. User can still specify bus/slot/function for the virtualized
>>> PCI devices in the XML.
>>>
>>> Thus uid/fid act as an extension to the PCI address and are stored in
>>> a new structure 'virZPCIDeviceAddress' which is a member of common PCI
>>> Address structure. Additionally, two hashtables are used for assignment
>>> and reservation of uid/fid.
>>>
>>> In support of extending the PCI address, a new PCI address extension flag is
>>> introduced. This extension flag allows is not only dedicated for the S390
>>> platform but also other architectures needing certain extensions to PCI
>>> address space.
> [...]
>>
>> The patches look OK to me, therefore:
>>
>> Reviewed-by: Ján Tomko <jtomko at redhat.com>
>>
>> But I'd like to hear from some other developer who touched the PCI code
>> last (Laine? Andrea?)
> 
> Thanks for bringing this to my attention - it had somehow managed
> to escape my radar so far :)
> 
> I've quickly gone through the patches and spotted some minor code
> style issues that I'd like to see addressed (I'll point them out
> separately); first, though, I have a couple of questions about the
> general idea behind the feature.
> 
> Looking at both the generated QEMU command line and the qemu-devel
> message linked above, I seem to understand the zpci device is
> basically some sort of adapter that sits between a regular PCI
> device (emulated or otherwise) and an s390 guest and presents an
> ID-based interface to the latter; so IIUC the s390 guest doesn't
> actually see a PCI device, but a couple of IDs that can (somehow)
> be used to access the underlying PCI device.
> 
>>From whatever little s390 knowledge I have, I recall the
> architecture using peculiar ways to address and access devices, so
> PCI not being usable wouldn't surprise me that much; what I find
> odd, however, is that regular PCI seems to be available at least
> on the host side, because otherwise stuff like
> 
>    -device zpci,uid=1,fid=1,target=vpci0
>    -device vfio-pci,host=0000:00:00.0,id=vpci0
> 
> wouldn't be possible. So, would anyone with s390 knowledge please
> spend a few words explaining how the various pieces fit together?
> 
> Assuming the above is a correct reading of the situation, it
> would seem the IDs are the only guest-visible part that we need
> to make sure doesn't change during the lifetime of the guest,
> while the usual bus/slot/function triplet assigned to the
> underlying PCI device doesn't actually matter. And if that's the
> case, wouldn't something like
> 
>    <address type='zpci' uid='1' fid='1'/>

Then a pci device on s390 would need a special address type zpci in 
libvirt and the design idea for libvirt is to stay compatible with pci.
Therefore uid and fid are optional attributes and if not specified on 
s390 they are generated. The patch series also allows on s390 to specify 
the pci address type just with attributes uid and/or fid causing the 
rest of the pci address attributes to be generated.
That means
<address type='pci' uid='1' fid='1'/>
is a valid pci address on s390 but would get expanded during definition 
into e.g.
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x0' uid='1' fid='1'/>
Also
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'>
is a valid pci address on s390 and is would be expanded during 
definition into e.g.
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x0' uid='1' fid='1'/>


> 
> be a better representation of the arrangement? We could leave it
> up to QEMU to assign addresses to the PCI devices in this case...
> But maybe they still matter for things such as migration? Or maybe
> I've just gotten it wrong altogether? :)
> 


-- 
Mit freundlichen Grüßen/Kind regards
    Boris Fiuczynski

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Martina Köderitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294