<div dir="ltr"><div>Hello Libvirt Developers,<br><br>I am looking for some feedback on a planned enhancement to Libvirt: the aim is<br>to store a portion of PCI(e) Vital Product Data (VPD) for each device along<br>with other PCI/PCIe device information already collected. Specifically, the SN<br>(Serial Number) read-only field of a VPD data structure of a device is of<br>interest which is described in PCI/PCIe specs (PCI local bus 2.1+ and PCIe<br>4.0+).<br><br>The context for this is the cross-project work in OpenStack (Nova, Neutron),<br>OVS and OVN to support for off-path SmartNIC DPUs ([1], [2], [3], [4]). The<br>Nova specification [1] provides an overview of the relevant hardware and the<br>use-case for board serial numbers, however, VPD is the standard capability in<br>the PCI/PCIe specifications not tied to the use-case in particular so the<br>suggestion from the Nova core team was to aim at introducing means of<br>collecting this information via Libvirt. It can then be retrieved by the<br>respective virt driver in Nova via Libvirt without having to introduce this<br>code into Nova itself.<br><br>Quoting the PCI(e) specs:<br><br>* "Vital Product Data (VPD) is the information that uniquely defines items such<br>as the hardware, software, and microcode elements of a system.";<br>* "Vital Product Data is made up of Small and Large Resource Data Types.";<br>* "Large resource type VPD-R Tag: This tag contains the read only VPD keywords<br>for an add-in card."<br>* SN read-only field: "The characters are alphanumeric and represent the unique<br>add-in card Serial Number."<br><br>The VPD capability is optional per the specification so it may or may not<br>appear for PCI(e) endpoints. The devices of interest (SmartNIC DPUs), however,<br>generally have it exposed.<br><br>The PCI/PCIe specs define a binary format for VPD and a sysfs entry exposing<br>a binary blob in that format has been available since kernel v2.6.26 [5]. The<br>relevant sections of specs are:<br><br>* "6.4. Vital Product Data" in the PCI Local Bus specification;<br>* "6.28 Vital Product Data (VPD)" in the PCIe 4.0 Base Specification.<br><br>Note that the serial number stored in VPD is not identical to the information<br>stored in the Device Serial Number (DSN) capability also present in the specs<br>as it may identify a component on a board which presents a multi-function<br>device but the board itself may have multiple components ([9] also makes a<br>distinction between a board serial and a device serial).<br><br>As a reference, there is some code to parse and print the VPD in lspci [6] and<br>there is a prototype along those lines in Python [7], a polished version of<br>which I plan to use in Nova until the relevant functionality appears in<br>Libvirt.<br><br>Likewise, the devlink kernel infrastructure, which is already used in Libvirt <br>to query additional device capabilities [8] (e.g. the presence of an eswitch <br>and its switchdev mode) has a devlink-info API [9] that exposes a way to query <br>a board serial number if a device driver exposes it (in turn, by querying <br>controller firmware or via PCIe VPD). This allows doing that in a <br>bus-independent manner (e.g. it would work for PCIe, platform devices or other <br>I/O interconnects) but in the context of devices that implement devlink API <br>only (which are not necessarily network devices [10] but most of them currently<br>are). <br> <br>I would like to suggest the following to be done in Libvirt:<br><br>1) adding the code for extracting a serial number from VPD for PCI/PCIe devices<br> in general and storing it for exposure via the Libvirt API;<br> More specifically, I propose adding a nested capability called "vpd" under<br> VIR_NODE_DEV_CAP_PCI_DEV:<br> <capability type='pci'><br> <capability type='vpd'><br> <serial>UNIQUESERIAL</serial><br> <br> </capability><br> <br> </capability><br>2) (optional) implementing functionality to obtain a board serial number via<br> devlink-info for PCIe devices if they do not expose a VPD capability<br> but the device driver can retrieve it via firmware. The board serial number<br> can be stored in the same element as suggested above.</div><div><br>Not all devices expose the devlink API and even fewer do expose board serial<br>via devlink-info:<br><br>* devlink was added in 4.10 [11];<br>* devlink-info was introduced in 5.1 [12];<br>* querying for board.serial_number was added in kernel 5.9 [13] and iproute2<br> 5.9.0 [14];<br>* Besides the generic devlink infrastructure support above, device drivers<br> also need to support exposing this field.<br><br>Therefore, implementing two approaches (sysfs VPD, devlink) is preferable for better compatibility.<br><br>I would appreciate any feedback on whether this potential addition makes sense.<br>If so, I can look into implementing this.<br><br>[1] <a href="https://review.opendev.org/c/openstack/nova-specs/+/787458">https://review.opendev.org/c/openstack/nova-specs/+/787458</a><br>[2] <a href="https://review.opendev.org/c/openstack/neutron-specs/+/788821">https://review.opendev.org/c/openstack/neutron-specs/+/788821</a><br>[3] <a href="https://patchwork.ozlabs.org/project/openvswitch/patch/20210323145032.453120-1-frode.nordahl@canonical.com/">https://patchwork.ozlabs.org/project/openvswitch/patch/20210323145032.453120-1-frode.nordahl@canonical.com/</a><br>[4] <a href="https://patchwork.ozlabs.org/project/ovn/patch/20210509140305.1910796-1-frode.nordahl@canonical.com/">https://patchwork.ozlabs.org/project/ovn/patch/20210509140305.1910796-1-frode.nordahl@canonical.com/</a><br>[5] <a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=94e6108803469a37ee1e3c92dafdd1d59298602f">https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=94e6108803469a37ee1e3c92dafdd1d59298602f</a><br>[6] <a href="https://github.com/pciutils/pciutils/blob/v3.7.0/ls-vpd.c#L95-L216">https://github.com/pciutils/pciutils/blob/v3.7.0/ls-vpd.c#L95-L216</a><br>[7] <a href="https://gist.github.com/dshcherb/40e982989599a757e5b1e25999501019">https://gist.github.com/dshcherb/40e982989599a757e5b1e25999501019</a><br>[8] <a href="https://github.com/libvirt/libvirt/blob/v7.3.0/src/util/virnetdev.c#L3167-L3245">https://github.com/libvirt/libvirt/blob/v7.3.0/src/util/virnetdev.c#L3167-L3245</a><br>[9] <a href="https://www.kernel.org/doc/html/latest/networking/devlink/devlink-info.html">https://www.kernel.org/doc/html/latest/networking/devlink/devlink-info.html</a><br>[10] <a href="https://www.kernel.org/doc/html/latest/networking/devlink/index.html">https://www.kernel.org/doc/html/latest/networking/devlink/index.html</a><br>[11] <a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bfcd3a46617209454cfc0947ab093e37fd1e84ef">https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bfcd3a46617209454cfc0947ab093e37fd1e84ef</a><br>[12] <a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f9cf22882c606f3ffe06f620bb6d03b9eff18d3d">https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f9cf22882c606f3ffe06f620bb6d03b9eff18d3d</a><br>[13] <a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b5872cd0e823e4cb50b3a75cd9522167eeb676a2">https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b5872cd0e823e4cb50b3a75cd9522167eeb676a2</a><br>[14] <a href="https://git.kernel.org/pub/scm/network/iproute2/iproute2.git/commit/?id=7332b188a6f8e5c5d67c9a03d1591a813a4c908c">https://git.kernel.org/pub/scm/network/iproute2/iproute2.git/commit/?id=7332b188a6f8e5c5d67c9a03d1591a813a4c908c</a><br></div><div><br></div><div><div dir="ltr" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Best Regards,<div>Dmitrii Shcherbakov</div><div><span style="color:rgb(136,136,136);font-size:12.8px">LP: ~dmitriis</span><br></div></div></div></div></div></div></div></div></div>