<div dir="ltr"><div class="gmail_chip gmail_drive_chip" style="width:396px;height:18px;max-height:18px;background-color:rgb(245,245,245);padding:5px;font-family:arial;font-weight:bold;border:1px solid rgb(221,221,221);line-height:1"><a href="https://drive.google.com/file/d/1AL8yXbcCa6LAJYcykE2OF31GkRUTP2um/view?usp=drive_web" target="_blank" style="display:inline-block;max-width:366px;overflow:hidden;text-overflow:ellipsis;white-space:nowrap;text-decoration:none;padding:1px 0;border:none"><img style="vertical-align: bottom; border: none;" src="https://ssl.gstatic.com/docs/doclist/images/icon_10_generic_list.png"> <span dir="ltr" style="color:rgb(17,85,204);text-decoration:none;vertical-align:bottom">CentOS-7.3-x86_64-SPDK-NVMe.img.tar.bz2</span></a><img src="//ssl.gstatic.com/ui/v1/icons/common/x_8px.png" style="opacity: 0.55; cursor: pointer; float: right; position: relative; top: -1px; display: none;"></div><div>Theoretically SPDK should be able to support multiple storage backends, including NVMe drive, Linux AIO and malloc ramdisk.<div><br><div>But after trying the latter two methods, I found the VM just couldn't boot up. Even if I had setup a virtual CDROM drive and reinstalled the guest OS inside. Don't know the reason yet, will consult SPDK community later.</div></div><div><br></div><div>So it looks like you do need to have a NVMe drive...</div><div><br></div><div>Below are the instructions of how to setup the test environment.</div><div><br></div><div>--------</div><div><br></div><div><div><font face="monospace, monospace">Host Environment: CentOS 7</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">1. Compile SPDK</font></div><div><font face="monospace, monospace">$ git clone <a href="https://github.com/spdk/spdk">https://github.com/spdk/spdk</a></font></div><div><font face="monospace, monospace">$ cd spdk</font></div><div><font face="monospace, monospace">$ git submodule update --init (clone submodule DPDK)</font></div><div><font face="monospace, monospace">$ sudo ./scripts/pkgdep.sh<span style="white-space:pre"> </span>(install dependencies, ignore unnecessary packages)</font></div><div><font face="monospace, monospace">$ ./configure</font></div><div><font face="monospace, monospace">$ make -j8</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">2. Install QEMU 2.12.0</font></div><div><font face="monospace, monospace">$ wget <a href="https://download.qemu.org/qemu-2.12.0.tar.xz">https://download.qemu.org/qemu-2.12.0.tar.xz</a></font></div><div><font face="monospace, monospace">$ tar xvf qemu-2.12.0.tar.xz && cd qemu-2.12.0</font></div><div><font face="monospace, monospace">$ ./configure --target-list=x86_64-softmmu</font></div><div><font face="monospace, monospace">$ make -j8 && sudo make install</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">3. Setup Hugepages</font></div><div><font face="monospace, monospace">Edit /etc/default/grub, add</font></div><div><font face="monospace, monospace"><span style="white-space:pre"> </span>intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=16 (reserve 16GB hugepages for testing)</font></div><div><font face="monospace, monospace">$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">4. Auto load vfio module</font></div><div><font face="monospace, monospace">Edit /etc/modules-load.d/vfio.conf, add</font></div><div><font face="monospace, monospace"><span style="white-space:pre"> </span>vfio_pci</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">5. Reboot</font></div><div><font face="monospace, monospace">$ sudo reboot</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">6. Setup SPDK</font></div><div><font face="monospace, monospace">$ cd spdk && sudo su</font></div><div><font face="monospace, monospace"># ulimit -l unlimited</font></div><div><font face="monospace, monospace"># HUGEMEM=16384 ./scripts/setup.sh</font></div><div><font face="monospace, monospace"># ./app/vhost/vhost -S /var/tmp -s 1024 -m 0x1 & <span style="white-space:pre"> </span>(start vhost daemon)</font></div><div><font face="monospace, monospace"># ./scripts/rpc.py construct_nvme_bdev -b nvme0 -t pcie -a 0000:03:00.0 (NVMe drive's pci address)</font></div><div><font face="monospace, monospace"># ./scripts/rpc.py construct_lvol_store nvme0n1 lvs.nvme0n1</font></div><div><font face="monospace, monospace"># ./scripts/rpc.py construct_lvol_bdev -l lvs.nvme0n1 lvol.0 20480<span style="white-space:pre"> </span>(create a 20GB logical volume)</font></div><div><font face="monospace, monospace"># ./scripts/rpc.py construct_vhost_scsi_controller vhost.0</font></div><div><font face="monospace, monospace"># ./scripts/rpc.py add_vhost_scsi_lun vhost.0 0 lvs.nvme0n1/lvol.0<span style="white-space:pre"> </span>(should then have a /var/tmp/vhost.0 socket)</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">7. Copy rootfs image into logical volume</font></div><div><font face="monospace, monospace"># ./scripts/rpc.py start_nbd_disk lvs.nvme0n1/lvol.0 /dev/nbd0</font></div><div><font face="monospace, monospace"># dd if=CentOS-7.3-x86_64-SPDK-NVMe.img of=/dev/nbd0 oflag=direct bs=1M<span style="white-space:pre"> </span>(Copy the 20GB image into the logical volume, see attached file)</font></div><div><font face="monospace, monospace"># ./scripts/rpc.py stop_nbd_disk /dev/nbd0</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">8. Download UEFI bootloader (<a href="https://github.com/tianocore/tianocore.github.io/wiki/OVMF">https://github.com/tianocore/tianocore.github.io/wiki/OVMF</a>)</font></div><div><font face="monospace, monospace"># wget <a href="https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-x64-0-20180612.196.gb420d98502.noarch.rpm">https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-x64-0-20180612.196.gb420d98502.noarch.rpm</a></font></div><div><font face="monospace, monospace"><span style="white-space:pre"> </span>rpm install ...</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">9. Start QEMU</font></div><div><font face="monospace, monospace"># /usr/local/bin/qemu-system-x86_64 \</font></div><div><font face="monospace, monospace">-enable-kvm -cpu host -smp 4 \</font></div><div><font face="monospace, monospace">-m 4G -object memory-backend-file,id=mem0,size=4G,mem-path=/dev/hugepages,share=on -numa node,memdev=mem0 \</font></div><div><font face="monospace, monospace">-drive file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,format=raw,if=pflash \</font></div><div><font face="monospace, monospace">-chardev socket,id=spdk_vhost_scsi0,path=/var/tmp/vhost.0 \</font></div><div><font face="monospace, monospace">-device vhost-user-scsi-pci,chardev=spdk_vhost_scsi0,num_queues=4,bootindex=0 \</font></div><div><font face="monospace, monospace">...</font></div><div><br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">Richard W.M. Jones <<a href="mailto:rjones@redhat.com">rjones@redhat.com</a>> 于2018年7月18日周三下午9:44写道：<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Wed, Jul 18, 2018 at 12:10:30PM +0800, Bob Chen wrote:<br> > Considering that the technic of SPDK + QEMU is making progress toward<br> > maturity.<br> > <br> > Personally I'd like to do the integration work. Not sure somebody would<br> > mind to give me some clue on that? Because I'm not familiar with libguestfs<br> > code structures.<br> <br> If qemu supports SPDK then we should get it "for free". At most we<br> might need to add support for the qemu command line like we do for<br> other non-file stuff (eg. NBD, iSCSI).<br> <br> What does a qemu command line accessing SPDK look like?<br> <br> What sort of resources do we need to set up an SPDK test environment?<br> (For example, are NVMe drives absolutely required?)<br> <br> Rich.<br> <br> -- <br> Richard Jones, Virtualization Group, Red Hat <a href="http://people.redhat.com/~rjones" rel="noreferrer" target="_blank">http://people.redhat.com/~rjones</a><br> Read my programming and virtualization blog: <a href="http://rwmj.wordpress.com" rel="noreferrer" target="_blank">http://rwmj.wordpress.com</a><br> virt-builder quickly builds VMs from scratch<br> <a href="http://libguestfs.org/virt-builder.1.html" rel="noreferrer" target="_blank">http://libguestfs.org/virt-builder.1.html</a><br> </blockquote></div>