<div dir="ltr">Hi,<div><br></div><div>I've done some TLS testing with this patch and results look good. The following test statically adds a VxHS disk to a guest in the TLS mode. Boots up the guest and makes sure that we can do read/writes to the VxHS disk from within the guest with TLS enabled.</div><div><br></div><div>(1) Create a backing store file /tmp/test_vxhs_disk_1 and start the VxHS test server "qnio_server" with TLS enabled.</div><div><br></div><div>(2) Client side TLS env was setup as follows -</div><div><div><br></div><div>[root@audi ~] 2017-09-18 13:56:13# grep -i vxhs /etc/libvirt/qemu.conf | grep -v "^#"</div><div>vxhs_tls = 1</div><div>vxhs_tls_x509_cert_dir = "/etc/pki/libvirt-vxhs"</div><div><br></div><div>[root@audi ~] 2017-09-18 13:56:22# ll /etc/pki/libvirt-vxhs</div><div>total 20</div><div>-r--r--r--. 1 root root 4062 Sep 17 23:15 ca-cert.pem</div><div>-rw-r--r--. 1 root root 1866 Sep 17 22:52 client-cert.pem</div><div>-r--------. 1 root root 1679 Sep 17 22:52 client-key.pem</div><div>[root@audi ~] 2017-09-18 13:56:35#</div></div><div><br></div><div>(3) virsh edit and add a new VxHS device with tls='yes'<br></div><div><br></div><div>The XML added to existing domain -</div><div><br></div><div><div>    <disk type='network' device='disk'></div><div>      <driver name='qemu' type='raw' cache='none'/></div><div>      <source protocol='vxhs' name='/tmp/test_vxhs_disk_1' tls='yes'></div><div>        <host name='127.0.0.1' port='9999'/></div><div>      </source></div><div>      <backingStore/></div><div>      <target dev='vdc' bus='virtio'/></div><div>      <serial>eb90327c-8302-4725-9e1b-4e85ed4dc251</serial></div><div>      <alias name='virtio-disk2'/></div><div>      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/></div><div>    </disk></div></div><div><br></div><div>(4) Start the domain and check if qemu command is correct</div><div><br></div><div><div>[root@audi ~] 2017-09-18 13:29:01# virsh start myfc24</div><div>Domain myfc24 started</div></div><div><br></div><div><div>[root@audi ~] 2017-09-18 13:29:20# ps -ef | grep qemu</div><div><br></div><div>root      9578     1 99 13:29 ?        00:00:20 /usr/bin/qemu-system-x86_64 -machine accel=kvm -name guest=myfc24,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-myfc24/master-key.aes -machine pc-i440fx-2.6,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu Opteron_G3 -m 1024 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 70454565-8185-4506-b50f-d2cf55d83796 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-myfc24/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/libvirt/images/myfc24_rootdisk.qcow2,format=qcow2,if=none,id=drive-ide0-0-0 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive if=none,id=drive-ide0-0-1,readonly=on -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -object tls-creds-x509,id=objvirtio-disk2_tls0,dir=/etc/pki/libvirt-vxhs,endpoint=client,verify-peer=yes -drive file.driver=vxhs,file.tls-creds=objvirtio-disk2_tls0,file.vdisk-id=/tmp/test_vxhs_disk_1,file.server.type=tcp,file.server.host=127.0.0.1,file.server.port=9999,format=raw,if=none,id=drive-virtio-disk2,serial=eb90327c-8302-4725-9e1b-4e85ed4dc251,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xa,drive=drive-virtio-disk2,id=virtio-disk2 -netdev tap,fd=27,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:e4:9e:30,bus=pci.0,addr=0x3 -netdev tap,fd=29,id=hostnet1,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:b1:43:c4,bus=pci.0,addr=0x8 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on</div></div><div><br></div><div>(5) Log in to the guest domain and make sure we see this VxHS disk</div><div><br></div><div><div>[root@camshaft ~] 2017-09-18 13:32:22# fdisk -l</div><div>...</div><div>Disk /dev/vda: 1 MiB, 1048576 bytes, 2048 sectors<br></div><div>Units: sectors of 1 * 512 = 512 bytes</div><div>Sector size (logical/physical): 512 bytes / 512 bytes</div><div>I/O size (minimum/optimal): 512 bytes / 512 bytes</div><div><br></div><div>Disk /dev/mapper/fedora-root: 45.4 GiB, 48704258048 bytes, 95125504 sectors<br></div></div><div>...</div><div><br></div><div>(6) Create a disk label and partition. Do mkfs and mount the FS. Copy some files to the disk and check general read/write operations.</div><div><br></div><div>[root@camshaft ~] 2017-09-18 13:32:35# fdisk /dev/vda<br></div><div><div>....</div><div>Created a new partition 1 of type 'Linux' and of size 1023.5 KiB.</div><div><br></div><div>Command (m for help): p</div><div>Disk /dev/vda: 1 MiB, 1048576 bytes, 2048 sectors</div><div>Units: sectors of 1 * 512 = 512 bytes</div><div>Sector size (logical/physical): 512 bytes / 512 bytes</div><div>I/O size (minimum/optimal): 512 bytes / 512 bytes</div><div>Disklabel type: dos</div><div>Disk identifier: 0xcfd93e87</div><div><br></div><div>Device     Boot Start   End Sectors    Size Id Type</div><div>/dev/vda1           1  2047    2047 1023.5K 83 Linux</div><div><br></div><div>Command (m for help): w</div><div>The partition table has been altered.</div><div>Calling ioctl() to re-read partition table.</div><div>Syncing disks.</div><div><br></div><div>[root@camshaft ~]</div></div><div><br></div><div><div>[root@camshaft ~] 2017-09-18 13:34:29# mkfs.ext3 /dev/vda1</div><div>mke2fs 1.42.13 (17-May-2015)</div><div><br></div><div>Filesystem too small for a journal</div><div>Creating filesystem with 1020 1k blocks and 128 inodes</div><div><br></div><div>Allocating group tables: done</div><div>Writing inode tables: done</div><div>Writing superblocks and filesystem accounting information: done</div><div><br></div><div>[root@camshaft ~] 2017-09-18 13:34:46# mount /dev/vda1 /mnt</div></div><div><br></div><div><div>[root@camshaft ~] 2017-09-18 13:34:56# cp /boot/System.map-4.* /mnt</div><div>cp: error writing '/mnt/System.map-4.5.5-300.fc24.x86_64': No space left on device</div><div>cp: error writing '/mnt/System.map-4.8.8-200.fc24.x86_64': No space left on device</div><div>[root@camshaft ~] 2017-09-18 13:35:08# df -h</div><div>Filesystem               Size  Used Avail Use% Mounted on</div><div>devtmpfs                 485M     0  485M   0% /dev</div><div>tmpfs                    497M  472K  496M   1% /dev/shm</div><div>tmpfs                    497M  1.3M  495M   1% /run</div><div>tmpfs                    497M     0  497M   0% /sys/fs/cgroup</div><div>/dev/mapper/fedora-root   45G  5.1G   38G  12% /</div><div>tmpfs                    497M   84K  497M   1% /tmp</div><div>/dev/mapper/fedora-home   22G  1.7G   19G   9% /home</div><div>/dev/sda1                477M  140M  308M  32% /boot</div><div>tmpfs                    100M   28K  100M   1% /run/user/42</div><div>tmpfs                    100M   20K  100M   1% /run/user/1000</div><div>/dev/vda1                999K  999K     0 100% /mnt</div><div>[root@camshaft ~] 2017-09-18 13:35:13#</div><div><br></div><div>[root@camshaft ~] 2017-09-18 13:37:07# dd if=/mnt/System.map-4.5.5-300.fc24.x86_64 of=/dev/null<br></div><div>1952+0 records in</div><div>1952+0 records out</div><div>999424 bytes (999 kB, 976 KiB) copied, 0.001174 s, 851 MB/s</div><div>[root@camshaft ~] 2017-09-18 13:37:14#</div><div><br></div></div><div><br></div><div><br></div><div>Regards,</div><div>Ashish</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 1, 2017 at 10:09 AM, John Ferlan <span dir="ltr"><<a href="mailto:jferlan@redhat.com" target="_blank">jferlan@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">From: Ashish Mittal <<a href="mailto:Ashish.Mittal@veritas.com">Ashish.Mittal@veritas.com</a>><br>
<br>
Alter qemu command line generation in order to possibly add TLS for<br>
a suitably configured domain.<br>
<br>
Sample TLS args generated by libvirt -<br>
<br>
    -object tls-creds-x509,id=objvirtio-<wbr>disk0_tls0,dir=/etc/pki/qemu,\<br>
    endpoint=client,verify-peer=<wbr>yes \<br>
    -drive file.driver=vxhs,file.tls-<wbr>creds=objvirtio-disk0_tls0,\<br>
    file.vdisk-id=eb90327c-8302-<wbr>4725-9e1b-4e85ed4dc251,\<br>
    file.server.0.type=tcp,file.<wbr>server.0.host=192.168.0.1,\<br>
    file.server.0.port=9999,<wbr>format=raw,if=none,\<br>
    id=drive-virtio-disk0,cache=<wbr>none \<br>
    -device virtio-blk-pci,bus=pci.0,addr=<wbr>0x4,drive=drive-virtio-disk0,\<br>
    id=virtio-disk0<br>
<br>
Update the qemuxml2argvtest with a couple of examples. One for a<br>
simple case and the other a bit more complex where multiple VxHS disks<br>
are added where at least one uses a VxHS that doesn't require TLS<br>
credentials and thus sets the domain disk source attribute "tls = 'no'".<br>
<br>
Update the hotplug to be able to handle processing the tlsAlias whether<br>
it's to add the TLS object when hotplugging a disk or to remove the TLS<br>
object when hot unplugging a disk.  The hot plug/unplug code is largely<br>
generic, but the addition code does make the VXHS specific checks only<br>
because it needs to grab the correct config directory and generate the<br>
object as the command line would do.<br>
<br>
Signed-off-by: Ashish Mittal <<a href="mailto:Ashish.Mittal@veritas.com">Ashish.Mittal@veritas.com</a>><br>
Signed-off-by: John Ferlan <<a href="mailto:jferlan@redhat.com">jferlan@redhat.com</a>><br>
---<br>
 src/qemu/qemu_block.c                              |  8 +++<br>
 src/qemu/qemu_command.c                            | 29 +++++++++<br>
 src/qemu/qemu_hotplug.c                            | 73 ++++++++++++++++++++++<br>
 ...-disk-drive-network-<wbr>tlsx509-multidisk-vxhs.args | 43 +++++++++++++<br>
 ...v-disk-drive-network-<wbr>tlsx509-multidisk-vxhs.xml | 50 +++++++++++++++<br>
 ...muxml2argv-disk-drive-<wbr>network-tlsx509-vxhs.args | 30 +++++++++<br>
 tests/qemuxml2argvtest.c                           |  7 +++<br>
 7 files changed, 240 insertions(+)<br>
 create mode 100644 tests/qemuxml2argvdata/<wbr>qemuxml2argv-disk-drive-<wbr>network-tlsx509-multidisk-<wbr>vxhs.args<br>
 create mode 100644 tests/qemuxml2argvdata/<wbr>qemuxml2argv-disk-drive-<wbr>network-tlsx509-multidisk-<wbr>vxhs.xml<br>
 create mode 100644 tests/qemuxml2argvdata/<wbr>qemuxml2argv-disk-drive-<wbr>network-tlsx509-vxhs.args<br>
<br>
diff --git a/src/qemu/qemu_block.c b/src/qemu/qemu_block.c<br>
index f5269fb..be4e8fa 100644<br>
--- a/src/qemu/qemu_block.c<br>
+++ b/src/qemu/qemu_block.c<br>
@@ -495,16 +495,24 @@ qemuBlockStorageSourceGetVxHSP<wbr>rops(virStorageSourcePtr src)<br>
         return NULL;<br>
     }<br>
<br>
+    if (src->haveTLS == VIR_TRISTATE_BOOL_YES && !src->tlsAlias) {<br>
+        virReportError(VIR_ERR_<wbr>INVALID_ARG, "%s",<br>
+                       _("VxHS disk does not have TLS alias set"));<br>
+        return NULL;<br>
+    }<br>
+<br>
     if (!(server = qemuBlockStorageSourceBuildHos<wbr>tsJSONSocketAddress(src, true)))<br>
         return NULL;<br>
<br>
     /* VxHS disk specification example:<br>
      * { driver:"vxhs",<br>
+     *   [tls-creds:"objvirtio-disk0_<wbr>tls0",]<br>
      *   vdisk-id:"eb90327c-8302-4725-<wbr>4e85ed4dc251",<br>
      *   server:[{type:"tcp", host:"1.2.3.4", port:9999}]}<br>
      */<br>
     if (virJSONValueObjectCreate(&<wbr>ret,<br>
                                  "s:driver", protocol,<br>
+                                 "S:tls-creds", src->tlsAlias,<br>
                                  "s:vdisk-id", src->path,<br>
                                  "a:server", server, NULL) < 0)<br>
         virJSONValueFree(server);<br>
diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c<br>
index b9e2ab3..d6b04a3 100644<br>
--- a/src/qemu/qemu_command.c<br>
+++ b/src/qemu/qemu_command.c<br>
@@ -791,6 +791,32 @@ qemuBuildTLSx509CommandLine(<wbr>virCommandPtr cmd,<br>
 }<br>
<br>
<br>
+/* qemuBuildDiskTLSx509CommandLin<wbr>e:<br>
+ *<br>
+ * Add TLS object if the disk uses a secure communication channel<br>
+ *<br>
+ * Returns 0 on success, -1 w/ error on some sort of failure.<br>
+ */<br>
+static int<br>
+<wbr>qemuBuildDiskTLSx509CommandLin<wbr>e(virCommandPtr cmd,<br>
+                                virQEMUDriverConfigPtr cfg,<br>
+                                virDomainDiskDefPtr disk,<br>
+                                virQEMUCapsPtr qemuCaps)<br>
+{<br>
+    virStorageSourcePtr src = disk->src;<br>
+<br>
+    /* other protocols may be added later */<br>
+    if (src->protocol == VIR_STORAGE_NET_PROTOCOL_VXHS &&<br>
+        disk->src->haveTLS == VIR_TRISTATE_BOOL_YES) {<br>
+        return qemuBuildTLSx509CommandLine(<wbr>cmd, cfg->vxhsTLSx509certdir,<br>
+                                          false, true, false,<br>
+                                          disk->info.alias, qemuCaps);<br>
+    }<br>
+<br>
+    return 0;<br>
+}<br>
+<br>
+<br>
 static char *<br>
 qemuBuildNetworkDriveURI(<wbr>virStorageSourcePtr src,<br>
                          qemuDomainSecretInfoPtr secinfo)<br>
@@ -2218,6 +2244,9 @@ qemuBuildDiskDriveCommandLine(<wbr>virCommandPtr cmd,<br>
         if (<wbr>qemuBuildDiskSecinfoCommandLin<wbr>e(cmd, encinfo) < 0)<br>
             return -1;<br>
<br>
+        if (<wbr>qemuBuildDiskTLSx509CommandLin<wbr>e(cmd, cfg, disk, qemuCaps) < 0)<br>
+            return -1;<br>
+<br>
         virCommandAddArg(cmd, "-drive");<br>
<br>
         if (!(optstr = qemuBuildDriveStr(disk, cfg, driveBoot, qemuCaps)))<br>
diff --git a/src/qemu/qemu_hotplug.c b/src/qemu/qemu_hotplug.c<br>
index 9611df5..4c1074d 100644<br>
--- a/src/qemu/qemu_hotplug.c<br>
+++ b/src/qemu/qemu_hotplug.c<br>
@@ -152,6 +152,55 @@ qemuDomainPrepareDisk(<wbr>virQEMUDriverPtr driver,<br>
<br>
<br>
 static int<br>
+qemuDomainAddDiskTLSObject(<wbr>virQEMUDriverPtr driver,<br>
+                           virDomainObjPtr vm,<br>
+                           virDomainDiskDefPtr disk,<br>
+                           char **tlsAlias)<br>
+{<br>
+    int ret = -1;<br>
+    virQEMUDriverConfigPtr cfg = virQEMUDriverGetConfig(driver)<wbr>;<br>
+    qemuDomainObjPrivatePtr priv = vm->privateData;<br>
+    virStorageSourcePtr src = disk->src;<br>
+    virJSONValuePtr tlsProps = NULL;<br>
+<br>
+    /* NB: This may alter haveTLS based on cfg */<br>
+    qemuDomainPrepareDiskSourceTLS<wbr>(src, disk->info.alias, cfg);<br>
+<br>
+    if (src->haveTLS != VIR_TRISTATE_BOOL_YES) {<br>
+        ret = 0;<br>
+        goto cleanup;<br>
+    }<br>
+<br>
+    /* Initial implementation doesn't require/use a secret to decrypt<br>
+     * a server certificate, so there's no need to manage a tlsSecAlias<br>
+     * and tlsSecProps. See qemuDomainAddChardevTLSObjects for the<br>
+     * methodology required to add a secret object. */<br>
+<br>
+    /* For a VxHS environment, create a TLS object for the client to<br>
+     * connect to the VxHS server. */<br>
+    if (src->type == VIR_STORAGE_TYPE_NETWORK &&<br>
+        src->protocol == VIR_STORAGE_NET_PROTOCOL_VXHS &&<br>
+        qemuDomainGetTLSObjects(priv-><wbr>qemuCaps, NULL,<br>
+                                cfg->vxhsTLSx509certdir, false, true,<br>
+                                disk->info.alias, &tlsProps, tlsAlias,<br>
+                                NULL, NULL) < 0)<br>
+        goto cleanup;<br>
+<br>
+    if (qemuDomainAddTLSObjects(<wbr>driver, vm, QEMU_ASYNC_JOB_NONE,<br>
+                                NULL, NULL, *tlsAlias, &tlsProps) < 0)<br>
+        goto cleanup;<br>
+<br>
+    ret = 0;<br>
+<br>
+ cleanup:<br>
+    virJSONValueFree(tlsProps);<br>
+    virObjectUnref(cfg);<br>
+<br>
+    return ret;<br>
+}<br>
+<br>
+<br>
+static int<br>
 qemuHotplugWaitForTrayEject(<wbr>virQEMUDriverPtr driver,<br>
                             virDomainObjPtr vm,<br>
                             virDomainDiskDefPtr disk,<br>
@@ -315,6 +364,7 @@ qemuDomainAttachVirtioDiskDevi<wbr>ce(virConnectPtr conn,<br>
     char *devstr = NULL;<br>
     char *drivestr = NULL;<br>
     char *drivealias = NULL;<br>
+    char *tlsAlias = NULL;<br>
     bool releaseaddr = false;<br>
     bool driveAdded = false;<br>
     bool secobjAdded = false;<br>
@@ -372,6 +422,9 @@ qemuDomainAttachVirtioDiskDevi<wbr>ce(virConnectPtr conn,<br>
     if (encinfo && qemuBuildSecretInfoProps(<wbr>encinfo, &encobjProps) < 0)<br>
         goto error;<br>
<br>
+    if (qemuDomainAddDiskTLSObject(<wbr>driver, vm, disk, &tlsAlias) < 0)<br>
+        goto error;<br>
+<br>
     if (!(drivestr = qemuBuildDriveStr(disk, cfg, false, priv->qemuCaps)))<br>
         goto error;<br>
<br>
@@ -422,6 +475,7 @@ qemuDomainAttachVirtioDiskDevi<wbr>ce(virConnectPtr conn,<br>
     ret = 0;<br>
<br>
  cleanup:<br>
+    VIR_FREE(tlsAlias);<br>
     virJSONValueFree(secobjProps);<br>
     virJSONValueFree(encobjProps);<br>
     qemuDomainSecretDiskDestroy(<wbr>disk);<br>
@@ -453,6 +507,8 @@ qemuDomainAttachVirtioDiskDevi<wbr>ce(virConnectPtr conn,<br>
     virDomainAuditDisk(vm, NULL, disk->src, "attach", false);<br>
<br>
  error:<br>
+    qemuDomainDelTLSObjects(<wbr>driver, vm, QEMU_ASYNC_JOB_NONE, NULL, tlsAlias);<br>
+<br>
     if (releaseaddr)<br>
         qemuDomainReleaseDeviceAddress<wbr>(vm, &disk->info, src);<br>
<br>
@@ -611,6 +667,7 @@ qemuDomainAttachSCSIDisk(<wbr>virConnectPtr conn,<br>
     virErrorPtr orig_err;<br>
     char *drivestr = NULL;<br>
     char *devstr = NULL;<br>
+    char *tlsAlias = NULL;<br>
     bool driveAdded = false;<br>
     bool encobjAdded = false;<br>
     bool secobjAdded = false;<br>
@@ -667,6 +724,9 @@ qemuDomainAttachSCSIDisk(<wbr>virConnectPtr conn,<br>
     if (!(devstr = qemuBuildDriveDevStr(vm->def, disk, 0, priv->qemuCaps)))<br>
         goto error;<br>
<br>
+    if (qemuDomainAddDiskTLSObject(<wbr>driver, vm, disk, &tlsAlias) < 0)<br>
+        goto error;<br>
+<br>
     if (!(drivestr = qemuBuildDriveStr(disk, cfg, false, priv->qemuCaps)))<br>
         goto error;<br>
<br>
@@ -712,6 +772,7 @@ qemuDomainAttachSCSIDisk(<wbr>virConnectPtr conn,<br>
     ret = 0;<br>
<br>
  cleanup:<br>
+    VIR_FREE(tlsAlias);<br>
     virJSONValueFree(secobjProps);<br>
     virJSONValueFree(encobjProps);<br>
     qemuDomainSecretDiskDestroy(<wbr>disk);<br>
@@ -740,6 +801,8 @@ qemuDomainAttachSCSIDisk(<wbr>virConnectPtr conn,<br>
     virDomainAuditDisk(vm, NULL, disk->src, "attach", false);<br>
<br>
  error:<br>
+    qemuDomainDelTLSObjects(<wbr>driver, vm, QEMU_ASYNC_JOB_NONE, NULL, tlsAlias);<br>
+<br>
     ignore_value(<wbr>qemuDomainPrepareDisk(driver, vm, disk, NULL, true));<br>
     goto cleanup;<br>
 }<br>
@@ -756,6 +819,7 @@ qemuDomainAttachUSBMassStorage<wbr>Device(virQEMUDriverPtr driver,<br>
     char *drivealias = NULL;<br>
     char *drivestr = NULL;<br>
     char *devstr = NULL;<br>
+    char *tlsAlias = NULL;<br>
     bool driveAdded = false;<br>
     virQEMUDriverConfigPtr cfg = virQEMUDriverGetConfig(driver)<wbr>;<br>
     const char *src = virDomainDiskGetSource(disk);<br>
@@ -780,6 +844,9 @@ qemuDomainAttachUSBMassStorage<wbr>Device(virQEMUDriverPtr driver,<br>
     if (qemuAssignDeviceDiskAlias(vm-<wbr>>def, disk, priv->qemuCaps) < 0)<br>
         goto error;<br>
<br>
+    if (qemuDomainAddDiskTLSObject(<wbr>driver, vm, disk, &tlsAlias) < 0)<br>
+        goto error;<br>
+<br>
     if (!(drivestr = qemuBuildDriveStr(disk, cfg, false, priv->qemuCaps)))<br>
         goto error;<br>
<br>
@@ -810,6 +877,7 @@ qemuDomainAttachUSBMassStorage<wbr>Device(virQEMUDriverPtr driver,<br>
     ret = 0;<br>
<br>
  cleanup:<br>
+    VIR_FREE(tlsAlias);<br>
     if (ret < 0 && releaseaddr)<br>
         virDomainUSBAddressRelease(<wbr>priv->usbaddrs, &disk->info);<br>
     VIR_FREE(devstr);<br>
@@ -833,6 +901,8 @@ qemuDomainAttachUSBMassStorage<wbr>Device(virQEMUDriverPtr driver,<br>
     virDomainAuditDisk(vm, NULL, disk->src, "attach", false);<br>
<br>
  error:<br>
+    qemuDomainDelTLSObjects(<wbr>driver, vm, QEMU_ASYNC_JOB_NONE, NULL, tlsAlias);<br>
+<br>
     ignore_value(<wbr>qemuDomainPrepareDisk(driver, vm, disk, NULL, true));<br>
     goto cleanup;<br>
 }<br>
@@ -3710,6 +3780,9 @@ qemuDomainRemoveDiskDevice(<wbr>virQEMUDriverPtr driver,<br>
         ignore_value(<wbr>qemuMonitorDelObject(priv-><wbr>mon, encAlias));<br>
     VIR_FREE(encAlias);<br>
<br>
+    if (disk->src->tlsAlias)<br>
+        ignore_value(<wbr>qemuMonitorDelObject(priv-><wbr>mon, disk->src->tlsAlias));<br>
+<br>
     if (qemuDomainObjExitMonitor(<wbr>driver, vm) < 0)<br>
         return -1;<br>
<br>
diff --git a/tests/qemuxml2argvdata/<wbr>qemuxml2argv-disk-drive-<wbr>network-tlsx509-multidisk-<wbr>vxhs.args b/tests/qemuxml2argvdata/<wbr>qemuxml2argv-disk-drive-<wbr>network-tlsx509-multidisk-<wbr>vxhs.args<br>
new file mode 100644<br>
index 0000000..dceae52<br>
--- /dev/null<br>
+++ b/tests/qemuxml2argvdata/<wbr>qemuxml2argv-disk-drive-<wbr>network-tlsx509-multidisk-<wbr>vxhs.args<br>
@@ -0,0 +1,43 @@<br>
+LC_ALL=C \<br>
+PATH=/bin \<br>
+HOME=/home/test \<br>
+USER=test \<br>
+LOGNAME=test \<br>
+QEMU_AUDIO_DRV=none \<br>
+/usr/bin/qemu-system-x86_64 \<br>
+-name QEMUGuest1 \<br>
+-S \<br>
+-M pc \<br>
+-cpu qemu32 \<br>
+-m 214 \<br>
+-smp 1,sockets=1,cores=1,threads=1 \<br>
+-uuid c7a5fdbd-edaf-9455-926a-<wbr>d65c16db1809 \<br>
+-nographic \<br>
+-nodefaults \<br>
+-chardev socket,id=charmonitor,path=/<wbr>tmp/lib/domain--1-QEMUGuest1/<wbr>monitor.sock,\<br>
+server,nowait \<br>
+-mon chardev=charmonitor,id=<wbr>monitor,mode=readline \<br>
+-no-acpi \<br>
+-boot c \<br>
+-usb \<br>
+-object tls-creds-x509,id=objvirtio-<wbr>disk0_tls0,dir=/etc/pki/qemu,\<br>
+endpoint=client,verify-peer=<wbr>yes \<br>
+-drive file.driver=vxhs,file.tls-<wbr>creds=objvirtio-disk0_tls0,\<br>
+file.vdisk-id=eb90327c-8302-<wbr>4725-9e1b-4e85ed4dc251,file.<wbr>server.0.type=tcp,\<br>
+file.server.0.host=192.168.0.<wbr>1,file.server.0.port=9999,<wbr>format=raw,if=none,\<br>
+id=drive-virtio-disk0,cache=<wbr>none \<br>
+-device virtio-blk-pci,bus=pci.0,addr=<wbr>0x4,drive=drive-virtio-disk0,\<br>
+id=virtio-disk0 \<br>
+-object tls-creds-x509,id=objvirtio-<wbr>disk1_tls0,dir=/etc/pki/qemu,\<br>
+endpoint=client,verify-peer=<wbr>yes \<br>
+-drive file.driver=vxhs,file.tls-<wbr>creds=objvirtio-disk1_tls0,\<br>
+file.vdisk-id=eb90327c-8302-<wbr>4725-9e1b-4e85ed4dc252,file.<wbr>server.0.type=tcp,\<br>
+file.server.0.host=192.168.0.<wbr>2,file.server.0.port=9999,<wbr>format=raw,if=none,\<br>
+id=drive-virtio-disk1,cache=<wbr>none \<br>
+-device virtio-blk-pci,bus=pci.0,addr=<wbr>0x5,drive=drive-virtio-disk1,\<br>
+id=virtio-disk1 \<br>
+-drive file.driver=vxhs,file.vdisk-<wbr>id=eb90327c-8302-4725-9e1b-<wbr>4e85ed4dc253,\<br>
+file.server.0.type=tcp,file.<wbr>server.0.host=192.168.0.3,<wbr>file.server.0.port=9999,\<br>
+format=raw,if=none,id=drive-<wbr>virtio-disk2,cache=none \<br>
+-device virtio-blk-pci,bus=pci.0,addr=<wbr>0x6,drive=drive-virtio-disk2,\<br>
+id=virtio-disk2<br>
diff --git a/tests/qemuxml2argvdata/<wbr>qemuxml2argv-disk-drive-<wbr>network-tlsx509-multidisk-<wbr>vxhs.xml b/tests/qemuxml2argvdata/<wbr>qemuxml2argv-disk-drive-<wbr>network-tlsx509-multidisk-<wbr>vxhs.xml<br>
new file mode 100644<br>
index 0000000..a66e81f<br>
--- /dev/null<br>
+++ b/tests/qemuxml2argvdata/<wbr>qemuxml2argv-disk-drive-<wbr>network-tlsx509-multidisk-<wbr>vxhs.xml<br>
@@ -0,0 +1,50 @@<br>
+<domain type='qemu'><br>
+  <name>QEMUGuest1</name><br>
+  <uuid>c7a5fdbd-edaf-9455-926a-<wbr>d65c16db1809</uuid><br>
+  <memory unit='KiB'>219136</memory><br>
+  <currentMemory unit='KiB'>219136</<wbr>currentMemory><br>
+  <vcpu placement='static'>1</vcpu><br>
+  <os><br>
+    <type arch='i686' machine='pc'>hvm</type><br>
+    <boot dev='hd'/><br>
+  </os><br>
+  <clock offset='utc'/><br>
+  <on_poweroff>destroy</on_<wbr>poweroff><br>
+  <on_reboot>restart</on_reboot><br>
+  <on_crash>destroy</on_crash><br>
+  <devices><br>
+    <emulator>/usr/bin/qemu-<wbr>system-x86_64</emulator><br>
+    <disk type='network' device='disk'><br>
+      <driver name='qemu' type='raw' cache='none'/><br>
+      <source protocol='vxhs' name='eb90327c-8302-4725-9e1b-<wbr>4e85ed4dc251'><br>
+        <host name='192.168.0.1' port='9999'/><br>
+      </source><br>
+      <target dev='vda' bus='virtio'/><br>
+      <serial>eb90327c-8302-4725-<wbr>9e1b-4e85ed4dc251</serial><br>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/><br>
+    </disk><br>
+    <disk type='network' device='disk'><br>
+      <driver name='qemu' type='raw' cache='none'/><br>
+      <source protocol='vxhs' name='eb90327c-8302-4725-9e1b-<wbr>4e85ed4dc252'><br>
+        <host name='192.168.0.2' port='9999'/><br>
+      </source><br>
+      <target dev='vdb' bus='virtio'/><br>
+      <serial>eb90327c-8302-4725-<wbr>9e1b-4e85ed4dc252</serial><br>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/><br>
+    </disk><br>
+    <disk type='network' device='disk'><br>
+      <driver name='qemu' type='raw' cache='none'/><br>
+      <source protocol='vxhs' name='eb90327c-8302-4725-9e1b-<wbr>4e85ed4dc253' tls='no'><br>
+        <host name='192.168.0.3' port='9999'/><br>
+      </source><br>
+      <target dev='vdc' bus='virtio'/><br>
+      <serial>eb90327c-8302-4725-<wbr>9e1b-4e85ed4dc252</serial><br>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/><br>
+    </disk><br>
+    <controller type='usb' index='0'/><br>
+    <controller type='pci' index='0' model='pci-root'/><br>
+    <input type='mouse' bus='ps2'/><br>
+    <input type='keyboard' bus='ps2'/><br>
+    <memballoon model='none'/><br>
+  </devices><br>
+</domain><br>
diff --git a/tests/qemuxml2argvdata/<wbr>qemuxml2argv-disk-drive-<wbr>network-tlsx509-vxhs.args b/tests/qemuxml2argvdata/<wbr>qemuxml2argv-disk-drive-<wbr>network-tlsx509-vxhs.args<br>
new file mode 100644<br>
index 0000000..5308a16<br>
--- /dev/null<br>
+++ b/tests/qemuxml2argvdata/<wbr>qemuxml2argv-disk-drive-<wbr>network-tlsx509-vxhs.args<br>
@@ -0,0 +1,30 @@<br>
+LC_ALL=C \<br>
+PATH=/bin \<br>
+HOME=/home/test \<br>
+USER=test \<br>
+LOGNAME=test \<br>
+QEMU_AUDIO_DRV=none \<br>
+/usr/bin/qemu-system-x86_64 \<br>
+-name QEMUGuest1 \<br>
+-S \<br>
+-M pc \<br>
+-cpu qemu32 \<br>
+-m 214 \<br>
+-smp 1,sockets=1,cores=1,threads=1 \<br>
+-uuid c7a5fdbd-edaf-9455-926a-<wbr>d65c16db1809 \<br>
+-nographic \<br>
+-nodefaults \<br>
+-chardev socket,id=charmonitor,path=/<wbr>tmp/lib/domain--1-QEMUGuest1/<wbr>monitor.sock,\<br>
+server,nowait \<br>
+-mon chardev=charmonitor,id=<wbr>monitor,mode=readline \<br>
+-no-acpi \<br>
+-boot c \<br>
+-usb \<br>
+-object tls-creds-x509,id=objvirtio-<wbr>disk0_tls0,dir=/etc/pki/qemu,\<br>
+endpoint=client,verify-peer=<wbr>yes \<br>
+-drive file.driver=vxhs,file.tls-<wbr>creds=objvirtio-disk0_tls0,\<br>
+file.vdisk-id=eb90327c-8302-<wbr>4725-9e1b-4e85ed4dc251,file.<wbr>server.0.type=tcp,\<br>
+file.server.0.host=192.168.0.<wbr>1,file.server.0.port=9999,<wbr>format=raw,if=none,\<br>
+id=drive-virtio-disk0,cache=<wbr>none \<br>
+-device virtio-blk-pci,bus=pci.0,addr=<wbr>0x4,drive=drive-virtio-disk0,\<br>
+id=virtio-disk0<br>
diff --git a/tests/qemuxml2argvtest.c b/tests/qemuxml2argvtest.c<br>
index b92ded8..0366dc3 100644<br>
--- a/tests/qemuxml2argvtest.c<br>
+++ b/tests/qemuxml2argvtest.c<br>
@@ -932,6 +932,13 @@ mymain(void)<br>
     DO_TEST("disk-drive-network-<wbr>rbd-ipv6", NONE);<br>
     DO_TEST_FAILURE("disk-drive-<wbr>network-rbd-no-colon", NONE);<br>
     DO_TEST("disk-drive-network-<wbr>vxhs", QEMU_CAPS_VXHS);<br>
+    driver.config->vxhsTLS = 1;<br>
+    DO_TEST("disk-drive-network-<wbr>tlsx509-vxhs", QEMU_CAPS_VXHS,<br>
+            QEMU_CAPS_OBJECT_TLS_CREDS_<wbr>X509);<br>
+    DO_TEST("disk-drive-network-<wbr>tlsx509-multidisk-vxhs", QEMU_CAPS_VXHS,<br>
+            QEMU_CAPS_OBJECT_TLS_CREDS_<wbr>X509);<br>
+    driver.config->vxhsTLS = 0;<br>
+    VIR_FREE(driver.config-><wbr>vxhsTLSx509certdir);<br>
     DO_TEST("disk-drive-no-boot",<br>
             QEMU_CAPS_BOOTINDEX);<br>
     DO_TEST_PARSE_ERROR("disk-<wbr>device-lun-type-invalid",<br>
<span class="HOEnZb"><font color="#888888">--<br>
2.9.5<br>
<br>
</font></span></blockquote></div><br></div>