<div dir="ltr"><div style="font-size:14px">I report a bug [Bug 1227257] . In the environment libvirt-1.2.16.tar.gz + qemu-img version 2.1.2 + ceph version 0.94.1.</div><div style="font-size:14px">libvirt pool will become inactive after one client does vol.delete and the other does pool.refresh in the same pool simultaneously.</div><div style="font-size:14px"><br></div><div style="font-size:14px">The reason is that rbd_list and rbd_open are not wrapped in an atomic operation, but two seperate operations.</div><div style="font-size:14px">For example, two clients are operating in the same pool at the same time. One client does rbd_list, and got 10 rbd images, meanwhile, the other client deletes one of the rbd image. In this situation, when the first client does next operation, such as rbd_open , the command may fail, because the rbd image has been removed.</div><div style="font-size:14px"><br></div><div style="font-size:14px">I write a testcase in python to reproduce this problem as follow(also have put it in the attachment):</div><div style="font-size:14px"><div>##################################</div><div>import libvirt</div><div>import sys</div><div>import time</div><div>import sys</div><div>#coding:utf-8</div><div><br></div><div>QEMU_URL = 'qemu:///system'</div><div><br></div><div>VOL_TEMPLATE='''</div><div><volume></div><div> <name>{vol}</name></div><div> <key>{pool}/{vol}</key></div><div> <source></div><div> </source></div><div> <capacity unit='MB'>{cap_size}</capacity></div><div> <allocation unit='MB'>{alloc_size}</allocation></div><div> <target></div><div> <path>rbd:{pool}/{vol}</path></div><div> <format type='unknown'/></div><div> <permissions></div><div> <mode>00</mode></div><div> <owner>0</owner></div><div> <group>0</group></div><div> </permissions></div><div> </target></div><div></volume>'''</div><div><br></div><div>def create_vol(pool_name, vol_name, cap_size, alloc_size):</div><div> conn = libvirt.open(QEMU_URL)</div><div> if conn == None:</div><div> print 'Failed to open connection to the hypervisor'</div><div> sys.exit(1)</div><div><br></div><div> try:</div><div> pool = conn.storagePoolLookupByName(pool_name)</div><div> pool.refresh()</div><div> template = VOL_TEMPLATE.format(pool=pool_name, vol=vol_name, cap_size=cap_size, alloc_size=alloc_size)</div><div> pool.createXML(template, 0)</div><div> except:</div><div> print 'Failed to open pool'</div><div> sys.exit(1)</div><div> finally:</div><div><span style="white-space:pre-wrap"> </span>if conn is not None:</div><div> conn.close()</div><div><br></div><div>def destroy_vol(pool_name, vol_name):</div><div> conn = libvirt.open(QEMU_URL)</div><div> if conn == None:</div><div> sys.exit(1)</div><div> pool = conn.storagePoolLookupByName(pool_name)</div><div> pool.refresh(0) </div><div> vol = pool.storageVolLookupByName(vol_name)</div><div> if vol is not None:</div><div> vol.delete(0)</div><div> if conn is not None:</div><div> conn.close()</div><div><br></div><div>if sys.argv[2] == 'create':</div><div> for i in range(1, 20):</div><div><span style="white-space:pre-wrap"> </span>volname = 'pool-down-test-%s-%d' % (sys.argv[1], i)</div><div> print 'create %s' % volname</div><div> create_vol('capacity', volname, '500', '500')</div><div>elif sys.argv[2] == 'destroy':</div><div> for i in range(1, 20):</div><div><span style="white-space:pre-wrap"> </span>volname = 'pool-down-test-%s-%d' % (sys.argv[1], i)</div><div> print 'destroy %s' % volname</div><div> destroy_vol('capacity', volname)</div><div>else:</div><div> print 'Usage: python vol-test.py clientId OPER'</div><div> print ' '</div><div> print 'where clientId : a num/string used to distinguish different client'</div><div> print ' OPER : create/destroy'</div></div><div style="font-size:14px"><br></div><div style="font-size:14px"><br></div><div style="font-size:14px"><br></div><div style="font-size:14px"><br></div><div style="font-size:14px">Patch for this is as fllow:</div><div style="font-size:14px"><br></div><div style="font-size:14px"><div>diff --git a/src/storage/storage_backend_rbd.c b/src/storage/storage_backend_rbd.c</div><div>index ae4bcb3..24fbc84 100644</div><div>--- a/src/storage/storage_backend_rbd.c</div><div>+++ b/src/storage/storage_backend_rbd.c</div><div>@@ -266,6 +266,46 @@ static int virStorageBackendRBDCloseRADOSConn(virStorageBackendRBDStatePtr ptr)</div><div> return ret;</div><div> }</div><div><br></div><div>+static int volStorageBackendRBDVolIsExist(char *volname, virStorageBackendRBDStatePtr ptr)</div><div>+{</div><div>+ int ret = -1;</div><div>+ char *name, *names = NULL;</div><div>+ size_t max_size = 1024;</div><div>+ int len = -1;</div><div>+</div><div>+ while (true) {</div><div>+ if (VIR_ALLOC_N(names, max_size) < 0)</div><div>+ goto cleanup;</div><div>+</div><div>+ len = rbd_list(ptr->ioctx, names, &max_size);</div><div>+ if (len >= 0)</div><div>+ break;</div><div>+ if (len != -ERANGE) {</div><div>+ VIR_WARN("%s", _("A problem occurred while listing RBD images"));</div><div>+ goto cleanup;</div><div>+ }</div><div>+ VIR_FREE(names);</div><div>+ }</div><div>+</div><div>+ for (name = names; name < names + max_size;) {</div><div>+</div><div>+ if (STREQ(name, ""))</div><div>+ break;</div><div>+</div><div>+ name += strlen(name) + 1;</div><div>+ if (STREQ(volname, name)) {</div><div>+ VIR_ERROR("RBD images '%s' is exist, but cannot open it", volname);</div><div>+ ret = -2;</div><div>+ break;</div><div>+ }</div><div>+ }</div><div>+ ret = 0;</div><div>+</div><div>+cleanup:</div><div>+ VIR_FREE(names);</div><span class="im"><div>+ return ret;</div><div>+}</div><div>+</div></span><div> static int volStorageBackendRBDRefreshVolInfo(virStorageVolDefPtr vol,</div><div> virStoragePoolObjPtr pool,</div><div> virStorageBackendRBDStatePtr ptr)</div><div>@@ -276,8 +316,15 @@ static int volStorageBackendRBDRefreshVolInfo(virStorageVolDefPtr vol,</div><div><br></div><div> r = rbd_open(ptr->ioctx, vol->name, &image, NULL);</div><div> if (r < 0) {</div><div>- virReportSystemError(-r, _("failed to open the RBD image '%s'"),</div><div>- vol->name);</div><div>+ VIR_DEBUG("failed to open RBD image '%s', check if it was still exist in its pool",\</div><div>+ vol->name);</div><div>+ if (volStorageBackendRBDVolIsExist(vol->name, ptr) == 0) {</div><div>+ VIR_DEBUG("vol '%s' may be removed by the other rbd client", vol->name);</div><div>+ ret = 0;</div><div>+ } else {</div><div>+ virReportSystemError(-r, _("failed to open the RBD image '%s'"),</div><div>+ vol->name);</div><div>+ }</div></div><div style="font-size:14px"><div> return ret;</div><div> }</div></div></div>