<div>Hello Christophe, Ben, Hannes, Martin, Bart, <div>I am a member of host-side software development team of ZXUSP storage project in ZTE Corporation. Facing the market demand, our team decides to write code to promote multipath efficiency next month. The whole idea is in the mail below.We hope to participate in and make progress with the open source community, so any suggestion and comment would be welcome. Thanks, Tang ------------------------------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------------------------------ 1. Problem In these scenarios, multipath processing efficiency is low: 1) Many paths exist in each multipath device, 2) Devices addition or deletion during iSCSI login/logout or FC link up/down. 2. Reasons Multipath process uevents one by one, and each one also produce a new dm addition change or deletion uevent to increased system resource consumption, actually most of these uevents have no sense at all. E.g. login procedure of 4 iSCSI sessions with 3 LUNs: 1) Multipath processes these uevents one by one: UDEV [89068.806214] add /devices/platform/host3/session44/target3:0:0/3:0:0:0/block/sdc (block) UDEV [89068.909457] add /devices/platform/host3/session44/target3:0:0/3:0:0:2/block/sdg (block) UDEV [89068.944956] add /devices/platform/host3/session44/target3:0:0/3:0:0:1/block/sde (block) UDEV [89068.959215] add /devices/platform/host5/session46/target5:0:0/5:0:0:0/block/sdh (block) UDEV [89068.978558] add /devices/platform/host5/session46/target5:0:0/5:0:0:2/block/sdk (block) UDEV [89069.004217] add /devices/platform/host5/session46/target5:0:0/5:0:0:1/block/sdj (block) UDEV [89069.486361] add /devices/platform/host4/session45/target4:0:0/4:0:0:1/block/sdf (block) UDEV [89069.495194] add /devices/platform/host4/session45/target4:0:0/4:0:0:0/block/sdd (block) UDEV [89069.511628] add /devices/platform/host4/session45/target4:0:0/4:0:0:2/block/sdi (block) UDEV [89069.716292] add /devices/platform/host6/session47/target6:0:0/6:0:0:0/block/sdl (block) UDEV [89069.748456] add /devices/platform/host6/session47/target6:0:0/6:0:0:1/block/sdm (block) UDEV [89069.789662] add /devices/platform/host6/session47/target6:0:0/6:0:0:2/block/sdn (block) 2) Multipath also produce DM uvents by step 1), which would be processed by udev and other process who listening kernel: KERNEL[89068.899614] add /devices/virtual/block/dm-2 (block) KERNEL[89068.902477] change /devices/virtual/block/dm-2 (block) KERNEL[89068.955364] add /devices/virtual/block/dm-3 (block) KERNEL[89068.960663] change /devices/virtual/block/dm-3 (block) KERNEL[89069.018903] add /devices/virtual/block/dm-4 (block) KERNEL[89069.042102] change /devices/virtual/block/dm-4 (block) KERNEL[89069.297252] change /devices/virtual/block/dm-2 (block) KERNEL[89069.346718] change /devices/virtual/block/dm-4 (block) KERNEL[89069.388361] change /devices/virtual/block/dm-3 (block) KERNEL[89069.548270] change /devices/virtual/block/dm-4 (block) KERNEL[89069.607306] change /devices/virtual/block/dm-2 (block) KERNEL[89070.118067] change /devices/virtual/block/dm-3 (block) KERNEL[89070.136256] change /devices/virtual/block/dm-2 (block) KERNEL[89070.157222] change /devices/virtual/block/dm-4 (block) KERNEL[89070.216269] change /devices/virtual/block/dm-3 (block) 3) After processing by udev in step 2), udev also transfers these uevents to multipathŖē UDEV [89068.926428] add /devices/virtual/block/dm-2 (block) UDEV [89069.007511] add /devices/virtual/block/dm-3 (block) UDEV [89069.098054] add /devices/virtual/block/dm-4 (block) UDEV [89069.291184] change /devices/virtual/block/dm-2 (block) UDEV [89069.320632] change /devices/virtual/block/dm-4 (block) UDEV [89069.381434] change /devices/virtual/block/dm-3 (block) UDEV [89069.637666] change /devices/virtual/block/dm-2 (block) UDEV [89069.682303] change /devices/virtual/block/dm-4 (block) UDEV [89069.860877] change /devices/virtual/block/dm-2 (block) UDEV [89069.904735] change /devices/virtual/block/dm-4 (block) UDEV [89070.327167] change /devices/virtual/block/dm-2 (block) UDEV [89070.371114] change /devices/virtual/block/dm-4 (block) UDEV [89070.434592] change /devices/virtual/block/dm-3 (block) UDEV [89070.572072] change /devices/virtual/block/dm-3 (block) UDEV [89070.703181] change /devices/virtual/block/dm-3 (block) 4) Multipath processes uevents above. The efficiency of processing uevents one by one is low, and it produces too many uevents, which further reducing the processing efficiency. The problem is similar in the logout procedure of iSCSI sessions. 3. Negative effect Multipath processes so slowly that it is not satisfied to some applications, For example, Openstack is often timeout in waiting for the creation of multipath devices. 4. Proposal Other than processing uevents one by one, uevents which coming from the same LUN devices can be mergered to one, and then uevent processing thread only needs to process it once, and it only produces one DM addition uevent which could reduce system resource consumption. The example in Chapter 2 is continued to use to explain the proposal: 1) Multipath receives block device addition uevents from udev: UDEV [89068.806214] add /devices/platform/host3/session44/target3:0:0/3:0:0:0/block/sdc (block) UDEV [89068.909457] add /devices/platform/host3/session44/target3:0:0/3:0:0:2/block/sdg (block) UDEV [89068.944956] add /devices/platform/host3/session44/target3:0:0/3:0:0:1/block/sde (block) UDEV [89068.959215] add /devices/platform/host5/session46/target5:0:0/5:0:0:0/block/sdh (block) UDEV [89068.978558] add /devices/platform/host5/session46/target5:0:0/5:0:0:2/block/sdk (block) UDEV [89069.004217] add /devices/platform/host5/session46/target5:0:0/5:0:0:1/block/sdj (block) UDEV [89069.486361] add /devices/platform/host4/session45/target4:0:0/4:0:0:1/block/sdf (block) UDEV [89069.495194] add /devices/platform/host4/session45/target4:0:0/4:0:0:0/block/sdd (block) UDEV [89069.511628] add /devices/platform/host4/session45/target4:0:0/4:0:0:2/block/sdi (block) UDEV [89069.716292] add /devices/platform/host6/session47/target6:0:0/6:0:0:0/block/sdl (block) UDEV [89069.748456] add /devices/platform/host6/session47/target6:0:0/6:0:0:1/block/sdm (block) UDEV [89069.789662] add /devices/platform/host6/session47/target6:0:0/6:0:0:2/block/sdn (block) 2) Multipath merges these 12 uevents to 3 internal uvents UEVENT add sdc sdh sdd sdl UEVENT add sde sdj sdf sdm UEVENT add sdg sdk sdi sdn 3) Multipath process these 3 uevents one by one, and only produce 3 addition DM uvents, no dm change uevent exists. KERNEL[89068.899614] add /devices/virtual/block/dm-2 (block) KERNEL[89068.955364] add /devices/virtual/block/dm-3 (block) KERNEL[89069.018903] add /devices/virtual/block/dm-4 (block) 4) Udev process these uevents above, and transfer it to multipath UDEV [89068.926428] add /devices/virtual/block/dm-2 (block) UDEV [89069.007511] add /devices/virtual/block/dm-3 (block) UDEV [89069.098054] add /devices/virtual/block/dm-4 (block) 5) Multipath processes these uevents above, and finishes the creation of multipath devices. 5. Coding After taking over uevents form uevent listening thread, uevent processing thread can merger these uevents before processingŖē int uevent_dispatch(int (*uev_trigger)(struct uevent *, void * trigger_data), void * trigger_data) { ... while (1) { ... list_splice_init(&uevq, &uevq_tmp); ... list_merger_uevents(&uevq_tmp); service_uevq(&uevq_tmp); } ... } In structure of Ą°struct ueventĄą , an additional member of Ą°char wwid[WWID_SIZE]Ąą is added to record each device WWID for addition or change uevent to identify whether these uevents coming from the same LUN, and an additional member of Ą°struct list_head merger_nodeĄą is added to record the list of uevents which having been merged with this uevent: struct uevent { struct list_head node; struct list_head merger_node; char wwid[WWID_SIZE] struct udev_device *udev; ... }; In list_merger_uevents(&uevq_tmp), each node is traversed from the latest to the oldest, and the older node with the same WWID and uevent type(e.g. add) would be moved to the merger_node list of the later node. If a deletion uevent node occurred, other older uevent nodes about this device would be filtered(Thanks to MartinĄ¯s idea). After above processing, attention must be paid to that the parameter Ą°struct uevent * uevĄą is not a single uevent any more in and after uev_trigger(), code need to be modified to process batch uevents in uev_add_path() and so on. </div></div>