[dm-devel] blk-mq request allocation stalls [was: Re: [PATCH v3 0/8] dm: add request-based blk-mq support]

[Jens Axboe]

On 01/07/2015 09:15 AM, Mike Snitzer wrote:
> On Wed, Jan 07 2015 at 10:32am -0500,
> Jens Axboe <axboe at kernel.dk> wrote:
>
>> On 01/07/2015 03:33 AM, Bart Van Assche wrote:
>>> On 01/06/15 17:15, Jens Axboe wrote:
>>>> blk-mq request allocation is pretty much as optimized/fast as it can be.
>>>> The slowdown must be due to one of two reasons:
>>>>
>>>> - A bug related to running out of requests, perhaps a missing queue run
>>>> or something like that.
>>>> - A smaller number of available requests, due to the requested queue
>>>> depth.
>>>>
>>>> Looking at Barts results, it looks like it's usually fast, but sometimes
>>>> very slow. That would seem to indicate it's option #1 above that is the
>>>> issue. Bart, since this seems to wait for quite a bit, would it be
>>>> possible to cat the 'tags' file for that queue when it is stuck like
>>>> that?
>>>
>>> Hello Jens,
>>>
>>> Thanks for the assistance. Is this the output you were looking for ?
>>>
>>> # dmsetup table /dev/dm-1
>>> 0 256000 multipath 0 0 2 1 service-time 0 1 2 8:32 1 1 service-time 0 1
>>> 2 8:48 1 1
>>>
>>> # ls -ld /dev/sd[cd]
>>> brw-rw---- 1 root disk 8, 32 Jan  7 11:16 /dev/sdc
>>> brw-rw---- 1 root disk 8, 48 Jan  7 11:16 /dev/sdd
>>>
>>> # time mkfs.xfs -f /dev/dm-1 &
>>> [ ... ]
>>> real    4m12.101s
>>>
>>> # for d in sdc sdd; do echo ==== $d; (cd /sys/block/$d/mq &&
>>>    find|cut -c3-|grep tag|xargs grep -aH ''); done
>>
>> You forgot dm-1, that's what mkfs is sleeping on. But given that
>> sdc/sdd look idle, it still looks like a case of dm-1 not
>> appropriately running the queues after insertion.
>
> DM never directly runs the queues of the underlying SCSI devices
> (e.g. sdc, sdd).
>
> request-based DM runs the DM device's queue on completion of a clone
> request:
>
> dm_end_request -> rq_completed -> blk_run_queue_async
>
> Which ultimately does seem to be the wrong way around (like you say:
> queue should run after insertion).

That does seem backwards. You'd normally run the queue at completion if 
you decided to stop it (or stop issuing) due to some resource 
constraints. What runs the queue after insertion of a clone to the 
underlying device?

-- 
Jens Axboe