[dm-devel] [PATCH] crash in md-raid1 and md-raid10 due to incorrect list manipulation

[Neil Brown]

Mikulas Patocka <mpatocka at redhat.com> writes:

> The commit 55ce74d4bfe1b9444436264c637f39a152d1e5ac (md/raid1: ensure 
> device failure recorded before write request returns) is causing crash in 
> the LVM2 testsuite test shell/lvchange-raid.sh. For me the crash is 100% 
> reproducible.
>
> The reason for the crash is that the newly added code in raid1d moves the 
> list from conf->bio_end_io_list to tmp, then tests if tmp is non-empty and 
> then incorrectly pops the bio from conf->bio_end_io_list (which is empty 
> because the list was alrady moved).
>
> Raid-10 has a similar bug.

Ouch.  I can't have been thinking when I wrote that code!

Thanks for finding and fixing this.  Patch will be sent to Linus in time
for next -rc.

Thanks,
NeilBrown

>
> Kernel Fault: Code=15 regs=000000006ccb8640 (Addr=0000000100000000)
> CPU: 3 PID: 1930 Comm: mdX_raid1 Not tainted 4.2.0-rc5-bisect+ #35
> task: 000000006cc1f258 ti: 000000006ccb8000 task.ti: 000000006ccb8000
>
>      YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI
> PSW: 00001000000001001111111000001111 Not tainted
> r00-03  000000ff0804fe0f 000000001059d000 000000001059f818 000000007f16be38
> r04-07  000000001059d000 000000007f16be08 0000000000200200 0000000000000001
> r08-11  000000006ccb8260 000000007b7934d0 0000000000000001 0000000000000000
> r12-15  000000004056f320 0000000000000000 0000000000013dd0 0000000000000000
> r16-19  00000000f0d00ae0 0000000000000000 0000000000000000 0000000000000001
> r20-23  000000000800000f 0000000042200390 0000000000000000 0000000000000000
> r24-27  0000000000000001 000000000800000f 000000007f16be08 000000001059d000
> r28-31  0000000100000000 000000006ccb8560 000000006ccb8640 0000000000000000
> sr00-03  0000000000249800 0000000000000000 0000000000000000 0000000000249800
> sr04-07  0000000000000000 0000000000000000 0000000000000000 0000000000000000
>
> IASQ: 0000000000000000 0000000000000000 IAOQ: 000000001059f61c 000000001059f620
>  IIR: 0f8010c6    ISR: 0000000000000000  IOR: 0000000100000000
>  CPU:        3   CR30: 000000006ccb8000 CR31: 0000000000000000
>  ORIG_R28: 000000001059d000
>  IAOQ[0]: call_bio_endio+0x34/0x1a8 [raid1]
>  IAOQ[1]: call_bio_endio+0x38/0x1a8 [raid1]
>  RP(r2): raid_end_bio_io+0x88/0x168 [raid1]
> Backtrace:
>  [<000000001059f818>] raid_end_bio_io+0x88/0x168 [raid1]
>  [<00000000105a4f64>] raid1d+0x144/0x1640 [raid1]
>  [<000000004017fd5c>] kthread+0x144/0x160
>
> Signed-off-by: Mikulas Patocka <mpatocka at redhat.com>
> Fixes: 55ce74d4bfe1 ("md/raid1: ensure device failure recorded before write request returns.")
> Fixes: 95af587e95aa ("md/raid10: ensure device failure recorded before write request returns.")
>
> ---
>  drivers/md/raid1.c  |    4 ++--
>  drivers/md/raid10.c |    4 ++--
>  2 files changed, 4 insertions(+), 4 deletions(-)
>
> Index: linux-2.6/drivers/md/raid1.c
> ===================================================================
> --- linux-2.6.orig/drivers/md/raid1.c	2015-10-01 21:10:05.000000000 +0200
> +++ linux-2.6/drivers/md/raid1.c	2015-10-01 21:10:58.000000000 +0200
> @@ -2437,8 +2437,8 @@ static void raid1d(struct md_thread *thr
>  		}
>  		spin_unlock_irqrestore(&conf->device_lock, flags);
>  		while (!list_empty(&tmp)) {
> -			r1_bio = list_first_entry(&conf->bio_end_io_list,
> -						  struct r1bio, retry_list);
> +			r1_bio = list_first_entry(&tmp, struct r1bio,
> +						  retry_list);
>  			list_del(&r1_bio->retry_list);
>  			raid_end_bio_io(r1_bio);
>  		}
> Index: linux-2.6/drivers/md/raid10.c
> ===================================================================
> --- linux-2.6.orig/drivers/md/raid10.c	2015-10-01 21:11:02.000000000 +0200
> +++ linux-2.6/drivers/md/raid10.c	2015-10-01 21:11:19.000000000 +0200
> @@ -2804,8 +2804,8 @@ static void raid10d(struct md_thread *th
>  		}
>  		spin_unlock_irqrestore(&conf->device_lock, flags);
>  		while (!list_empty(&tmp)) {
> -			r10_bio = list_first_entry(&conf->bio_end_io_list,
> -						  struct r10bio, retry_list);
> +			r10_bio = list_first_entry(&tmp, struct r10bio,
> +						   retry_list);
>  			list_del(&r10_bio->retry_list);
>  			raid_end_bio_io(r10_bio);
>  		}
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/dm-devel/attachments/20151009/3aa3bbab/attachment.sig>