[dm-devel] Does dm-zoned support buffered write?

Mon May 15 06:06:39 UTC 2023

On 5/12/23 19:41, Ming Lin wrote:
> On Thu, May 11, 2023 at 11:56 AM Hannes Reinecke <hare at suse.de> wrote:
>>
>> On 5/11/23 20:41, Ming Lin wrote:
>>> Hi list,
>>>
>>> I have an application that needs to use buffered_io to access SMR disk
>>> for good performance.
>>>
>>>   From "ZBD Support Restrictions" at https://zonedstorage.io/docs/linux/overview
>>> "
>>> Direct IO Writes The kernel page cache does not guarantee that cached
>>> dirty pages will be flushed to a block device in sequential sector
>>> order. This can lead to unaligned write errors if an application uses
>>> buffered writes to write to the sequential write required zones of a
>>> device. To avoid this pitfall, applications that directly use a zoned
>>> block device without a file system should always use direct I/O
>>> operations to write to the sequential write required zones of a
>>> host-managed disk (that is, they should issue write() system calls
>>> with a block device "file open" that uses the O_DIRECT flag).
>>> "
>>>
>>> Raw zbd disk only supports direct_io.
>>>
>>> Does dm-zoned support buffered io (without O_DIRECT)?
>>>
>> Yes. But I _think_ the above paragraph is ever so slightly outdated, as
>> we've spent quite a lot of time fixing sequential writes (cf blk-zoned
>> etc). So while dm-zoned is using bufferet writes there won't be any
>> sequential write issues.
>>
>> At least, I have not uncovered any of those during testing.
> 
> Hi Hannes,
> 
> I use 5.10.90 kernel and smr disk capacity is 24T.
> I followed the below guide to create dm_zone device on top of smr disk.
> https://zonedstorage.io/docs/linux/dm
> 
> Then mkfs.ext4 /dev/dm-0, but it seems hung.
> Any ideas?
> 
> [37552.217472] device-mapper: uevent: version 1.0.3
> [37552.217549] device-mapper: ioctl: 4.43.0-ioctl (2020-10-01)
> initialised: dm-devel at redhat.com
> [37575.608500] device-mapper: zoned metadata: (dmz-5000cca2bfc0db21):
> DM-Zoned metadata version 2
> [37575.608502] device-mapper: zoned metadata: (sdx): Host-managed
> zoned block device
> [37575.608503] device-mapper: zoned metadata: (sdx):   50782535680
> 512-byte logical sectors (offset 0)
> [37575.608503] device-mapper: zoned metadata: (sdx):   96860 zones of
> 524288 512-byte logical sectors (offset 0)
> [37575.608504] device-mapper: zoned metadata: (dmz-5000cca2bfc0db21):
>   96860 zones of 524288 512-byte logical sectors
> [37575.609204] device-mapper: zoned: (dmz-5000cca2bfc0db21): Target
> device: 50771001344 512-byte logical sectors (6346375168 blocks)
> [38101.543353] INFO: task mkfs.ext4:1411791 blocked for more than 122 seconds.
> [38101.543380]       Tainted: G           OE     5.10.90.bm.1-amd64+ #2
> [38101.543395] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [38101.543411] task:mkfs.ext4       state:D stack:    0 pid:1411791
> ppid:1388660 flags:0x00004000
> [38101.543415] Call Trace:
> [38101.543422]  __schedule+0x3fd/0x760
> [38101.543425]  schedule+0x46/0xb0
> [38101.543426]  io_schedule+0x12/0x40
> [38101.543429]  wait_on_page_bit+0x133/0x270
> [38101.543431]  ? __page_cache_alloc+0xa0/0xa0
> [38101.543432]  wait_on_page_writeback+0x25/0x70
> [38101.543434]  __filemap_fdatawait_range+0x86/0xf0
> [38101.543435]  file_write_and_wait_range+0x74/0xb0
> [38101.543438]  blkdev_fsync+0x16/0x40
> [38101.543441]  do_fsync+0x38/0x60
> [38101.543442]  __x64_sys_fsync+0x10/0x20
> [38101.543445]  do_syscall_64+0x2d/0x70
> [38101.543446]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> 
> ===
> Below are the steps I did:
> 
> root at smr_dev:~# blkzone reset /dev/sdx
> 
> root at smr_dev:~# dmzadm --format /dev/sdx
> /dev/sdx: 50782535680 512-byte sectors (24215 GiB)
>    Host-managed device
>    96860 zones, offset 0
>    96860 zones of 524288 512-byte sectors (256 MiB)
>    65536 4KB data blocks per zone
> Resetting sequential zones
> Writing primary metadata set
>    Writing mapping table
>    Writing bitmap blocks
>    Writing super block to sdx block 0
> Writing secondary metadata set
>    Writing mapping table
>    Writing bitmap blocks
>    Writing super block to sdx block 196608
> Syncing disk
> Done.
> 
Hmm. I don't actually see how many CMR zones the drive has.

> root at smr_dev:~# dmzadm --start /dev/sdx
> /dev/sdx: 50782535680 512-byte sectors (24215 GiB)
>    Host-managed device
>    96860 zones, offset 0
>    96860 zones of 524288 512-byte sectors (256 MiB)
>    65536 4KB data blocks per zone
> sdx: starting dmz-5000cca2bfc0db21, metadata ver. 2, uuid
> 7495e21a-23d9-49f4-832a-76b32136078b
> 
> root at smr_dev:~# mkfs.ext4 /dev/dm-0
> mke2fs 1.44.5 (15-Dec-2018)
> Discarding device blocks: done
> Creating filesystem with 6346375168 4k blocks and 396648448 inodes
> Filesystem UUID: c47de06d-6cf6-4a85-9502-7830ca2f4526
> Superblock backups stored on blocks:
>          32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
>          4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
>          102400000, 214990848, 512000000, 550731776, 644972544, 1934917632,
>          2560000000, 3855122432, 5804752896
> 
> Allocating group tables: done
> Writing inode tables: done
> Creating journal (262144 blocks): done
> Writing superblocks and filesystem accounting information:
> 
> ===
> At another terminal,
> 
> root at smr_dev:~# ps aux | grep mkfs.ext4
> root     1411791  2.8  0.0  30992 19864 pts/1    D+   01:30   0:01
> mkfs.ext4 /dev/dm-0
> root     1413640  0.0  0.0  13972  2496 pts/0    S+   01:31   0:00
> grep mkfs.ext4
> 
> root at smr_dev:~# cat /proc/1411791/stack
> [<0>] wait_on_page_bit+0x133/0x270
> [<0>] wait_on_page_writeback+0x25/0x70
> [<0>] __filemap_fdatawait_range+0x86/0xf0
> [<0>] file_write_and_wait_range+0x74/0xb0
> [<0>] blkdev_fsync+0x16/0x40
> [<0>] do_fsync+0x38/0x60
> [<0>] __x64_sys_fsync+0x10/0x20
> [<0>] do_syscall_64+0x2d/0x70
> [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

But that just means that we're waiting for I/O to complete; there must 
be another thread processing the I/O.
If this is the only active thread in you system something is seriously 
hosed.

But I guess I don't need to tell _you_ that :-)

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman