<html><head></head><body><div><br></div><div><span></span></div><div><br></div><div>On Tue, 2021-05-11 at 00:15 +0000, Chaitanya Kulkarni wrote:</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>Hi,<br></div><div><br></div><div>* Background :-<br></div><div>-----------------------------------------------------------------------<br></div><div><br></div><div>Copy offload is a feature that allows file-systems or storage devices<br></div><div>to be instructed to copy files/logical blocks without requiring<br></div><div>involvement of the local CPU.<br></div><div><br></div><div>With reference to the RISC-V summit keynote [1] single threaded<br></div><div>performance is limiting due to Denard scaling and multi-threaded<br></div><div>performance is slowing down due Moore's law limitations. With the rise<br></div><div>of SNIA Computation Technical Storage Working Group (TWG) [2],<br></div><div>offloading computations to the device or over the fabrics is becoming<br></div><div>popular as there are several solutions available [2]. One of the common<br></div><div>operation which is popular in the kernel and is not merged yet is Copy<br></div><div>offload over the fabrics or on to the device.<br></div><div><br></div><div>* Problem :-<br></div><div>-----------------------------------------------------------------------<br></div><div><br></div><div>The original work which is done by Martin is present here [3]. The<br></div><div>latest work which is posted by Mikulas [4] is not merged yet. These two<br></div><div>approaches are totally different from each other. Several storage<br></div><div>vendors discourage mixing copy offload requests with regular READ/WRITE<br></div><div>I/O. Also, the fact that the operation fails if a copy request ever<br></div><div>needs to be split as it traverses the stack it has the unfortunate<br></div><div>side-effect of preventing copy offload from working in pretty much<br></div><div>every common deployment configuration out there.<br></div><div><br></div><div>* Current state of the work :-<br></div><div>-----------------------------------------------------------------------<br></div><div><br></div><div>With [3] being hard to handle arbitrary DM/MD stacking without<br></div><div>splitting the command in two, one for copying IN and one for copying<br></div><div>OUT. Which is then demonstrated by the [4] why [3] it is not a suitable<br></div><div>candidate. Also, with [4] there is an unresolved problem with the<br></div><div>two-command approach about how to handle changes to the DM layout<br></div><div>between an IN and OUT operations.<br></div><div><br></div><div>* Why Linux Kernel Storage System needs Copy Offload support now ?<br></div><div>-----------------------------------------------------------------------<br></div><div><br></div><div>With the rise of the SNIA Computational Storage TWG and solutions [2],<br></div><div>existing SCSI XCopy support in the protocol, recent advancement in the<br></div><div>Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer<br></div><div>DMA support in the Linux Kernel mainly for NVMe devices [7] and<br></div><div>eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit<br></div><div>from Copy offload operation.<br></div><div><br></div><div>With this background we have significant number of use-cases which are<br></div><div>strong candidates waiting for outstanding Linux Kernel Block Layer Copy<br></div><div>Offload support, so that Linux Kernel Storage subsystem can to address<br></div><div>previously mentioned problems [1] and allow efficient offloading of the<br></div><div>data related operations. (Such as move/copy etc.)<br></div><div><br></div><div>For reference following is the list of the use-cases/candidates waiting<br></div><div>for Copy Offload support :-<br></div><div><br></div><div>1. SCSI-attached storage arrays.<br></div><div>2. Stacking drivers supporting XCopy DM/MD.<br></div><div>3. Computational Storage solutions.<br></div><div>7. File systems :- Local, NFS and Zonefs.<br></div><div>4. Block devices :- Distributed, local, and Zoned devices.<br></div><div>5. Peer to Peer DMA support solutions.<br></div><div>6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF.<br></div><div><br></div><div>* What we will discuss in the proposed session ?<br></div><div>-----------------------------------------------------------------------<br></div><div><br></div><div>I'd like to propose a session to go over this topic to understand :-<br></div><div><br></div><div>1. What are the blockers for Copy Offload implementation ?<br></div><div>2. Discussion about having a file system interface.<br></div><div>3. Discussion about having right system call for user-space.<br></div><div>4. What is the right way to move this work forward ?<br></div><div>5. How can we help to contribute and move this work forward ?<br></div><div><br></div><div>* Required Participants :-<br></div><div>-----------------------------------------------------------------------<br></div><div><br></div><div>I'd like to invite file system, block layer, and device drivers<br></div><div>developers to:-<br></div><div><br></div><div>1. Share their opinion on the topic.<br></div><div>2. Share their experience and any other issues with [4].<br></div><div>3. Uncover additional details that are missing from this proposal.<br></div><div><br></div><div>Required attendees :-<br></div><div><br></div><div>Martin K. Petersen<br></div><div>Jens Axboe<br></div><div>Christoph Hellwig<br></div><div>Bart Van Assche<br></div><div>Zach Brown<br></div><div>Roland Dreier<br></div><div>Ric Wheeler<br></div><div>Trond Myklebust<br></div><div>Mike Snitzer<br></div><div>Keith Busch<br></div><div>Sagi Grimberg<br></div><div>Hannes Reinecke<br></div><div>Frederick Knight<br></div><div>Mikulas Patocka<br></div><div>Keith Busch<br></div><div><br></div><div>Regards,<br></div><div>Chaitanya<br></div><div><br></div></blockquote><div><br></div><div>+1 here. I would like to see how this pans out as many differences may be observed from a standards, implementation and operations point of view.</div><div><br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>[1]<a href="https://content.riscv.org/wp-content/uploads/2018/12/A-New-Golden-Age-for-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterson-.pdf">https://content.riscv.org/wp-content/uploads/2018/12/A-New-Golden-Age-for-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterson-.pdf</a><br></div><div>[2] <a href="https://www.snia.org/computational">https://www.snia.org/computational</a><br></div><div><a href="https://www.napatech.com/support/resources/solution-descriptions/napatech-smartnic-solution-for-hardware-offload/">https://www.napatech.com/support/resources/solution-descriptions/napatech-smartnic-solution-for-hardware-offload/</a><br></div><div> <a href="https://www.eideticom.com/products.html">https://www.eideticom.com/products.html</a><br></div><div><a href="https://www.xilinx.com/applications/data-center/computational-storage.html">https://www.xilinx.com/applications/data-center/computational-storage.html</a><br></div><div>[3] git://git.kernel.org/pub/scm/linux/kernel/git/mkp/linux.git xcopy<br></div><div>[4] <a href="https://www.spinics.net/lists/linux-block/msg00599.html">https://www.spinics.net/lists/linux-block/msg00599.html</a><br></div><div>[5] <a href="https://lwn.net/Articles/793585/">https://lwn.net/Articles/793585/</a><br></div><div>[6] <a href="https://nvmexpress.org/new-nvmetm-specification-defines-zoned">https://nvmexpress.org/new-nvmetm-specification-defines-zoned</a>-<br></div><div>namespaces-zns-as-go-to-industry-technology/<br></div><div>[7] <a href="https://github.com/sbates130272/linux-p2pmem">https://github.com/sbates130272/linux-p2pmem</a><br></div><div>[8] <a href="https://kernel.dk/io_uring.pdf">https://kernel.dk/io_uring.pdf</a><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>--<br></div><div>dm-devel mailing list<br></div><div><a href="mailto:dm-devel@redhat.com">dm-devel@redhat.com</a><br></div><div><a href="https://listman.redhat.com/mailman/listinfo/dm-devel">https://listman.redhat.com/mailman/listinfo/dm-devel</a><br></div><div><br></div></blockquote></body></html>