[dm-devel] Any device mapper target that stores data in files?

[Doug Dumitru]

Mr. Padilha,

In general, what you are trying to do is discouraged.  It is possible to
open and read/write files from kernel space, but the architecture is not
very portable.  I actually do this all the time, but only for debugging
traces.

I would recommend another implementation method if you want to implement a
block device backed by files.  The "Network Block Device" is a project that
includes a kernel module (nbd.ko) and a user-space server.  The kernel
module is usable without the user space module and lets you implement a
decent, multi-threaded, block device completely from user space code.

nbd.ko is reached by opening a master device (which is a character device)
and then reading/writing commands into the single file handle.  Operations
are "async" so they multi-thread easily and you can easily implement all of
the block device functions including new stuff like discard.

Normally, the user-space nature of this would be a performance hit, but if
you are using the "cloud", the extra latency might not matter much.

Doug Dumitru
EasyCo LLC


On Sun, Aug 16, 2015 at 9:55 PM, Thiago Padilha <tpadilha84 at gmail.com>
wrote:

> Hi
>
> First of all, I'm very new on the subject(kernel programming/device
> mappers/block devices), so forgive if I say anything stupid.
>
> I need a device mapper target that stores data in files of fixed sizes,
> probably defined when the virtual device is first created. For the sake of
> explanation, lets call this target as "dirdm" and assume there's an
> userspace tool of the same name that can be used to manage such virtual
> devices. Now let's say I have an empty directory "/dirdm" and want to
> create a virtual device with 4k "block size" on top of it:
>
>     dirdm create --size 10G --block-size 4k /dirdm
>
> After this command is executed, there's a new file with 4k size at
> /dirdm/0. This file will be used to store the first block of the device.
> Here are the filenames that represent some blocks of this virtual device:
>
> - block 0: 0
> - block 1: 4096
> - block 2 8192
> - block 3: 12288
> - block 4: 16384
>
> That is, each block is stored in a file named after the first byte on the
> block. Querying a block that has no data written will simply return 0(block
> files are also initialized with 0), so it behaves like a sparse block
> device.
>
> Is there any device mapper implementation that manages data in a similar
> manner? Note that these details about filenames or directory structure are
> not important, all I care about is that the dm target splits data in
> relatively small files on a directory.
>
> The reason for these requirements is to be able to efficiently store
> arbitrary block devices in a cloud storage service such a google
> drive/onedrive without syncing a big loop device on every modification(Only
> files representing affected blocks are modified). One useful application
> would be to create a big device(such as 100g size) and add luks/ext4 on top
> of it, which would be a great way to store private data.
>
> If there are no existing dm targets matching these requirements, what
> would be a recommended reading/documentation to get started on implementing
> this on my own?
>
>
> --
> dm-devel mailing list
> dm-devel at redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>



-- 
Doug Dumitru
EasyCo LLC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/dm-devel/attachments/20150817/9ba802ff/attachment.htm>