[linux-lvm] Confusing error: No space left on device

[Stuart Gathman]

On 02/17/2015 11:20 AM, Jason Pyeron wrote:
> This is likely a problem between keyboard and chair, but it has got me confused all the same.
>
> I was having issues with rsync, so I switched to dd and the same problem persists.
>
> Any suggestions?
>
> History says "lvcreate --size 8589934592b --name ciphershed-dmz-http vg_five66
"No space left on device" is a filesystem error.   This happens to me 
occasionally when the /dev/... path does not actually exist for various 
reasons, and the output is going into the root filesystem instead.

In your case, the "varying reason" is rsync.  Rsync removes 
/dev/vg_five66/ciphershed-dmz-http, and creates a regular file - which 
it proceeds to fill with data from your source lv until running out of 
space in your root fs.  When you removed and recreated the LV, this 
removed the rsync handiwork and your subsequent dd worked again.  But 
when you ran rsync again - bye bye link to LV device node.

There is a patch to rsync for copying devices nodes - but the /dev/... 
pathnames are often symlinks.
There are utilities for copying block devices with rsync protocol - I 
have a python script that I have used around somewhere, but I can't put 
my fingers on it at the moment.