gorged harddrive

[Roger Heflin]

Todd Denniston wrote:
> Ed Greshko wrote, On 03/31/2008 10:57 PM:
>> charles f. zeitler wrote:
>>> --- Ed Greshko <Ed.Greshko at greshko.com> wrote:
>>>
>>>> charles f. zeitler wrote:
>>>>> i've been pruning my "downloads" disk,
>>>>> rather drastically, and not making a dent.
>>>>>
>>>>> today some more, less drastic but still
>>>>> hefty, same result.
>>>>>
>>>>> revisited du- checked it twice - three
>>>>> times- yup, it reports one directory at
>>>>> 800+ gb- on a 400gb disk!
>>>>>
>>>>> fsck (forced) failed to report any problems,
>>>>> there don't seem to be any symlinks,
>>>>> and the sub-direcory sizes are sane...
>>>>>
>>>>> any ideas welcome, and appreciated.
>>>> Instead of telling people what you are seeing it would be better to 
>>>> show the actual commands and output.
>>>>
>>> good point.
>>>
>>>
>>> [fedora_8 at Nyarlethotep ~]$ df
>>> Filesystem           1K-blocks      Used Available Use% Mounted on
>>> /dev/sdb1            384578164 330445976  34596748  91% 
>>> /home/fedora_8/torrents_isos
>>>
>>> /dev/sdb1 is the drive under discussion.
>>>
>>>
>>> [fedora_8 at Nyarlethotep ~]$ du -sb t*s/*
>>> 34256010522     torrents_isos/backup
>>> 883393808812    torrents_isos/data
>>> 58352749159     torrents_isos/finished
> <snip>
>>> somethings wrong with t*s/data  ....
>>>
>>
>> OK....  I believe I know what the problem is.  The torrents_isos/porn 
>> directory makes things seem larger than what they really are....
>>
>> No, just kidding.....
>>
>> I believe you may have a bunch of non-completed torrent downloads.  
>> When you start a torrent download the client will reserve the space 
>> and it will be reflected in the output of "du" but *not* in the output 
>> of "df".  Thus with "du" you can have a situation where it "thinks" 
>> more disk space is being used than it actually is.  FWIW, this is normal.
>>
> 
> I think Ed is on the right track, but the strange thing is the reserve 
> would happen sparse...
> cd /dev/shm/
> mkdir normal sparse
> dd if=/dev/zero of=normal/bunchozeros bs=1M count=20
> cp --sparse=always normal/bunchozeros sparse/
> 
> ls -l normal/* sparse/*
> -rw-r--r-- 1 root root 20971520 2008-04-01 09:28 normal/bunchozeros
> -rw-r--r-- 1 root root 20971520 2008-04-01 09:30 sparse/bunchozeros
> 
> du -sb *
> 20971580        normal
> 20971580        sparse
> 
> du -s *
> 20504   normal
> 0       sparse
> 
> df -h .
> Filesystem            Size  Used Avail Use% Mounted on
> tmpfs                1014M   21M  994M   2% /dev/shm
> 
> OK!!!
> df is showing how much disk is used/free based on inodes and filesizes 
> (you can run out of space if you have no inodes left).

Incorrect.  df reads out the free blocks on the filesytem, file sizes and inodes 
don't have anything to do with it.

The only way for du to differ from df (without a system issue) is for there to 
be files that have been deleted but are still being accessed, see below for 
things caused by system issues.

And if you run out of inodes it can show a lot of space being left free even 
though you don't have any inodes to be able to use it.

> du -s, du -sm show how much disk space is REALLY used on disk by the files.
> du -b implies --apparent-size
> du --apparent-size "print apparent sizes,  rather  than  disk  usage"
> 
All correct.


I have seen corrupt directory entries that indicate that blocks are being used 
(for a given file) that are not actually being used (or possibly don't even 
exist at all) cause du to return wrong results, one would need to look through 
the files in the funny directory and see if any of the sizes look wrong, given 
that they are certain kinds of files there is a certain range that should be 
valid, and an entry having something way outside of that range should be a hint 
that that directory entry is corrupted.    Often fsck will be unable to remove 
the entries, I have had to use filesystem debuggers to remove this type of file 
before, but even with those tools it is often difficult.   I would expect one or 
more files to be of a obviously wrong size.

If the systems crashes at exactly the wrong time, you can get some corrupted 
entries.

                               Roger