database mess up

[Panu Matilainen]

On Sat, 24 Jan 2009, Patrick Dupre wrote:

> On Fri, 23 Jan 2009, Panu Matilainen wrote:
>
>> On Thu, 22 Jan 2009, Patrick Dupre wrote:
>> 
>>>>> Hello,
>>>>> 
>>>>> For a reason that I ignore my database is totallt mess up.
>>>>> rpm --rebuilddb only rebuild iy partially.
>>>>> The packages are installed, but rpm --rebuilddb does not see them.
>>>>> How can I recover them without resintalling them manually ?
>>>> 
>>>> Find the latest intact /var/log/rpmpkgs* file (ie one that got generated 
>>>> before the db got corrupted, file size should be a good indicator) and 
>>>> copy it somewhere safe, say /root/rpmpkgs.backup. Now you should be able 
>>>> to make fairly good recovery with something like:
>>>> 
>>>> # mv /var/lib/rpm /var/lib/rpm.busted
>>>> # mkdir /var/tmp/download; cd /var/tmp/download
>>>> # yumdownloader `sed -e "s/.rpm$//g" /root/rpmpkgs.backup`
>>>> # rpm -Uvh --notriggers --noscripts --justdb *.rpm
>>>> 
>>>> The question of course is, what got the database corrupted to begin with.
>>>> Did anything out of the ordinary happen at that time, like /var getting 
>>>> full? Segfaults logged in /var/log/messages*? What filesystem is /var on?
>>>> 
>>> Hello Panu,
>>> 
>>> The larger rpmpkgs file is the following one:
>>> rpmpkgs-20090111
>>> from 2008-12-30
>>> The following one is only 462 block compared to 61748
>>> Concerning the message files, I attached the last one, I do not see 
>>> anything bad, the CPU0 temperature is 34 °C, so I do not thing that it is
>>> wrong. However the -12V and +12V are wrong according to gkrellM system 
>>> monitor. But is it right, I doubt that the machine would work with 0.63 
>>> and 3.95 V instead.
>>> Furthermore concerning the messages file, /messages-20090111 is empty
>>> as well as the following one: messages-20090118
>>> 
>>> /var is ext3 (on /)
>>> /usr, /usr/lib, /usr/local are lvm2
>>> 
>>> What do you thing ?
>> 
>> Nothing out of ordinary there.. what does 'stat -f /var' say on these 
>> problematic systems (as you said you have two systems with these problems)?
>
> It gives:
>
>  File: "/var"
>    ID: 590562c6b464a80b Namelen: 255     Type: ext2/ext3
> Block size: 1024       Fundamental block size: 1024
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Bingo...

> Blocks: Total: 3019460    Free: 1030413    Available: 876994
> Inodes: Total: 384000     Free: 356630
>
>> 
>> Noticed from the df output in another mail that the root partition was 
>> fairly small so it might be subject to a more or less known issue of 
>> filesystem blocksize of 1024 (at least on ext3) causing rpmdb corruption.
>> 
> do you think that blocksize of 1024 is bad ?

It's almost certainly the cause of the rpmdb mess you're seeing. This has 
been seen on both Fedora and Mandriva:
https://bugzilla.redhat.com/show_bug.cgi?id=181363
https://qa.mandriva.com/show_bug.cgi?id=32547

...and reformatting with a block size of 4096 or moving rpmdb to another 
fs with that block size is known to avoid the problems, caused by what is 
supposedly a kernel/fs bug.

 	- Panu -