[Freeipa-users] disaster recovery

Robert Story rstory at tislabs.com
Mon Jun 27 12:59:14 UTC 2016 

On Mon, 27 Jun 2016 08:09:59 +0200 Martin wrote:
MB> On 26.06.2016 08:17, Robert Story wrote:
MB> > Hello,
MB> >
MB> > I was running a single ipa instance on Centos 7 for a small lab
MB> > (ipa-server-4.2.0-15.0.1.el7.centos.17.x86_64), and the disk was corrupted.
MB> > I have a (mostly) full backup (/var/log/ and /var/run/ excluded), which I
MB> > restored. ipa server didn't start, and wanted me to run
MB> > ipa-server-upgrade. This failed, and I see this in the log:
MB> >
MB> > 2016-06-25T23:16:37Z DEBUG Mounting ipaserver.rpcserver.jsonserver_kerb() at '/json'
MB> > 2016-06-25T23:16:37Z DEBUG session_auth_duration: 0:20:00
MB> > 2016-06-25T23:16:37Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
MB> > 2016-06-25T23:16:37Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute
MB> >      return_value = self.run()
MB> >    File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 47, in run
MB> >      server.upgrade_check(self.options)
MB> >    File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1573, in upgrade_check
MB> >      sys.exit(1)
MB> >
MB> > 2016-06-25T23:16:37Z DEBUG The ipa-server-upgrade command failed, exception: SystemExit: 1
MB> >
MB> >
MB> > I tried starting dirsrv at DOMAIN manually, and I get thisin the dirsrv log:
MB> >
MB> >
MB> > [26/Jun/2016:01:46:54 -0400] - 389-Directory/1.3.4.0 B2016.175.1716 starting up
MB> > [26/Jun/2016:01:46:54 -0400] - WARNING: changelog: entry cache size 2097152B is less than db size 143196160B; We recommend to increase the entry cache size nsslapd-cachememsize.
MB> > [26/Jun/2016:01:46:54 -0400] - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
MB> > [26/Jun/2016:01:46:55 -0400] - libdb: BDB2506 file userRoot/id2entry.db has LSN 4336/2969724, past end of log at 1/176
MB> > [26/Jun/2016:01:46:56 -0400] - libdb: BDB2507 Commonly caused by moving a database from one database environment
MB> > [26/Jun/2016:01:46:56 -0400] - libdb: BDB2508 to another without clearing the database LSNs, or by removing all of
MB> > [26/Jun/2016:01:46:56 -0400] - libdb: BDB2509 the log files from a database environment
MB> > [26/Jun/2016:01:46:57 -0400] - dbp->open("userRoot/id2entry.db") failed: Invalid argument (22)
MB> > [26/Jun/2016:01:46:57 -0400] - dblayer_instance_start fail: Invalid argument (22)
MB> > [26/Jun/2016:01:46:57 -0400] - libdb: BDB2506 file ipaca/id2entry.db has LSN 4336/2990140, past end of log at 1/288
MB> > [26/Jun/2016:01:46:57 -0400] - libdb: BDB2507 Commonly caused by moving a database from one database environment
MB> > [26/Jun/2016:01:46:57 -0400] - libdb: BDB2508 to another without clearing the database LSNs, or by removing all of
MB> > [26/Jun/2016:01:46:57 -0400] - libdb: BDB2509 the log files from a database environment
MB> > [26/Jun/2016:01:46:57 -0400] - dbp->open("ipaca/id2entry.db") failed: Invalid argument (22)
MB> > [26/Jun/2016:01:46:58 -0400] - dblayer_instance_start fail: Invalid argument (22)
MB> > [26/Jun/2016:01:46:58 -0400] - libdb: BDB2506 file changelog/id2entry.db has LSN 4336/2921967, past end of log at 1/288
MB> > [26/Jun/2016:01:46:58 -0400] - libdb: BDB2507 Commonly caused by moving a database from one database environment
MB> > [26/Jun/2016:01:46:58 -0400] - libdb: BDB2508 to another without clearing the database LSNs, or by removing all of
MB> > [26/Jun/2016:01:46:58 -0400] - libdb: BDB2509 the log files from a database environment
MB> > [26/Jun/2016:01:46:58 -0400] - dbp->open("changelog/id2entry.db") failed: Invalid argument (22)
MB> > [26/Jun/2016:01:46:58 -0400] - dblayer_instance_start fail: Invalid argument (22)
MB> > [26/Jun/2016:01:46:58 -0400] - start: Failed to start databases, err=22 Invalid argument
MB> >
MB> >
MB> > So I'm trying to figure out if I can salvage this restored VM, or if I need
MB> > to reinstall from scratch; and if I do reinstall, am I going to be able to
MB> > restore my old data somehow. I have a funny feeling that there are
MB> > important files in /var/log and/or /var/run and I'm up the creek without a
MB> > paddle.
MB> >
MB> > And yes, once I have a working system again I'm going to set up a replica
MB> > to help avoid this mess in the future.
MB> >
MB> > Robert
MB> >
MB> >
MB> >  
MB> 
MB> Hello, upgrader refuses to upgrade because check which requires 
MB> /var/lib/ipa  failed. Upgrader thinks that IPA is not installed.
MB> 
MB> So are you sure you have backup of /var/lib/ipa ?

Yep, /var/lib/ipa is there:

 ls -lR
.:
total 4
drwx------. 2 root root    6 Jun 24 08:10 backup
drwxr-xr-x. 3 root root   20 Jun 24 08:10 pki-ca
drwx------. 2 root root 4096 Jun 24 08:10 sysrestore
drwx------. 2 root root   29 Jun 24 08:10 sysupgrade

./backup:
total 0

./pki-ca:
total 0
drwxrwxr-x. 2 root pkiuser 26 Jun 25 19:38 publish

./pki-ca/publish:
total 0
lrwxrwxrwx. 1 pkiuser pkiuser 57 Jun 24 21:00 MasterCRL.bin -> /var/lib/ipa/pki-ca/publish/MasterCRL-20160624-210000.der

./sysrestore:
total 68
-rw-r--r--. 1 root root   14 Sep 15  2015 07b33009095935b8-krb5kdc
-rw-r--r--. 1 root root  495 Sep 15  2015 126a0615510e0df6-krb5.conf
-rw-r--r--. 1 root root 2045 Aug  5  2015 1459a73f06d5e29c-dirsrv
-rw-r--r--. 1 root root   45 Jun 23  2015 1bc4913116370139-ntpd
-rw-r--r--. 1 root root 9534 Mar  5  2015 1d4cccdbe2db6338-nss.conf
-rw-r--r--. 1 root root  158 Jun  7  2013 33ef02044e7e32c4-hosts
-rw-r--r--. 1 root root 2045 Feb 17 08:37 3ab32f97ac1f896a-dirsrv
-rw-r--r--. 1 root root 2045 Aug  5  2015 7d1b4474370581db-dirsrv
-rw-r--r--. 1 root root 2045 Sep 21  2015 b3a9575e954a66ff-dirsrv
-rw-r--r--. 1 root root 1984 Aug 19  2015 cdfa12db5eab40ef-ntp.conf
-rw-------. 1 root root  451 Sep 15  2015 d3df0140545921df-kdc.conf
-rw-r--r--. 1 root root 2045 Dec 15  2015 e41f8dd1839f3670-dirsrv
-rw-r--r--. 1 root root 2045 Mar 14 09:17 f656872d26e358ed-dirsrv
-rw-r--r--. 1 root root  757 Apr 14 07:30 sysrestore.index
-rw-r--r--. 1 root root  556 Jun 26 01:59 sysrestore.state

./sysupgrade:
total 4
-rw-r--r--. 1 root root 582 Apr 14 07:30 sysupgrade.state


Looking through the backups, I see that there are no MasterCRL files from
the 25th (the backup I restored), but a bunch from the 24th, so maybe I
need to try another restore with files from then...


Robert

-- 
Senior Software Engineer @ Parsons
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20160627/4817bd53/attachment.sig>

More information about the Freeipa-users mailing list