<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Miles,</div><div><br></div>It seems like what's probably happened is that LVM detected the raw device instead of the MD device at some point early in the boot process. This may be because the MD detection happened after LVM setup. I'm unsure if it's possible for LVM to "steal" the device from MD.<div><br></div><div>Depending on your distribution, this may require different things to fix. Stop worrying about downtime. If the data is important, just don't worry about downtime. If downtime is really important, build a second machine, get it working right, and transfer the data. Being in a hurry and attempting to "optimize" the recovery process is a really good way to lose the data.</div><div><br></div><div>Assuming that you're going to try to fix this setup, I'd start out with a backup. This is critical. Everybody always says to do a backup. Nobody ever does it. Really, do one. Get an S3 account, use an S3 backup utility. There's just not an excuse these days. Your data is one-MD-mistake away from oblivion.</div><div><br></div><div>So, right now MD should have sda/sdb but only has sda. sdb is now newer than sda and may have important data if this server stores anything like that. The challenge is that, according to MD, sda is newer. Since MD isn't handling writes to sdb, it won't be updating its metadata to know that it's newer. There are two options that I can think of, both ugly. Pick one of:</div><div><br></div><div>1. Destroy the MD. Create a new one with the same UUID and sdb3 as the source. (which you listed, the UUID part can trip you up)</div><div>2. Sync the updated data from sdb3 onto md2. Wipe sdb3. Add it back into md2. (might be less downtime depending on data size, doesn't nuke MD)</div><div>3. Build another machine. Get it working right. Transfer data with Rsync. (least downtime, most expensive)</div><div><br></div><div>In the first two cases, this only sets you up for it to break again. The core problem is figuring out what happened during boot. In a perfect world, you would just tell LVM to only consider MD devices. That's not hard, but it's complicated by the fact that you have LVM on /. This means that the configuration that's used is likely not the version on / but a copy of it that is made when you set up your boot ramdisk (a.k.a. initrd, or possibly an initramfs). Even if we get LVM locked down to use just MDs and get that config used to boot-time, there's the possibility that the MD won't get assembled (since it already may not have been when LVM was first activated) and the system won't boot. Again, fraught with peril.</div><div><br></div><div>If you want to fix the MD, first steps will be using a rescue LiveCD to boot up and do all of this. With that LiveCD, you can also adjust the LVM configuration and update the initrd (or whatever is used for boot). You may need to chroot into the system and/or trick the initrd into seeing the right devices. I don't really think I can walk you through this via an e-mail.</div><div><br></div><div>The LVM part is pretty easy. Just set a filter line (you only get one, so disable any other filter lines) in <root of system>/etc/lvm.conf to:</div><div><br></div><div><blockquote type="cite">filter = [ "a|^/dev/md.*$|", "r/.*/" ]</blockquote></div><div><br></div><div>That will prevent you from using anything but the MD.</div><div><br></div><div>To update the initrd with this information depends on distro (and distro version) . It's usually either some invocation of "mkinitrd" or some script that wraps it. It will get the LVM configuration available at boot-time. This *MIGHT* sort out the MD problem. It might not. If it doesn't, I'm not sure where to tell you to start. If mdadm is being used by your initrd, you'll need to tweak its configuration. If it's relying on MD autodetection, you might have turned that off in your kernel. If you have an IDE controller that takes too long to initialize, that can also cause this sort of thing (although that's REALLY unlikely these days).</div><div><br></div><div>I hope that some of this helps. Although, it will be hard for anyone to give you really solid advice without a little more insight into why the MD isn't getting assembled prior to LVM's scan.</div><div><br></div><div><div><div>On Apr 5, 2009, at 10:05 AM, Miles Fidelman wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Hello again Folks,<br><br>So.. I'm getting closer to fixing this messed up machine.<br><br>Where things stand:<br><br>I have root defined as an LVM2 LV, that should use /dev/md2 as it's PV.<br>/dev/md2 in turn is a RAID1 array built from /dev/sda3 /dev/sdb3 and /dev/sdc3<br><br>Instead, LVM is reporting: "Found duplicate PV 2ppSS2q0kO3t0tuf8t6S19qY3ypWBOxF: using /dev/sdb3 not /dev/sda3"<br>and the /dev/md2 is reporting itself as inactive (cat /proc/mdstat) and active,degraded (mdadm --detail)<br><br>---<br>I'm guessing that, during boot:<br><br>- the raid array failed to start<br>- LVM found both copies of the PV, and picked one (/dev/sdb3)<br>- everything then came up and my server is humming away<br><br>but: the md array can't rebuild because the most current device in it is already in use<br><br>so... I'm looking for the right sequence of events, with the minimum downtime to:<br><br>1. stop changes to /dev/sdb3 (actually, to / - which complicates things)<br>2. rebuild the RAID1 array, making sure to use /dev/sdb3 as the starting point for current data<br>3. restart in such a way that LVM finds /dev/md2 as the right PVM instead of one of its components<br><br>Each of these is just tricky enough that I'm sure there are lots of gotchas to watch out for.<br><br>So.. any suggestions?<br><br>Thanks very much,<br><br>Miles Fidelman<br><br><br><br><br>_______________________________________________<br>linux-lvm mailing list<br><a href="mailto:linux-lvm@redhat.com">linux-lvm@redhat.com</a><br>https://www.redhat.com/mailman/listinfo/linux-lvm<br>read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/<br></div></blockquote></div></div><br><div apple-content-edited="true"> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; -webkit-text-decorations-in-effect: none; text-indent: 0px; -webkit-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; -webkit-text-decorations-in-effect: none; text-indent: 0px; -webkit-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; -webkit-text-decorations-in-effect: none; text-indent: 0px; -webkit-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><div>-- </div><div>Jayson Vantuyl</div><div><font class="Apple-style-span" size="2"><span class="Apple-style-span" style="font-size: 10px; "><span class="Apple-style-span" style="font-size: 10px; "><span class="Apple-style-span" style="font-size: 10px; ">Founder and Architect</span></span></span></font></div><div><b style="font-weight: bold; "><a href="http://www.engineyard.com"><span class="Apple-style-span" style="font-weight: normal; color: rgb(0, 0, 238); -webkit-text-decorations-in-effect: underline; "><span class="Apple-style-span" style="color: rgb(0, 0, 238); -webkit-text-decorations-in-effect: underline; ">Engine Yard</span></span></a></b></div><div><font class="Apple-style-span" size="2"><span class="Apple-style-span" style="font-size: 10px; "><span class="Apple-style-span" style="font-size: 10px; "><span class="Apple-style-span" style="font-size: 10px; "><a href="mailto:jvantuyl@engineyard.com">jvantuyl@engineyard.com</a></span></span></span></font></div><div><font class="Apple-style-span" size="2"><span class="Apple-style-span" style="font-size: 10px; ">1 866 518 9275 ext 204</span></font></div><div><font class="Apple-style-span" size="2"><span class="Apple-style-span" style="font-size: 10px; ">IRC (freenode): kagato</span></font></div></span></span></div></span></div></span></div></span></div></span></div></span></div></span> </div><br></body></html>