[linux-lvm] lvconvert --merge invalidating snapshot

[Luke Bigum]

Hello,

We're running in to an issue with merging snapshots and need some direction on how to debug further.  We have some jobs in Jenkins that are repeatedly creating and merging snapshots.  After several consecutive runs they fail inside the lvconvert command.  Sometimes this is 3 runs, sometimes it's hundreds, but eventually it will fail.  This is on CentOS 6.8 with lvm2-2.02.143-7.el6.x86_64, but I can reproduce the same problem when upgrading to lvm2-2.02.143-12.el6.x86_64 from 6.9. The error is:

  journal: Merging snapshot invalidated. Aborting merge.

Which is thrown by lvconvert during it's progress polling code when it gets back -1 (DM_PERCENT_INVALID) as the merge percentage.  The snapshot is healthy before merging, there are very few (if any) block changes in the LV when our Jenkins jobs are running, and the snapshot and source volume are quite small (few hundred MiB).  I've managed to reproduce the issue outside of the Jenkins tests with this Bash loop. It generally takes 15-30 minutes of running this to fail:


while [[ 1 ]]; do
  date
  /sbin/lvremove -f vg_os/journal_reserved_snap
  /sbin/lvcreate -s -n journal_snap -L 160.00m vg_os/journal
  /bin/umount /mnt
  /sbin/lvconvert --merge -i 5 vg_os/journal_snap
  /bin/mount /mnt
  /sbin/lvcreate -L 160.00m -n vg_os/journal_reserved_snap
  echo
done


When the merge fails, the snapshot is left in the merging state but invalid, and oddly 100% full:


# lvs --all
...
  [journal_snap]        vg_os Swi-I-s--- 160.00m      journal 100.00                                 


Device-Mapper says this about the merge:

May  5 10:45:55 lddev-build-scotty04 kernel: device-mapper: snapshots: Cancelling snapshot handover.
May  5 10:45:55 lddev-build-scotty04 kernel: device-mapper: snapshots: Snapshot is invalid: can't merge


And the *-cow and *-real DMs still exist:

# dmsetup ls | grep journal
vg_os-journal	(253:3)
vg_os-journal_snap	(253:6)
vg_os-journal_snap-cow	(253:5)
vg_os-journal-real	(253:4)


I can clean up the snapshot with 'lvremove' and start the process all over again.  I can also reproduce the problem on bare metal hardware and in a KVM instance (not that it should make a difference).

I'm at a bit of a loss on how to debug this any further.  I've done a little bit of experimenting with rolling back metadata changes to before the merge, but I don't really know what I'm looking for, and I generally always end up locking up Device-Mapper in some way and having to reboot :-)

Can anyone suggest a way forward here?

As an aside, while ruling out possible causes I tried the same Bash loop on a CentOS 6.8 machine with a non-standard kernel by accident. The result is different; it never fails to merge, but the LVM operations start really fast and get slower and slower overnight. When it would take less than a second to complete a loop first it was taking 30+ seconds in the end, and giving an interesting message about reserved memory. A resource leak?  I mention it just in case it's sheds light on the first problem, I don't really expect you to help me when using our custom kernel :-)

  Internal error: Reserved memory (15560704) not enough: used 25010176. Increase activation/reserved_memory?
  Logical volume "journal_reserved_snap" successfully removed

Thanks,

-- 
Luke Bigum
Lead Engineer

Information Systems
---

LMAX Exchange, Yellow Building, 1A Nicholas Road, London W11 4AN
http://www.LMAX.com/

Recognised by the most prestigious technology and business awards
 
Financial technology awards
2017 Best FX trading venue, Fund Technology & WSL Awards
2016 Best trading & execution venue, HFM US Technology Awards

FX industry awards
2016, 2015, 2014, 2013 Winner, Profit & Loss Readers' Choice Awards
2016, 2015, 2014, 2013 Winner, WSL Institutional Trading Awards

Business growth awards
2016, 2015 Winner, Deloitte UK Technology Fast 50
2015, 2014, 2013, Winner, The Sunday Times Tech Track 100
2016, 2015 Winner, Deloitte EMEA Technology Fast 500
2015 Winner, Tech City UK Future 50

---

FX and CFDs are leveraged products that can result in losses exceeding your deposit. They are not suitable for everyone so please ensure you fully understand the risks involved.

This message and its attachments are confidential, may not be disclosed or used by any person other than the addressee and are intended only for the named recipient(s). This message is not intended for any recipient(s) who based on their nationality, place of business, domicile or for any other reason, is/are subject to local laws or regulations which prohibit the provision of such products and services. This message is subject to the following terms (http://lmax.com/pdf/general-disclaimers.pdf), if you cannot access these, please notify us by replying to this email and we will send you the terms. If you are not the intended recipient, please notify the sender immediately and delete any copies of this message.

LMAX Exchange is the trading name of LMAX Limited. LMAX Limited operates a multilateral trading facility. LMAX Limited is authorised and regulated by the Financial Conduct Authority (firm registration number 509778) and is a company registered in England and Wales (number 6505809).

LMAX Hong Kong Limited is a wholly-owned subsidiary of LMAX Limited. LMAX Hong Kong is licensed by the Securities and Futures Commission in Hong Kong to conduct Type 3 (leveraged foreign exchange trading) regulated activity with CE Number BDV088.