<div dir="ltr"><div>I searched around for this a bit, and although other users may have hit this, I didn't find a good explanation offered. I suspect the users clean it up manually and then it disappears for another 2 years. I hope this message will get captured by Google, and help somebody else out. Also, I hope to have some discussion about this as it seems like an easily preventable problem.</div><div><br></div><div>The archive file names are generated like:<br></div><div><br></div><div><div><font face="monospace, monospace"> if (dm_snprintf(archive_name, sizeof(archive_name),</font></div><div><font face="monospace, monospace"> "%s/%s_%05u-%<a href="http://d.vg">d.vg</a>",</font></div><div><font face="monospace, monospace"> dir, vg->name, ix, rnum) < 0) {</font></div></div><div><br></div><div>The directory scanning code that loads the archive file names into memory recognizes a problem, although it isn't explicit about what the problem is:</div><div><br></div><div><div><font face="monospace, monospace"> /* Sort fails beyond 5-digit indexes */</font></div><div><font face="monospace, monospace"> if ((count = scandir(dir, &dirent, NULL, alphasort)) < 0) {</font></div><div><font face="monospace, monospace"> log_error("Couldn't scan the archive directory (%s).", dir);</font></div><div><font face="monospace, monospace"> return 0;</font></div><div><font face="monospace, monospace"> }</font></div><div><br></div></div><div>The file names encode the index like "00000". The sorting code uses "alphasort", which will only work properly as long as the index stays within 5 digits. As soon as it exceeds 5 digits, it begins to sort the "100000" to the beginning, and "99999" to the end. Then, new archives seems to *all* be "100000". We had some 40,000 indexes with "100000" before we noticed. And, because the index is followed by a random number, it would only expire a few of the "100000" before it would hit one that was younger than the 30 days retention period set by default. When I reduced the retention period to 7 days, it expired only about 12 archive files of 40,000 archive files. This behaviour is probably due to random number distribution ensuring that there are always some recent records near 0?</div><div><br></div><div>This issue eventually affects everyone, although obviously the people that use features like snapshots more frequently (we use it every 15 minutes, across multiple volumes) will hit it sooner, </div><div><br></div><div>There are a few fixes possible... Probably, "alphasort" should not be used at all, but a context aware sort should be used, that can filter and sort as it goes, decoding the index correctly as a number, and comparing it as a number. Then, if performance is desirable, and scalability, it would be ideal if it did it in a single pass, and buffering only the minimum needed to expire the correct archive files.</div><div><br></div><div>We hit this on RHEL 7.2. I wasn't surprised to find it in RHEL 7.2, but I was surprised that it still exists on "master". "git blame" says this has been an issue since 2002:</div><div><br></div><div><div><font face="monospace, monospace">5be981bab5 (Alasdair Kergon 2002-05-07 12:47:11 +0000 139) /* Sort fails beyond 5-digit indexes */</font></div><div><font face="monospace, monospace">59d6420b9a (Joe Thornber 2002-02-08 11:58:18 +0000 140) if ((count = scandir(dir, &dirent, NULL, alphasort)) < 0) {</font></div><div><font face="monospace, monospace">b8f47d5f69 (Alasdair Kergon 2009-07-15 20:02:46 +0000 141) log_error("Couldn't scan the archive directory (%s).", dir);</font></div><div><font face="monospace, monospace">952d12a5f5 (Alasdair Kergon 2002-01-09 19:16:48 +0000 142) return 0;</font></div><div><font face="monospace, monospace">952d12a5f5 (Alasdair Kergon 2002-01-09 19:16:48 +0000 143) }</font></div></div><div><br></div><div>Ouch... :-)</div><div><br></div><div>For anybody that does hit this.... Prune the archive files with index < 100000 is effective. It starts counting from 100000, and you now have 9X more life before it will happen again... :-)</div><div><br></div>-- <br><div class="gmail_signature">Mark Mielke <<a href="mailto:mark.mielke@gmail.com" target="_blank">mark.mielke@gmail.com</a>><br><br></div>
</div>