<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word"> <div class="WordSection1"> Hello,<o:p></o:p> <o:p> </o:p> As the current maintainer of the PyKdump extension, I have been investigating a reported discrepancy between the timestamps generated by crash’s log -T and PyKdump’s tslog command, which provides the same functionality of timestamping the kernel log entries. What I have found is that *both* log -T and tslog generate some inaccurate timestamps on vmcores with more than a small amount of uptime, but in different ways. Specifically, log -T timestamps are accurate at boot time but become less accurate later in the log, while tslog timestamps are just the opposite: they’re accurate around crash time but less accurate earlier in the log.<o:p></o:p> <o:p> </o:p> Here’s an example from a panic on a kernel 3.10.0-957.5.1 system with a moderate uptime:<o:p></o:p> <o:p> </o:p> DATE: Tue Jul 6 09:50:58 +03 2021<o:p></o:p> UPTIME: 32 days, 16:29:29<o:p></o:p> PANIC: "BUG: unable to handle kernel "<o:p></o:p> <o:p> </o:p> Crash correctly computes the boot time based on the date (crash time) and uptime shown above. From “help -k”:<o:p></o:p> <o:p> </o:p> boot_date: Thu Jun 3 17:21:29 +03 2021<o:p></o:p> <o:p> </o:p> log -T shows the correct timestamps of the boot messages:<o:p></o:p> <o:p> </o:p> [Thu Jun 3 17:21:29 +03 2021] Initializing cgroup subsys cpuset<o:p></o:p> [Thu Jun 3 17:21:29 +03 2021] Initializing cgroup subsys cpu<o:p></o:p> [Thu Jun 3 17:21:29 +03 2021] Initializing cgroup subsys cpuacct<o:p></o:p> <o:p> </o:p> However, the timestamps of the BUG and surrounding messages are off by 150 seconds or 2.5 minutes (09:53:28, but the crash occurred at 09:50:58):<o:p></o:p> <o:p> </o:p> [Tue Jul 6 09:53:28 +03 2021] BUG: unable to handle kernel<o:p></o:p> [Tue Jul 6 09:53:28 +03 2021] NULL pointer dereference at (null)<o:p></o:p> [Tue Jul 6 09:53:28 +03 2021] IP: [<ffffffff91cd6e66>] scsi_uninit_cmd+0x26/0x40<o:p></o:p> <o:p> </o:p> For comparison, tslog has the correct timestamps around the crash:<o:p></o:p> <o:p> </o:p> 2021-07-06 09:50:58 [2824319.188837] BUG: unable to handle kernel<o:p></o:p> 2021-07-06 09:50:58 [2824319.193440] NULL pointer dereference at (null)<o:p></o:p> 2021-07-06 09:50:58 [2824319.197853] IP: [<ffffffff91cd6e66>] scsi_uninit_cmd+0x26/0x40<o:p></o:p> <o:p> </o:p> But it’s off by the same 2.5-minute interval during the boot messages (17:18:59, but it should be 17:21:29):<o:p></o:p> <o:p> </o:p> 2021-06-03 17:18:59 [ 0.000000] Initializing cgroup subsys cpuset<o:p></o:p> 2021-06-03 17:18:59 [ 0.000000] Initializing cgroup subsys cpu<o:p></o:p> 2021-06-03 17:18:59 [ 0.000000] Initializing cgroup subsys cpuacct<o:p></o:p> <o:p> </o:p> The reasons for these inaccuracies are:<o:p></o:p> <o:p> </o:p> <![if !supportLists]>1. <![endif]>log -T computes boot time as (crash time minus uptime), and then adds the seconds part of the time stored in each log entry to generate the entry’s timestamp.<o:p></o:p> <o:p> </o:p> <![if !supportLists]>2. <![endif]>tslog computes a base time as (crash time minus seconds part of the final log entry) and then adds the seconds part of the time stored in each log entry.<o:p></o:p> <o:p> </o:p> <![if !supportLists]>3. <![endif]>The key to the problem: printk() stores the log entry timestamp based on a call to local_clock(), which provides the raw time in the local CPU, which is not adjusted by NTP. As such, inaccuracy accumulates as the clock drifts over time; in the example above, the discrepancy has reached 150 seconds by the time of the crash. <o:p></o:p> <o:p> </o:p> I have seen this now in numerous vmcores from kernels 3.0 through 4.12, which is the latest version I’ve checked. The discrepancy exists in all of them with more than a few hours of uptime and can be positive or negative. In one vmcore it was over 13 hours after 239 days of uptime.<o:p></o:p> <o:p> </o:p> From the PyKdump perspective, I could live with the current tslog behavior because it’s accurate for log entries near the crash time, which are generally the most useful for crash analysis. But in an attempt to improve it, I’m testing an experimental version of tslog that smooths out the discrepancy. It computes the boot time the same way that crash does (crash time minus uptime) and a scale factor for the time discrepancy equal to (uptime)/(seconds part of final log entry). Then each log entry’s timestamp is computed as (boot time) + (seconds part of log entry)*(scale factor). This version results in correct timestamps at both ends of the log:<o:p></o:p> <o:p> </o:p> 2021-06-03 17:21:29 [ 0.000000] Initializing cgroup subsys cpuset<o:p></o:p> 2021-06-03 17:21:29 [ 0.000000] Initializing cgroup subsys cpu<o:p></o:p> 2021-06-03 17:21:29 [ 0.000000] Initializing cgroup subsys cpuacct<o:p></o:p> :<o:p></o:p> 2021-07-06 09:50:58 [2824319.188837] BUG: unable to handle kernel<o:p></o:p> 2021-07-06 09:50:58 [2824319.193440] NULL pointer dereference at (null)<o:p></o:p> 2021-07-06 09:50:58 [2824319.197853] IP: [<ffffffff91cd6e66>] scsi_uninit_cmd+0x26/0x40<o:p></o:p> <o:p> </o:p> This is something of a hack since there’s no guarantee that the clock will drift uniformly during the system uptime, so timestamps in the middle of the log may still be somewhat inaccurate (but probably more accurate than without this adjustment).<o:p></o:p> <o:p> </o:p> Regards,<o:p></o:p> Martin<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Explore the HPE digital support experience: <a href="https://support.hpe.com"> https://support.hpe.com</a><o:p></o:p> <o:p> </o:p> Martin Moore Linux Engineering Resolution Team<o:p></o:p> HPE Pointnext Services Hewlett Packard Enterprise <a href="mailto:Martin.Moore@hpe.com">Martin.Moore@hpe.com</a> 8AM-5PM EDT (UTC-4) Monday-Friday Manager: Becky McBride (<a href="mailto:becky.mcbride@hpe.com">becky.mcbride@hpe.com</a>)<o:p></o:p> <o:p> </o:p> </div> </body> </html>