[Crash-utility] linux_banner has garbage

[Dave Anderson]

----- Original Message -----
> Thank you for the guidance Dave.
> 
> I have two questions regarding runq.
> 
> 1. Could you please let me know how the active task has spent more time than
> uptime on some CPUs?

I'm not sure, other than the uptime is calculated based upon jiffies, and
the runq -m option uses each run queue's per-cpu timestamp.
 
> 
> crash> runq -m
> CPU 0: [0 00:23:29.808] PID: 529 TASK: ffff88079d0d1e40 COMMAND: "kworker/u141:1"
> CPU 1: [1 12:10:42.840] PID: 0 TASK: ffff88079df48000 COMMAND: "swapper/1"
> CPU 2: [1 12:10:42.841] PID: 0 TASK: ffff88079df4bc80 COMMAND: "swapper/2"
> CPU 3: [1 12:10:42.841] PID: 0 TASK: ffff88079df4dac0 COMMAND: "swapper/3"
> CPU 4: [1 12:10:42.841] PID: 0 TASK: ffff88079df49e40 COMMAND: "swapper/4"
> CPU 5: [1 12:10:42.841] PID: 0 TASK: ffff88079df58000 COMMAND: "swapper/5"
> 
> crash> sys
> KERNEL: ./usr/lib/debug/usr/lib/modules/4.14.19-coreos/vmlinux
> DUMPFILE: gt-user2-gmt-612746ca.vmss
> CPUS: 70
> DATE: Wed Feb 21 14:53:20 2018
> UPTIME: 1 days, 11:52:25
> LOAD AVERAGE: 70.70, 30.98, 12.88
> TASKS: 2312
> NODENAME: gt-user2-gmt.com
> RELEASE: 4.14.19-coreos
> VERSION: #1 SMP Wed Feb 14 03:18:05 UTC 2018
> MACHINE: x86_64 (2094 Mhz)
> MEMORY: 60 GB
> PANIC: ""
> crash>
>
> 2. Is there a way to:: find out why some CPUs have time lag in run queue ?

I don't know, but I would certainly look at the task backtraces of the cpus
that have the large lag values. 

> 
> CPU 32: 0.00 secs
> CPU 65: 0.00 secs
> CPU 54: 0.00 secs
> CPU 0: 0.01 secs
> CPU 16: 84.22 secs
> CPU 66: 268.75 secs
> CPU 58: 268.75 secs
> CPU 57: 268.75 secs
> CPU 43: 268.75 secs
> CPU 20: 268.75 secs
> CPU 7: 268.75 secs
> 
> crash>

> I'm struggling to find out why my VM hung(unresponsive to ping/ssh and couple
> of CPUs at 100% utilization).
> 
> -Eshak
> 
> On Thu, Feb 22, 2018 at 6:27 AM, Dave Anderson < anderson at redhat.com > wrote:
> 
> 
> ----- Original Message -----
> > Hello Dave,
> > 
> > I got a kernel freeze yesterday and am able to successfully open the memory
> > image using crash utility.
> > 
> > crash> sys
> > KERNEL: ./usr/lib/debug/usr/lib/modules/4.14.19-coreos/vmlinux
> > DUMPFILE: gt-Server02-gmt-612746ca.vmss
> > CPUS: 70
> > DATE: Wed Feb 21 14:53:20 2018
> > UPTIME: 1 days, 11:52:25
> > LOAD AVERAGE: 70.70, 30.98, 12.88
> > TASKS: 2312
> > NODENAME: gt-Server02-gmt.com
> > RELEASE: 4.14.19-coreos
> > VERSION: #1 SMP Wed Feb 14 03:18:05 UTC 2018
> > MACHINE: x86_64 (2094 Mhz)
> > MEMORY: 60 GB
> > PANIC: ""
> > crash>
> > 
> > Could you please guide me about couple of things I should check in case of
> > a kernel freeze before diving in deep to find the root cause ?
> 
> I'm not sure what you mean by a "kernel freeze", but typically something
> would complain about a hard or soft lockup in the system log. So I would
> first run "log" to see if there's anything of interest. Run "bt -a" on
> the active tasks to see if the active tasks are contesting for something,
> or work your way through "foreach bt" to see what the tasks of interest are
> doing/waiting on. It would seem that some task has taken control of
> something,
> a lock, or counter, or whatever, and many other tasks have blocked waiting
> for its release. So there's probably a common theme among the blocked tasks
> that might give you a clue.
> 
> Dave
> 
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
> 
> 
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility