[Spacewalk-list] cannot boot after kernel update
Coffman, Anthony J
Tony.Coffman at snapon.com
Wed Jun 7 13:04:49 UTC 2017
We talked about this last September also [1]. I don't think anybody found a smoking gun root cause.
After re-reading that old thread, it reminded me that I also had a related issue last July where a bunch of C7 systems had the yum transaction abort (making a mess to clean up).
[1] https://www.redhat.com/archives/spacewalk-list/2016-September/msg00031.html
FWIW - I switched most of my C7 to Foreman/Katello in January and I don't think I've seen this issue crop up there yet.
--Tony
-----Original Message-----
From: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Ree, Jan-Albert van
Sent: Wednesday, June 7, 2017 3:15 AM
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] cannot boot after kernel update
The problem appears to be some sort of time-out We've seen the same behaviour if we push (ie select kernel packages and send out the update) If I use the 'remote command' option in spacewalk and do a 'yum -y update kernel' and set the script timeout high (7200s is what I use) things do work for us.
--
Jan-Albert
Jan-Albert van Ree | Linux System Administrator | MARIN Support Group MARIN | T +31 317 49 35 48 | mailto:J.A.v.Ree at marin.nl | http://www.marin.nl
MARIN news: http://www.marin.nl/web/News/News-items/Topsector-Water-wordt-Topsector-Water-en-Maritiem.htm
________________________________________
From: spacewalk-list-bounces at redhat.com <spacewalk-list-bounces at redhat.com> on behalf of Huber, Peter <peter.huber at uni-wh.de>
Sent: Wednesday, June 07, 2017 09:11
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] cannot boot after kernel update
Thanks and yes it is very weird. I don't think it is a problem with the space, only 35% used 477M 156M 292M 35% /boot
The root cause is that the ramdisk is not created if I update the kernel via spacewalk, initramfs-3.10.0-514.21.1.el7.x86_64.img was missing. I verified this on the two server I updated.
If I reboot into the previous kernel and then do a yum update, the ramdisk is created and everything is fine.
But in this situation I can't plan any update if there is a new kernel included, because I do know if the server is coming up after a reboot...
Regards,
Peter
-----Ursprüngliche Nachricht-----
Von: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-bounces at redhat.com] Im Auftrag von Coffman, Anthony J
Gesendet: Dienstag, 6. Juni 2017 22:03
An: spacewalk-list at redhat.com
Betreff: Re: [Spacewalk-list] cannot boot after kernel update
It's possible you might be low on space on the /boot partition. Yum or RPM disk requirement calculations aren't perfect and it can fail if you are right on the edge.
But I've seen similar behavior in the past 5-6 months when I wasn't low on space - seemingly at random except that they are all CentOS 7 systems. However, I have plenty of C7 systems that seem to work fine and there is no substantial difference between them.
My fix for this is to simply boot into the previous kernel and do a "yum reinstall kernel-3.10.0-514.21.1.el7.x86_64" and then reboot it one more time. It seems like the initrd may not be properly built after the kernel update.
I don't know what the root cause is - it seems like there is some kind of bug when updating the kernel that doesn't trigger consistently.
The other bug that I think might be related to the same root cause is that the default kernel isn't updated even though /etc/sysconfig/kernel is set to updatedefault. I catch this when I do post-patch auditing and fix it pretty much the same way. If you do a yum reinstall a second time - the default is updated correctly.
Very weird.
Regards,
--Tony
-----Original Message-----
From: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Huber, Peter
Sent: Tuesday, June 6, 2017 5:50 AM
To: spacewalk-list at redhat.com
Subject: [Spacewalk-list] cannot boot after kernel update
I have updated 2 centos hosts today. The following updates were installed:
firewalld-0.4.3.2-8.1.el7_3.3.noarch
libnetfilter_conntrack-1.0.6-1.el7_3.x86_64
sudo-1.8.6p7-22.el7_3.x86_64
dracut-config-rescue-033-463.el7_3.1.x86_64
nss-sysinit-3.28.4-1.2.el7_3.x86_64
yum-rhn-plugin-2.6.4-1.el7.noarch
NetworkManager-wifi-1.4.0-20.el7_3:1.x86_64
glibc-2.17-157.el7_3.2.x86_64
nss-tools-3.28.4-1.2.el7_3.x86_64
systemd-sysv-219-30.el7_3.9.x86_64
python-perf-3.10.0-514.21.1.el7.centos.plus.x86_64
systemd-219-30.el7_3.9.x86_64
kpartx-0.4.9-99.el7_3.3.x86_64
systemd-libs-219-30.el7_3.9.x86_64
firewalld-filesystem-0.4.3.2-8.1.el7_3.3.noarch
java-1.8.0-openjdk-devel-1.8.0.131-3.b12.el7_3:1.x86_64
NetworkManager-team-1.4.0-20.el7_3:1.x86_64
java-1.8.0-openjdk-headless-1.8.0.131-3.b12.el7_3:1.x86_64
polkit-0.112-12.el7_3.x86_64
systemd-python-219-30.el7_3.9.x86_64
tuned-2.7.1-3.el7_3.2.noarch
nss-3.28.4-1.2.el7_3.x86_64
rdma-7.3_4.7_rc2-6.el7_3.noarch
NetworkManager-1.4.0-20.el7_3:1.x86_64
NetworkManager-tui-1.4.0-20.el7_3:1.x86_64
dracut-033-463.el7_3.1.x86_64
java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3:1.x86_64
dracut-network-033-463.el7_3.1.x86_64
kernel-tools-3.10.0-514.21.1.el7.x86_64
libgudev1-219-30.el7_3.9.x86_64
python-firewall-0.4.3.2-8.1.el7_3.3.noarch
kernel-tools-libs-3.10.0-514.21.1.el7.x86_64
glibc-common-2.17-157.el7_3.2.x86_64
NetworkManager-libnm-1.4.0-20.el7_3:1.x86_64
kernel-3.10.0-514.21.1.el7.x86_64
after a reboot, both servers do not start anymore. There is a kernel panic - not syncing: VFS: Unable to mount root fs on unknown wm block(0,0)
Any idea what can cause this problem? Both servers got this kernel-update....
Regards
Peter
Private Universität Witten/Herdecke gGmbH Alfred-Herrhausen-Straße 50 D - 58448 Witten
Homepage: http://www.uni-wh.de
Twitter: http://twitter.com/UniWH
Facebook: http://www.facebook.com/UniWH
Geschäftsführung: Prof. Dr. Martin Butzlaff (Präsident), Dipl. oec. Jan Peter Nonnenkamp (Kanzler)
Sitz der Gesellschaft: Witten
Handelsregister des Amtsgerichts Bochum Nr. HRB 8671
_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list
_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list
Private Universität Witten/Herdecke gGmbH Alfred-Herrhausen-Straße 50 D - 58448 Witten
Homepage: http://www.uni-wh.de
Twitter: http://twitter.com/UniWH
Facebook: http://www.facebook.com/UniWH
Geschäftsführung: Prof. Dr. Martin Butzlaff (Präsident), Dipl. oec. Jan Peter Nonnenkamp (Kanzler)
Sitz der Gesellschaft: Witten
Handelsregister des Amtsgerichts Bochum Nr. HRB 8671
_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list
_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list
More information about the Spacewalk-list
mailing list