f11 ppc64 woes

Sat Jun 6 12:27:49 UTC 2009

I reproduced what jwb reported on the powerstation.
Mine is with F10 + updates userland, only the kernel seems to matter.
The test case is:

	# modprobe iscsi_tcp
	Illegal Instruction
	#

On -16[278], same oops that jwb saw, wrong text appearing at a page boundary.

This kernel:
http://kojipkgs.fedoraproject.org/scratch/roland/task_1396640/kernel-vanilla-2.6.29.4-168.fc11.ppc64.rpm

does not exhibit the problem.  That should be all the same buildroot stuff,
and 2.6.29.4 with no extra patches.

OTOH, this kernel:
http://kojipkgs.fedoraproject.org/scratch/roland/task_1396192/kernel-2.6.29.4-167.fc10.ppc64.rpm

also does not exibit the problem.  That is normal -167 with all the same
patches, but built in dist-f10-updates-candidate buildroots.

But contrary to jwb's reports:
On my powerstation 2.6.29.3-159.fc11.ppc64 fails to boot:

    Loading ipr moduipr 0001:01:01.0: IOA initialized.
    le
    scsi0 : IBM 572C Storage Adapter
    scsi 0:0:0:0: Direct-Access     IBM-ESXS ST373455SS       BA23 PQ: 0 ANSI: 5
    Oops: Exception in kernel mode, sig: 4 [#1]
    SMP NR_CPUS=128 NUMA Maple
    Modules linked in: ipr(+)
    NIP: c000000000400000 LR: c0000000003ffff8 CTR: c00000000044304c
    REGS: c0000000781a7620 TRAP: 0700   Not tainted  (2.6.29.3-159.fc11.ppc64)
    MSR: 9000000000089032 <EE,ME,IR,DR>  CR: 24000024  XER: 000fffff
    TASK = c000000078193500[70] 'scsi_scan_0' THREAD: c0000000781a4000 CPU: 2
    GPR00: c0000000003ffff8 c0000000781a78a0 c000000000e6c210 c000000000db0390 
    GPR04: 0000000000000058 c00000007be6e000 000000000000000a ffffffffffffffff 
    GPR08: 0000000000000000 c0000000781a4000 c00000007be6e618 0000000000000000 
    GPR12: 0000000044000028 c000000000ea2800 000000000021a344 000000000021a394 
    GPR16: 000000000021a368 0000000000000000 c00000007be6e000 c0000000781a7b50 
    GPR20: c000000078090800 0000000000000000 0000000000000000 c00000007be6e060 
    GPR24: c000000078090828 0000000000000000 0000000000000000 fffffffffffffffa 
    GPR28: c000000078400120 0000000000000000 c000000000e0d710 c0000000781a7810 
    NIP [c000000000400000] .transport_destroy_device+0x3c/0x50
    LR [c0000000003ffff8] .transport_destroy_device+0x34/0x50
    Call Trace:
    [c0000000781a78a0] [c000000000446c9c] .scsi_alloc_sdev+0x220/0x284 (unreliable)
    [c0000000781a7950] [c000000000446f8c] .scsi_probe_and_add_lun+0x168/0xda8
    [c0000000781a7ad0] [c000000000447fb0] .__scsi_scan_target+0x104/0x730
    [c0000000781a7c20] [c000000000448654] .scsi_scan_channel+0x78/0xe8
    [c0000000781a7ce0] [c0000000004489d4] .scsi_scan_host_selected+0x11c/0x1a8
    [c0000000781a7da0] [c000000000448b48] .do_scsi_scan_host+0xe8/0x10c
    [c0000000781a7e40] [c000000000448bb0] .do_scan_async+0x44/0x220
    [c0000000781a7ef0] [c0000000000c57f8] .kthread+0x90/0xe4
    [c0000000781a7f90] [c00000000002f730] .kernel_thread+0x54/0x70
    Instruction dump:
    fbe1fff8 f821ff71 7c3f0b78 ebc2d0f0 f87f0070 60000000 60000000 e87f0070 
    e89e8000 4bfff8cd 60000000 383f0090 <00001010> 00000008 00001013 0000000f 
    ---[ end trace 998c8eb2fd0a3b41 ]---
    usb 2-1.4: new full speed USB device using ohci_hcd and address 3
    usb 2-1.4: New USB device found, idVendor=05ac, idProduct=1003

It prints some more usb probe msgs, but nothing else.
I can then reboot it with Ctl-Alt-Del on the USB keyboard.

This is obviously a variant of the same problem.  
It's losing on clobbered instructions at a page boundary.

Same for -158.
Same for -157.
Same for -155.
Same for -154.

Man but these bastards boot slow.

Same for -152.

-142 does not do that, nor exhibit the problem in "modprobe iscsi_tcp".

buildroot diffs -142 vs -152:

	@@ -23,8 +23,8 @@
	 device-mapper-1.02.31-4.fc11.ppc64
	 device-mapper-libs-1.02.31-4.fc11.ppc64
	 diffutils-2.8.1-23.fc11.ppc64
	-e2fsprogs-1.41.4-8.fc11.ppc64
	-e2fsprogs-libs-1.41.4-8.fc11.ppc64
	+e2fsprogs-1.41.4-9.fc11.ppc64
	+e2fsprogs-libs-1.41.4-9.fc11.ppc64
	 elfutils-0.140-2.fc11.ppc64
	 elfutils-libelf-0.140-2.fc11.ppc64
	 elfutils-libs-0.140-2.fc11.ppc64
	@@ -89,7 +89,7 @@
	 nss-3.12.3-4.fc11.ppc64
	 nss-softokn-freebl-3.12.3-4.fc11.ppc64
	 openldap-2.4.15-3.fc11.ppc64
	-openssl-0.9.8k-1.fc11.ppc64
	+openssl-0.9.8k-4.fc11.ppc64
	 pam-1.0.91-6.fc11.ppc64
	 patch-2.5.4-38.fc11.ppc64
	 pcre-7.8-2.fc11.ppc64
	@@ -102,7 +102,7 @@
	 pkgconfig-0.23-8.fc11.ppc64
	 policycoreutils-2.0.62-12.2.fc11.ppc64
	 popt-1.13-5.fc11.ppc64
	-ppl-0.10.1-1.fc11.ppc64
	+ppl-0.10.2-2.fc11.ppc64
	 procps-3.2.7-27.fc11.ppc64
	 psmisc-22.6-9.fc11.ppc64
	 readline-5.2-14.fc11.ppc64

ppl is a library used by gcc.  I eyeballed 0.10.1->0.10.2 changes
and I doubt it's involved (AFAIK no actual code changes there).

http://koji.fedoraproject.org/koji/taskinfo?taskID=1396771
is a scratch build of -142 in the current buildroots.
I'll go to sleep before it finishes, try it tomorrow.

I also tried:
koji build --nowait --scratch --arch-override=ppc64 --repo-id=72872 dist-f11 'cvs://cvs.fedoraproject.org/cvs/pkgs?rpms/kernel/F-11#kernel-2_6_29_3-152_fc11'

but:
	BuildError: Bad repo: 72872 (DELETED)

That would have been -152 but built in the buildroot -142 was built in.
I guess we'd need a temporary tag or something to recover the right repo
to do that build in koji.

lbr () 
{ 
    koji rpminfo $1 | awk '$1 == "Buildroot:" { print $2 }' | xargs -n1 koji list-buildroot
}

diff -u <(lbr kernel-2.6.29.3-142.fc11.ppc64) <(koji list-buildroot 470988)

for the buildroot diffs from -142 to the scratch -142 (current buildroot).

Now you know what I know and we still know nothing.

Oh, and note the two variant crashes in different kernels are in different
routines in different builds, but always at PC 0xc000000000400000,
and always clobbered the next few words with:
	00001010 00000008 00001013 0000000f 

The magic PAGE_OFFSET+4MB effect.  So, youse gots to wonder, and...

On 2.6.29.3-142.fc11.ppc64, which has "no problem", I built the appended module.
It printed this:

Instruction dump:
e8090000 f8410028 7f83e378 e9690010 7fa5eb78 7c0903a6 e8490008 4e800421 
<00001010> 00000008 00001013 0000000f 7961626f 6f740000 00101600 00000c00 
				      ^^^^^^^^ ^^^^ <-- spells "yaboot"
00000400 00101100 00000800 7fa3eb78 4bfff24d 60000000 38600000 383f00b0 
	 	  	   ^^^^^^^^ <-- goes to correct text again from here

The magic 44 bytes of bogon at PAGE_OFFSET+4MB effect.
We have no idea how long we have been screwed.

I updated to yaboot-1.3.14-12.fc11.ppc (was f10), ran ybin, no help.

So long and thanks for all the geese,
Roland

======
#include <linux/module.h>
#include <asm/ptrace.h>
#include <asm/uaccess.h>

MODULE_DESCRIPTION("fmh");
MODULE_LICENSE("GPL");

int ninsn = 24;
module_param(ninsn, int, 0);

static void __exit exit_fmh(void)
{
}

static int __init init_fmh(void)
{
	int i;
	unsigned long pc = 0xc000000000400000 - 8*4;

	printk(KERN_ERR "Instruction dump:");

	for (i = 0; i < ninsn; i++) {
		int instr;

		if (!(i % 8))
			printk("\n");

		/* We use __get_user here *only* to avoid an OOPS on a
		 * bad address because the pc *should* only be a
		 * kernel address.
		 */
		if (
		     __get_user(instr, (unsigned int __user *)pc)) {
			printk("XXXXXXXX ");
		} else {
			if (0xc000000000400000 == pc)
				printk("<%08x> ", instr);
			else
				printk("%08x ", instr);
		}

		pc += sizeof(int);
	}

	printk("\n");

	return -EAGAIN;
}

module_init(init_fmh);
module_exit(exit_fmh);