[Linux-cachefs] CacheFiles: Error: Unexpected object collision in 2.6.35.5

[Mark Moseley]

I've been running 2.6.35.5 with fscache on with pretty good results on
moderately loaded boxes. Moving to a more heavily loaded box, fscache
only lasted a couple of hours. I saw this same error a number of
months back. There's pages and pages more of soft lockup errors. The
box was pretty much unusable at this point.

This is a Dell Poweredge 1950, running Lenny 64-bit. The fscache
directory lives on its own disk, but it's a regular disk (waiting on
SSDs for these boxes), which may complicate things. One other
interesting thing is that the NFS server here is a Netapp GX cluster,
so there's only one explicit mount and the various submounts
underneath it happen in the background (i.e. there's only one entry in
fstab and then as the submounts get used, the new mount entries show
up in /proc/mounts).

Here's the relevant BUG. I included some 'preemptive burial' messages
too but I abbreviated them a bit, since there were hundreds. The
"Object already preemptively buried" was an error I don't recall
seeing before either, but only occurs a good hour prior to the BUG
message.


Sep 28 21:53:09 server kernel: [ 3364.596729] [kslowd] preemptive
burial: OBJ123e8 [OBJECT_RECYCLING] ffff88008c160900
... Lots more "preemptive burial" ...
Sep 28 21:53:36 server kernel: [ 3391.332790] CacheFiles: Error:
Object already preemptively buried
Sep 28 21:53:36 server kernel: [ 3391.333675] [kslowd] preemptive
burial: OBJ127d1 [OBJECT_RECYCLING] ffff88009d776780
... Lots more "preemptive burial" ...
Sep 28 21:53:36 server kernel: [ 3391.334814] [kslowd] preemptive
burial: OBJ127d4 [OBJECT_RECYCLING] ffff880094fb8a40
Sep 28 22:00:20 server kernel: [ 3795.025010] REISERFS warning (device
sdb3): jdm-20004 reiserfs_delete_xattrs: Couldn't delete all xattrs
(-2)
Sep 28 22:00:20 server kernel: [ 3795.025012]
Sep 28 22:05:35 server kernel: [ 4110.162196] [kslowd] preemptive
burial: OBJ16bd2 [OBJECT_RECYCLING] ffff880014b15500
... Lots more "preemptive burial" ...
Sep 28 22:05:38 server kernel: [ 4113.285955] [kslowd] preemptive
burial: OBJ16bb3 [OBJECT_RECYCLING] ffff88007dc94140
Sep 28 22:25:55 server kernel: [ 5330.667045] REISERFS warning (device
sdb3): jdm-20002 reiserfs_xattr_get: Invalid hash for xattr
(user.CacheFiles.cache) associated with [8891 170520 0x0 SD]
Sep 28 22:35:15 server kernel: [ 5890.668455] [kslowd] preemptive
burial: OBJ22648 [OBJECT_RECYCLING] ffff8800840bd440
... Lots more "preemptive burial" ...
Sep 28 22:35:27 server kernel: [ 5902.658162] [kslowd] preemptive
burial: OBJ22098 [OBJECT_RECYCLING] ffff8800186a7440
Sep 28 23:04:12 server kernel: [ 7625.014006]
Sep 28 23:04:12 server kernel: [ 7625.014010] CacheFiles: Error:
Unexpected object collision
Sep 28 23:04:12 server kernel: [ 7625.014015] object: OBJ2da67
Sep 28 23:04:12 server kernel: [ 7625.014018]
objstate=OBJECT_LOOKING_UP fl=0 swfl=a ev=0[7b]
Sep 28 23:04:12 server kernel: [ 7625.014021] ops=0 inp=0 exc=0
Sep 28 23:04:12 server kernel: [ 7625.014024] parent=ffff8800cab7fc00
Sep 28 23:04:12 server kernel: [ 7625.014028] cookie=ffff880046822c48
[pr=ffff88012d3e2f18 nd=ffff88000a15ac00 fl=7]
Sep 28 23:04:12 server kernel: [ 7625.014031] key=[44]
'000100001bb5008000000000f1157c01dd169a054dab0080000000
Sep 28 23:04:12 server kernel: [ 7625.014051] BUG: unable to handle
kernel NULL pointer dereference at 0000000000000040
Sep 28 23:04:12 server kernel: [ 7625.014057] IP: [<ffffffff8114db49>]
__fscache_read_or_alloc_pages+0x1ba/0x2f7
Sep 28 23:04:12 server kernel: [ 7625.014068] PGD 9db59067 PUD 65e79067 PMD 0
Sep 28 23:04:12 server kernel: [ 7625.014073] Oops: 0000 [#1] SMP
Sep 28 23:04:12 server kernel: [ 7625.014075] last sysfs file:
/sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
Sep 28 23:04:12 server kernel: [ 7625.014080] CPU 6
Sep 28 23:04:12 server kernel: [ 7625.014082] Modules linked in: dummy
xt_mac ipt_addrtype reiserfs xfs power_meter loop joydev fan
i5000_edac psmouse pcspkr evdev edac_core serio_raw tpm_tis shpchp tpm
i5k_amb tpm_bios pci_hotplug rng_core processor button dcdbas sd_mod
crc_t10dif sg sr_mod cdrom ide_pci_generic piix ide_core ata_generic
mptsas ata_piix mptscsih libata mptbase scsi_transport_sas ehci_hcd
uhci_hcd scsi_mod bnx2 thermal thermal_sys [last unloaded:
scsi_wait_scan]
Sep 28 23:04:12 server kernel: [ 7625.014116]
Sep 28 23:04:12 server kernel: [ 7625.014120] Pid: 30340, comm: httpd
Not tainted 2.6.35.5 #1 0DT097/PowerEdge 1950
Sep 28 23:04:12 server kernel: [ 7625.014123] RIP:
0010:[<ffffffff8114db49>]  [<ffffffff8114db49>]
__fscache_read_or_alloc_pages+0x1ba/0x2f7
Sep 28 23:04:12 server kernel: [ 7625.014129] RSP:
0018:ffff880045ab1a88  EFLAGS: 00010246
Sep 28 23:04:12 server kernel: [ 7625.014131] RAX: 0000000000000000
RBX: 0000000000000000 RCX: ffff880001d91140
Sep 28 23:04:12 server kernel: [ 7625.014134] RDX: 0000000000000000
RSI: ffff880001bd7628 RDI: ffff880001bd7628
Sep 28 23:04:12 server kernel: [ 7625.014137] RBP: ffff880045ab1ac8
R08: ffffffff8114ad15 R09: 0000000000000000
Sep 28 23:04:12 server kernel: [ 7625.014140] R10: 0000000000000af2
R11: ffff880001d83f08 R12: ffff880037b81c00
Sep 28 23:04:12 server kernel: [ 7625.014143] R13: ffff880046822c48
R14: ffff8800c3363b00 R15: ffff880045ab1c08
Sep 28 23:04:12 server kernel: [ 7625.014147] FS:
00000339b30406e0(0000) GS:ffff880001d80000(0000)
knlGS:0000000000000000
Sep 28 23:04:12 server kernel: [ 7625.014150] CS:  0010 DS: 0000 ES:
0000 CR0: 0000000080050033
Sep 28 23:04:12 server kernel: [ 7625.014153] CR2: 0000000000000040
CR3: 000000000185a000 CR4: 00000000000006f0
Sep 28 23:04:12 server kernel: [ 7625.014156] DR0: 0000000000000000
DR1: 0000000000000000 DR2: 0000000000000000
Sep 28 23:04:12 server kernel: [ 7625.014159] DR3: 0000000000000000
DR6: 00000000ffff0ff0 DR7: 0000000000000400
Sep 28 23:04:12 server kernel: [ 7625.014162] Process httpd (pid:
30340, threadinfo ffff880045ab0000, task ffff880040f52640)
Sep 28 23:04:12 server kernel: [ 7625.014164] Stack:
Sep 28 23:04:12 server kernel: [ 7625.014166]  ffff880045ab1b78
ffff880045ab1b34 ffff88000a15aed0 00000000ffffff97
Sep 28 23:04:12 server kernel: [ 7625.014170] <0> ffff88000a15adb8
0000000000000001 ffff880045ab1b34 ffff880045ab1c08
Sep 28 23:04:12 server kernel: [ 7625.014174] <0> ffff880045ab1b18
ffffffff811d6ea9 00000000000200da ffff88000a15aed0
Sep 28 23:04:12 server kernel: [ 7625.014178] Call Trace:
Sep 28 23:04:12 server kernel: [ 7625.014186]  [<ffffffff811d6ea9>]
__nfs_readpages_from_fscache+0x7f/0x148
Sep 28 23:04:12 server kernel: [ 7625.014191]  [<ffffffff811bd43a>]
nfs_readpages+0x108/0x1a7
Sep 28 23:04:12 server kernel: [ 7625.014198]  [<ffffffff810e32bb>] ?
alloc_pages_current+0xa8/0xd1
Sep 28 23:04:12 server kernel: [ 7625.014203]  [<ffffffff810bdb18>]
__do_page_cache_readahead+0x108/0x194
Sep 28 23:04:12 server kernel: [ 7625.014207]  [<ffffffff810bdbc0>]
ra_submit+0x1c/0x20
Sep 28 23:04:12 server kernel: [ 7625.014211]  [<ffffffff810bde50>]
ondemand_readahead+0x1b2/0x1c5
Sep 28 23:04:12 server kernel: [ 7625.014215]  [<ffffffff810bdf3d>]
page_cache_sync_readahead+0x38/0x3a
Sep 28 23:04:12 server kernel: [ 7625.014221]  [<ffffffff810b748a>]
generic_file_aio_read+0x27c/0x5ec
Sep 28 23:04:12 server kernel: [ 7625.014227]  [<ffffffff811b2dff>]
nfs_file_read+0xb1/0xde
Sep 28 23:04:12 server kernel: [ 7625.014232]  [<ffffffff810ed8bd>]
do_sync_read+0xc7/0x10d
Sep 28 23:04:12 server kernel: [ 7625.014238]  [<ffffffff81218600>] ?
security_file_permission+0x11/0x13
Sep 28 23:04:12 server kernel: [ 7625.014242]  [<ffffffff810ee519>]
vfs_read+0x11b/0x1e0
Sep 28 23:04:12 server kernel: [ 7625.014245]  [<ffffffff810ee6a2>]
sys_read+0x47/0x70
Sep 28 23:04:12 server kernel: [ 7625.014251]  [<ffffffff81002b1b>]
system_call_fastpath+0x16/0x1b
Sep 28 23:04:12 server kernel: [ 7625.014253] Code: 28 ff d0 48 c7 c1
bc ed b4 81 48 c7 c2 c4 ed b4 81 4c 89 f6 4c 89 e7 e8 a1 fb ff ff 89
c3 85 c0 0f 88 8c 00 00 00 49 8b 44 24 70 <f6> 40 40 04 74 43 f0 ff 05
56 13 a0 00 71 09 f0 ff 0d 4d 13 a0
Sep 28 23:04:12 server kernel: [ 7625.014277] RIP
[<ffffffff8114db49>] __fscache_read_or_alloc_pages+0x1ba/0x2f7
Sep 28 23:04:12 server kernel: [ 7625.014282]  RSP <ffff880045ab1a88>
Sep 28 23:04:12 server kernel: [ 7625.014284] CR2: 0000000000000040
Sep 28 23:04:12 server kernel: [ 7625.014774] ---[ end trace
4404c9c8f276dd3b ]---
Sep 28 23:04:12 server kernel: [ 7626.950525]
00400000009bdf51454000000000000000'
Sep 28 23:04:12 server kernel: [ 7626.978624] xobject: OBJ2da66
Sep 28 23:04:12 server kernel: [ 7626.996390] xobjstate=OBJECT_DYING
fl=0 swfl=8 ev=20[5]
Sep 28 23:04:18 server kernel: [ 7633.240556] xops=1 inp=1 exc=0
Sep 28 23:04:18 server kernel: [ 7633.258847] xparent=ffff8800cab7fc00
Sep 28 23:04:18 server kernel: [ 7633.280255] xcookie=NULL
Sep 28 23:04:20 server kernel: [ 7633.295456] ------------[ cut here
]------------
Sep 28 23:04:20 server kernel: [ 7633.305414] kernel BUG at
fs/cachefiles/namei.c:201!
Sep 28 23:04:20 server kernel: [ 7633.305414] invalid opcode: 0000 [#2] SMP
Sep 28 23:04:20 server kernel: [ 7633.305414] last sysfs file:
/sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
Sep 28 23:04:20 server kernel: [ 7633.305414] CPU 2
Sep 28 23:04:20 server kernel: [ 7633.305414] Modules linked in: dummy
xt_mac ipt_addrtype reiserfs xfs power_meter loop joydev fan
i5000_edac psmouse pcspkr evdev edac_core serio_raw tpm_tis shpchp tpm
i5k_amb tpm_bios pci_hotplug rng_core processor button dcdbas sd_mod
crc_t10dif sg sr_mod cdrom ide_pci_generic piix ide_core ata_generic
mptsas ata_piix mptscsih libata mptbase scsi_transport_sas ehci_hcd
uhci_hcd scsi_mod bnx2 thermal thermal_sys [last unloaded:
scsi_wait_scan]
Sep 28 23:04:20 server kernel: [ 7633.305414]
Sep 28 23:04:20 server kernel: [ 7633.305414] Pid: 6097, comm:
kslowd003 Tainted: G      D     2.6.35.5 #1 0DT097/PowerEdge 1950
Sep 28 23:04:20 server kernel: [ 7633.305414] RIP:
0010:[<ffffffff81204d85>]  [<ffffffff81204d85>]
cachefiles_walk_to_object+0x575/0x8e4
Sep 28 23:04:20 server kernel: [ 7633.305414] RSP:
0018:ffff8800ca30fcb0  EFLAGS: 00010282
Sep 28 23:04:20 server kernel: [ 7633.305414] RAX: 000000000000000f
RBX: 00000001000b2d55 RCX: 0000000000000005
Sep 28 23:04:20 server kernel: [ 7633.305414] RDX: 000000000000000e
RSI: ffff88004da53800 RDI: 0000003000000008
Sep 28 23:04:20 server kernel: [ 7633.305414] RBP: ffff8800ca30fd80
R08: 00000000003a7aff R09: 0000000000000002
Sep 28 23:04:20 server kernel: [ 7633.305414] R10: 000000000014a000
R11: ffff8800ca30f990 R12: ffff880037b81c00
Sep 28 23:04:20 server kernel: [ 7633.305414] R13: ffff88009ab73c00
R14: ffff880037b81d20 R15: ffff8800666ca6c0
Sep 28 23:04:20 server kernel: [ 7633.305414] FS:
0000000000000000(0000) GS:ffff880001c80000(0000)
knlGS:0000000000000000
Sep 28 23:04:20 server kernel: [ 7633.305414] CS:  0010 DS: 0000 ES:
0000 CR0: 000000008005003b
Sep 28 23:04:20 server kernel: [ 7633.305414] CR2: ffffffffff600400
CR3: 0000000001856000 CR4: 00000000000006f0
Sep 28 23:04:20 server kernel: [ 7633.305414] DR0: 0000000000000000
DR1: 0000000000000000 DR2: 0000000000000000
Sep 28 23:04:20 server kernel: [ 7633.305414] DR3: 0000000000000000
DR6: 00000000ffff0ff0 DR7: 0000000000000400
Sep 28 23:04:20 server kernel: [ 7633.305414] Process kslowd003 (pid:
6097, threadinfo ffff8800ca30e000, task ffff8800ca232440)
Sep 28 23:04:20 server kernel: [ 7633.305414] Stack:
Sep 28 23:04:20 server kernel: [ 7633.305414]  0000000000000100
ffff8800ca30fd38 ffff8800ca232440 ffff88004da53800
Sep 28 23:04:20 server kernel: [ 7633.305414] <0> ffff880065c257c4
ffff8801043e7e00 ffff8800b5400900 0000004100000001
Sep 28 23:04:20 server kernel: [ 7633.305414] <0> 0000000000000000
ffff8801043e7f10 ffff88009ab73c80 0000000000000000
Sep 28 23:04:20 server kernel: [ 7633.305414] Call Trace:
Sep 28 23:04:20 server kernel: [ 7633.305414]  [<ffffffff812035a1>]
cachefiles_lookup_object+0x77/0xd5
Sep 28 23:04:20 server kernel: [ 7633.305414]  [<ffffffff8114b13f>]
fscache_lookup_object+0x142/0x18e
Sep 28 23:04:20 server kernel: [ 7633.305414]  [<ffffffff8114b794>]
fscache_object_slow_work_execute+0x232/0x853
Sep 28 23:04:20 server kernel: [ 7633.305414]  [<ffffffff810ac50a>]
slow_work_execute+0x194/0x2af
Sep 28 23:04:20 server kernel: [ 7633.305414]  [<ffffffff8105b8db>] ?
finish_wait+0x5d/0x65
Sep 28 23:04:20 server kernel: [ 7633.305414]  [<ffffffff810ac7af>]
slow_work_thread+0x18a/0x2a8
Sep 28 23:04:20 server kernel: [ 7633.305414]  [<ffffffff8105b794>] ?
autoremove_wake_function+0x0/0x38
Sep 28 23:04:20 server kernel: [ 7633.305414]  [<ffffffff810ac625>] ?
slow_work_thread+0x0/0x2a8
Sep 28 23:04:20 server kernel: [ 7633.305414]  [<ffffffff8105b2ee>]
kthread+0x7d/0x85
Sep 28 23:04:20 server kernel: [ 7633.305414]  [<ffffffff81003914>]
kernel_thread_helper+0x4/0x10
Sep 28 23:04:20 server kernel: [ 7633.305414]  [<ffffffff8140b930>] ?
restore_args+0x0/0x30
Sep 28 23:04:20 server kernel: [ 7633.305414]  [<ffffffff8105b271>] ?
kthread+0x0/0x85
Sep 28 23:04:20 server kernel: [ 7633.305414]  [<ffffffff81003910>] ?
kernel_thread_helper+0x0/0x10
Sep 28 23:04:20 server kernel: [ 7633.305414] Code: 24 05 77 2b 48 c7
c7 ee 93 58 81 31 c0 e8 68 3d 20 00 48 c7 c7 d1 9b 59 81 31 c0 e8 5a
3d 20 00 4c 89 e6 4c 89 ef e8 29 f2 ff ff <0f> 0b eb fe f0 41 ff 84 24
10 01 00 00 71 0b f0 41 ff 8c 24 10
Sep 28 23:04:20 server kernel: [ 7633.305414] RIP
[<ffffffff81204d85>] cachefiles_walk_to_object+0x575/0x8e4
Sep 28 23:04:20 server kernel: [ 7633.305414]  RSP <ffff8800ca30fcb0>
Sep 28 23:04:20 server kernel: [ 7635.100156] ---[ end trace
4404c9c8f276dd3c ]---
Sep 28 23:05:16 server kernel: [ 7691.213124] BUG: soft lockup - CPU#5
stuck for 61s! [kslowd000:308]
Sep 28 23:05:17 server kernel: [ 7691.213126] Modules linked in: dummy
xt_mac ipt_addrtype reiserfs xfs power_meter loop joydev fan
i5000_edac psmouse pcspkr evdev edac_core serio_raw tpm_tis shpchp tpm
i5k_amb tpm_bios pci_hotplug rng_core processor button dcdbas sd_mod
crc_t10dif sg sr_mod cdrom ide_pci_generic piix ide_core ata_generic
mptsas ata_piix mptscsih libata mptbase scsi_transport_sas ehci_hcd
uhci_hcd scsi_mod bnx2 thermal thermal_sys [last unloaded:
scsi_wait_scan]
Sep 28 23:05:17 server kernel: [ 7691.213126] CPU 5
Sep 28 23:05:17 server kernel: [ 7691.213126] Modules linked in: dummy
xt_mac ipt_addrtype reiserfs xfs power_meter loop joydev fan
i5000_edac psmouse pcspkr evdev edac_core serio_raw tpm_tis shpchp tpm
i5k_amb tpm_bios pci_hotplug rng_core processor button dcdbas sd_mod
crc_t10dif sg sr_mod cdrom ide_pci_generic piix ide_core ata_generic
mptsas ata_piix mptscsih libata mptbase scsi_transport_sas ehci_hcd
uhci_hcd scsi_mod bnx2 thermal thermal_sys [last unloaded:
scsi_wait_scan]
Sep 28 23:05:17 server kernel: [ 7691.213126]
Sep 28 23:05:17 server kernel: [ 7691.213126] Pid: 308, comm:
kslowd000 Tainted: G      D     2.6.35.5 #1 0DT097/PowerEdge 1950
Sep 28 23:05:17 server kernel: [ 7691.213126] RIP:
0010:[<ffffffff8125044f>]  [<ffffffff8125044f>]
__write_lock_failed+0xf/0x20
Sep 28 23:05:17 server kernel: [ 7691.213126] RSP:
0018:ffff88012f20dc98  EFLAGS: 00000287
Sep 28 23:05:17 server kernel: [ 7691.213126] RAX: ffff88012f20dd38
RBX: ffff88012f20dca0 RCX: ffff8800c3163c00
Sep 28 23:05:17 server kernel: [ 7691.213126] RDX: ffff88012f3be3c0
RSI: ffff88012f20dc20 RDI: ffff8801043e7f18
Sep 28 23:05:17 server kernel: [ 7691.213126] RBP: ffffffff810034ce
R08: 0000000000000000 R09: ffff88008df6f003
Sep 28 23:05:17 server kernel: [ 7691.213126] R10: 0000000000000005
R11: 0000000000000000 R12: 0000000000000010
Sep 28 23:05:17 server kernel: [ 7691.213126] R13: 0000000000011140
R14: ffff88012f20c000 R15: 0000000000000000
Sep 28 23:05:17 server kernel: [ 7691.213126] FS:
0000000000000000(0000) GS:ffff880001d40000(0000)
knlGS:0000000000000000
Sep 28 23:05:17 server kernel: [ 7691.213126] CS:  0010 DS: 0000 ES:
0000 CR0: 000000008005003b
Sep 28 23:05:17 server kernel: [ 7691.213126] CR2: ffffffffff600400
CR3: 0000000001859000 CR4: 00000000000006f0
Sep 28 23:05:17 server kernel: [ 7691.213126] DR0: 0000000000000000
DR1: 0000000000000000 DR2: 0000000000000000
Sep 28 23:05:17 server kernel: [ 7691.213126] DR3: 0000000000000000
DR6: 00000000ffff0ff0 DR7: 0000000000000400
Sep 28 23:05:17 server kernel: [ 7691.213126] Process kslowd000 (pid:
308, threadinfo ffff88012f20c000, task ffff88012f3be3c0)
Sep 28 23:05:17 server kernel: [ 7691.213126] Stack:
Sep 28 23:05:17 server kernel: [ 7691.213126]  ffffffff8140b447
ffff88012f20dd80 ffffffff81204c76 0000000000000000
Sep 28 23:05:17 server kernel: [ 7691.213126] <0>
Sep 28 23:05:19 server kernel: [ 7691.213126]  ffff88012f20dd38
ffff88012f3be3c0 ffff8800c3163c00 ffff880045ade844
Sep 28 23:05:19 server kernel: [ 7691.213126] <0> ffff8801043e7e00
ffff88007e4f4b40 0000004100000000 0000000000000000
Sep 28 23:05:19 server kernel: [ 7691.213126] Call Trace:
Sep 28 23:05:19 server kernel: [ 7691.213126]  [<ffffffff8140b447>] ?
_raw_write_lock+0x16/0x18
Sep 28 23:05:19 server kernel: [ 7691.213126]  [<ffffffff81204c76>] ?
cachefiles_walk_to_object+0x466/0x8e4
Sep 28 23:05:19 server kernel: [ 7691.213126]  [<ffffffff812035a1>] ?
cachefiles_lookup_object+0x77/0xd5
Sep 28 23:05:19 server kernel: [ 7691.213126]  [<ffffffff8114b13f>] ?
fscache_lookup_object+0x142/0x18e
Sep 28 23:05:19 server kernel: [ 7691.213126]  [<ffffffff8114b794>] ?
fscache_object_slow_work_execute+0x232/0x853
Sep 28 23:05:19 server kernel: [ 7691.213126]  [<ffffffff810ac50a>] ?
slow_work_execute+0x194/0x2af
Sep 28 23:05:19 server kernel: [ 7691.213126]  [<ffffffff8105b8db>] ?
finish_wait+0x5d/0x65
Sep 28 23:05:19 server kernel: [ 7691.213126]  [<ffffffff810ac7af>] ?
slow_work_thread+0x18a/0x2a8
Sep 28 23:05:19 server kernel: [ 7691.213126]  [<ffffffff8105b794>] ?
autoremove_wake_function+0x0/0x38
Sep 28 23:05:19 server kernel: [ 7691.213126]  [<ffffffff810ac625>] ?
slow_work_thread+0x0/0x2a8
Sep 28 23:05:19 server kernel: [ 7691.213126]  [<ffffffff8105b2ee>] ?
kthread+0x7d/0x85
Sep 28 23:05:19 server kernel: [ 7691.213126]  [<ffffffff81003914>] ?
kernel_thread_helper+0x4/0x10
Sep 28 23:05:19 server kernel: [ 7691.213126]  [<ffffffff8140b930>] ?
restore_args+0x0/0x30
Sep 28 23:05:19 server kernel: [ 7691.213126]  [<ffffffff8105b271>] ?
kthread+0x0/0x85
Sep 28 23:05:19 server kernel: [ 7691.213126]  [<ffffffff81003910>] ?
kernel_thread_helper+0x0/0x10
Sep 28 23:05:19 server kernel: [ 7691.213126] Code: 48 83 eb 07 48 39
d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90
Sep 28 23:05:23 server kernel: [ 7691.213126] 90 90 f0 81 07 00 00 00
01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff
ff c3 90 f0 ff 07 f3
Sep 28 23:05:23 server kernel: [ 7691.213126] Call Trace:
Sep 28 23:05:23 server kernel: [ 7691.213126]  [<ffffffff8140b447>] ?
_raw_write_lock+0x16/0x18
Sep 28 23:05:23 server kernel: [ 7691.213126]  [<ffffffff81204c76>] ?
cachefiles_walk_to_object+0x466/0x8e4
Sep 28 23:05:23 server kernel: [ 7691.213126]  [<ffffffff812035a1>] ?
cachefiles_lookup_object+0x77/0xd5
Sep 28 23:05:23 server kernel: [ 7691.213126]  [<ffffffff8114b13f>] ?
fscache_lookup_object+0x142/0x18e
Sep 28 23:05:23 server kernel: [ 7691.213126]  [<ffffffff8114b794>] ?
fscache_object_slow_work_execute+0x232/0x853
Sep 28 23:05:23 server kernel: [ 7691.213126]  [<ffffffff810ac50a>] ?
slow_work_execute+0x194/0x2af
Sep 28 23:05:23 server kernel: [ 7691.213126]  [<ffffffff8105b8db>] ?
finish_wait+0x5d/0x65
Sep 28 23:05:23 server kernel: [ 7691.213126]  [<ffffffff810ac7af>] ?
slow_work_thread+0x18a/0x2a8
Sep 28 23:05:23 server kernel: [ 7691.213126]  [<ffffffff8105b794>] ?
autoremove_wake_function+0x0/0x38
Sep 28 23:05:23 server kernel: [ 7691.213126]  [<ffffffff810ac625>] ?
slow_work_thread+0x0/0x2a8
Sep 28 23:05:23 server kernel: [ 7691.213126]  [<ffffffff8105b2ee>] ?
kthread+0x7d/0x85
Sep 28 23:05:23 server kernel: [ 7691.213126]  [<ffffffff81003914>] ?
kernel_thread_helper+0x4/0x10
Sep 28 23:05:23 server kernel: [ 7691.213126]  [<ffffffff8140b930>] ?
restore_args+0x0/0x30
Sep 28 23:05:23 server kernel: [ 7691.213126]  [<ffffffff8105b271>] ?
kthread+0x0/0x85
Sep 28 23:05:23 server kernel: [ 7691.213126]  [<ffffffff81003910>] ?
kernel_thread_helper+0x0/0x10