Filesystem is busy after umount and won't fsck

[Jeremy Sanders]

This is on Fedora 16, kernel 3.3.7-1.fc16.x86_64. sdb1/3 are ext3
devices on a Hitachi HDS721010KLA330.

# fsck /dev/sdb1
fsck from util-linux 2.20.1
e2fsck 1.41.14 (22-Dec-2010)
fsck.ext3: Device or resource busy while trying to open /dev/sdb1
Filesystem mounted or opened exclusively by another program?
# fsck /dev/sdb3
fsck from util-linux 2.20.1
e2fsck 1.41.14 (22-Dec-2010)
fsck.ext3: Device or resource busy while trying to open /dev/sdb3
Filesystem mounted or opened exclusively by another program?
# umount /dev/sdb1
umount: /dev/sdb1: not mounted
# umount /dev/sdb3
umount: /dev/sdb3: not mounted
# grep sdb /proc/mounts

# mount /dev/sdb1 /mnt/tmp
# umount /mnt/tmp
# fsck /dev/sdb1
fsck from util-linux 2.20.1
e2fsck 1.41.14 (22-Dec-2010)
fsck.ext3: Device or resource busy while trying to open /dev/sdb1
Filesystem mounted or opened exclusively by another program?

# lsof|grep sdb
jbd2/sdb1  3835          root  cwd       DIR                8,1     4096
         2 /
jbd2/sdb1  3835          root  rtd       DIR                8,1
4096          2 /
jbd2/sdb1  3835          root  txt   unknown
          /proc/3835/exe
jbd2/sdb3  3858          root  cwd       DIR                8,1
4096          2 /
jbd2/sdb3  3858          root  rtd       DIR                8,1
4096          2 /
jbd2/sdb3  3858          root  txt unknown

/proc/3858/exe

Any idea what is going on? The device is usually fscked, then mounted
nightly, data is rsynced onto it, then unmounted. We'll probably upgrade
it to ext4 soon to see whether the problem goes away.

This has happened twice on this kernel (rebooted after it happened the
first time) and never before.

This is dumpe2fs on /dev/sdb1:
Filesystem volume name:   <none>
Last mounted on:          /mnt/root_backup
Filesystem UUID:          2aeb7822-f81a-4b84-a9fd-76c4c5c27bba
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index
filetype needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1835008
Block count:              3664820
Reserved block count:     183241
Free blocks:              1237712
Free inodes:              1545330
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      894
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16384
Inode blocks per group:   512
Filesystem created:       Thu May 22 15:09:53 2008
Last mount time:          Wed Jun 13 03:47:45 2012
Last write time:          Wed Jun 13 03:47:45 2012
Mount count:              5
Maximum mount count:      28
Last checked:             Sat Jun  9 09:37:27 2012
Check interval:           15552000 (6 months)
Next check after:         Thu Dec  6 08:37:27 2012
Lifetime writes:          87 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      706df78a-3ca8-46e8-a831-5f8ba43d2609
Journal backup:           inode blocks
Journal features:         journal_incompat_revoke
Journal size:             128M
Journal length:           32768
Journal sequence:         0x0011fbd9
Journal start:            1


Thanks

Jeremy