Oracle/Red Hat/NetApp

Mon Aug 20 20:37:48 UTC 2007

On 20/08/07, Stephen Carville <scarville at landam.com> wrote:
> Troy Knabe wrote:
> > We have several Oracle instances 10.2 (some RAC, some not, but all are
> > having the same issue) all running on Red Hat 4.5.  The binaries and data
> > files are all on the NetApp and mounted via NFS.
> >
> >
> >
> > Sporadically when we shut the database down, we are unable to restart it.
> >
> >
> >
> >
> >
> >
> > /u1/app/oracle/admin/prodrac/bdump/prodrac1_dbw0_10023.trc
> >
> >
> >
> > Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit
> > Production
> >
> > With the Partitioning, Real Application Clusters, OLAP and Data Mining
> > options
> >
> > ORACLE_HOME = /u1/app/oracle/product/10.2.0
> >
> > System name:   Linux
> >
> > Node name:       orarac1
> >
> > Release:               2.6.9-55.0.2.ELsmp
> >
> > Version:               #1 SMP Tue Jun 12 17:58:20 EDT 2007
> >
> > Machine:             x86_64
> >
> > Instance name: prodrac1
> >
> > Redo thread mounted by this instance: 1
> >
> > Oracle process number: 8
> >
> > Unix process pid: 10023, image: oracle at orarac1 (DBW0)
> >
> >
> >
> > *** SERVICE NAME:() 2007-08-18 02:05:27.878
> >
> > *** SESSION ID:(1194.1) 2007-08-18 02:05:27.878
> >
> > ORA-01157: cannot identify/lock data file 1 - see DBWR trace file
> >
> > ORA-01110: data file 1: '/u1/oradata/prod/system/system01.dbf'
> >
> > ORA-27086: unable to lock file - already in use
> >
> > Linux-x86_64 Error: 11: Resource temporarily unavailable
> >
> > Additional information: 8
> >
> > ORA-01157: cannot identify/lock data file 2 - see DBWR trace file
> >
> > ORA-01110: data file 2: '/u1/oradata/prod/temp/undotbs01.dbf'
> >
> > ORA-27086: unable to lock file - already in use
> >
> > Linux-x86_64 Error: 11: Resource temporarily unavailable
> >
> > Additional information: 8
> >
> > ORA-01157: cannot identify/lock data file 3 - see DBWR trace file
> >
> > ORA-01110: data file 3: '/u1/oradata/prod/system/sysaux01.dbf'
> >
> > ORA-27086: unable to lock file - already in use
> >
> > Linux-x86_64 Error: 11: Resource temporarily unavailable
> >
> > Additional information: 8
> >
> > ORA-01157: cannot identify/lock data file 4 - see DBWR trace file
> >
> > ORA-01110: data file 4: '/u1/oradata/prod/table/users01.dbf'
> >
> > ORA-27086: unable to lock file - already in use
> >
> > Linux-x86_64 Error: 11: Resource temporarily unavailable
> >
> > Additional information: 8
> >
> > ORA-01157: cannot identify/lock data file 5 - see DBWR trace file
> >
> > ORA-01110: data file 5: '/u1/oradata/prod/temp/undotbs02.dbf'
> >
> > ORA-27086: unable to lock file - already in use
> >
> > Linux-x86_64 Error: 11: Resource temporarily unavailable
> >
> > Additional information: 8
> >
> > ORA-01157: cannot identify/lock data file 6 - see DBWR trace file
> >
> > ORA-01110: data file 6: '/u1/oradata/prod/temp/undotbs03.dbf'
> >
> > ORA-27086: unable to lock file - already in use
> >
> > Linux-x86_64 Error: 11: Resource temporarily unavailable
> >
> > Additional information: 8
> >
> > ORA-01157: cannot identify/lock data file 7 - see DBWR trace file
> >
> > ORA-01110: data file 7: '/u1/oradata/prod/table/esis_codes_data01.dbf'
> >
> > ORA-27086: unable to lock file - already in use
> >
> > Linux-x86_64 Error: 11: Resource temporarily unavailable
> >
> > Additional information: 8
> >
> > ORA-01157: cannot identify/lock data file 8 - see DBWR trace file
> >
> > ORA-01110: data file 8: '/u1/oradata/prod/index/esis_codes_indx01.dbf'
> >
> > ORA-27086: unable to lock file - already in use
> >
> > Linux-x86_64 Error: 11: Resource temporarily unavailable
> >
> > Additional information: 8
> >
> > ORA-01157: cannot identify/lock data file 9 - see DBWR trace file
> >
> > ORA-01110: data file 9: '/u1/oradata/prod/index/esis_large_indx.dbf'
> >
> > ORA-27086: unable to lock file - already in use
> >
> > Linux-x86_64 Error: 11: Resource temporarily unavailable
> >
> > Additional information: 8
> >
> > ORA-01157: cannot identify/lock data file 10 - see DBWR trace file
> >
> > ORA-01110: data file 10: '/u1/oradata/prod/index/esis_temp_indx.dbf'
> >
> > ORA-27086: unable to lock file - already in use
> >
> > Linux-x86_64 Error: 11: Resource temporarily unavailable
> >
> > Additional information: 8
> >
> > ORA-01157: cannot identify/lock data file 11 - see DBWR trace file
> >
> > ORA-01110: data file 11: '/u1/oradata/prod/index/esis_indx.dbf'
> >
> > ORA-27086: unable to lock file - already in use
> >
> > Linux-x86_64 Error: 11: Resource temporarily unavailable
> >
> > Additional information: 8
> >
> > ORA-01157: cannot identify/lock data file 12 - see DBWR trace file
> >
> > ORA-01110: data file 12: '/u1/oradata/prod/table/school_connects.dbf'
> >
> > ORA-27086: unable to lock file - already in use
> >
> > Linux-x86_64 Error: 11: Resource temporarily unavailable
> >
> > Additional information: 8
> >
> > ORA-01157: cannot identify/lock data file 13 - see DBWR trace file
> >
> > ORA-01110: data file 13: '/u1/oradata/prod/table/esis_temp_data01.dbf'
> >
> > ORA-27086: unable to lock file - already in use
> >
> > Linux-x86_64 Error: 11: Resource temporarily unavailable
> >
> > Additional information: 8
> >
> > ORA-01157: cannot identify/lock data file 14 - see DBWR trace file
> >
> > ORA-01110: data file 14: '/u1/oradata/prod/table/esis_large_data01.dbf'
> >
> > ORA-27086: unable to lock file - already in use
> >
> > Linux-x86_64 Error: 11: Resource temporarily unavailable
> >
> > Additional information: 8
> >
> > ORA-01157: cannot identify/lock data file 15 - see DBWR trace file
> >
> > ORA-01110: data file 15: '/u1/oradata/prod/table/esis_data01.dbf'
> >
> > ORA-27086: unable to lock file - already in use
> >
> > Linux-x86_64 Error: 11: Resource temporarily unavailable
> >
> > Additional information: 8
> >
> > ORA-01157: cannot identify/lock data file 16 - see DBWR trace file
> >
> > ORA-01110: data file 16: '/u1/oradata/prod/table/tools01.dbf'
> >
> > ORA-27086: unable to lock file - already in use
> >
> > Linux-x86_64 Error: 11: Resource temporarily unavailable
> >
> > Additional information: 8
> >
> >
> >
> > The work around that we have found is to copy all of the files off of that
> > mount point and then move them back.  We even tried stopping everything
> > and umounting/remounting the file system, but it still has the same
> > errors.
> >
> >
> >
> > Oracle and Netapp are both pointing to other as the root of the problem.
> > But to me it sounds like an NFS locking issue.  Has anyone else
> > experienced this, or have any suggestions on a better work around that
> > having to copy the data files off and move them back?
> >
>
> (sorry if this is a duplicate)
>
> Im assuming you have checked to be certain no firewall is blocking any
> NFS ports.  If not, do that first.  At the very least, on the client run
> rpcinfo -p <server>.
>
> Check your NFS mount parameters.  Oracle can be very picky and the error
> messages are usually not very helpful.  Here is what I've found works
> (YMMV):
>
> server:/etc/exports (rw,insecure,root_squash,no_subtree_check,sync)
>
> client:/etc/fstab  user,tcp,rsize=32768,wsize=32768,hard,intr,nfsvers=3
>
> If the system is RAC'ed add noac to the list
>
> The rsize and wsize can be critical under heavy I/O.
>
> >
> > Thanks
> >
> > -Troy
> >
> >
> >
>
>
> --
> Stephen Carville <stephen at totalflood.com>
> Systems Engineer
> Land America
> 1.626.667.1450 X1326
>
>
> --
Have you verified that it isn't the NetApp box that holds the locks?
There are a few Metalink articles that describe how to check this.
Cf. 429912.1 and - for older boxes - 145194.1. )

Kind regards,

Herta