[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Oracle/Red Hat/NetApp



Troy Knabe wrote:
We have several Oracle instances 10.2 (some RAC, some not, but all are
having the same issue) all running on Red Hat 4.5.  The binaries and data
files are all on the NetApp and mounted via NFS.

Sporadically when we shut the database down, we are unable to restart it.


/u1/app/oracle/admin/prodrac/bdump/prodrac1_dbw0_10023.trc

Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit
Production

With the Partitioning, Real Application Clusters, OLAP and Data Mining
options

ORACLE_HOME = /u1/app/oracle/product/10.2.0

System name:   Linux

Node name:       orarac1

Release:               2.6.9-55.0.2.ELsmp

Version:               #1 SMP Tue Jun 12 17:58:20 EDT 2007

Machine:             x86_64

Instance name: prodrac1

Redo thread mounted by this instance: 1

Oracle process number: 8

Unix process pid: 10023, image: oracle orarac1 (DBW0)

*** SERVICE NAME:() 2007-08-18 02:05:27.878

*** SESSION ID:(1194.1) 2007-08-18 02:05:27.878

ORA-01157: cannot identify/lock data file 1 - see DBWR trace file

ORA-01110: data file 1: '/u1/oradata/prod/system/system01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 2 - see DBWR trace file

ORA-01110: data file 2: '/u1/oradata/prod/temp/undotbs01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 3 - see DBWR trace file

ORA-01110: data file 3: '/u1/oradata/prod/system/sysaux01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 4 - see DBWR trace file

ORA-01110: data file 4: '/u1/oradata/prod/table/users01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 5 - see DBWR trace file

ORA-01110: data file 5: '/u1/oradata/prod/temp/undotbs02.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 6 - see DBWR trace file

ORA-01110: data file 6: '/u1/oradata/prod/temp/undotbs03.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 7 - see DBWR trace file

ORA-01110: data file 7: '/u1/oradata/prod/table/esis_codes_data01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 8 - see DBWR trace file

ORA-01110: data file 8: '/u1/oradata/prod/index/esis_codes_indx01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 9 - see DBWR trace file

ORA-01110: data file 9: '/u1/oradata/prod/index/esis_large_indx.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 10 - see DBWR trace file

ORA-01110: data file 10: '/u1/oradata/prod/index/esis_temp_indx.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 11 - see DBWR trace file

ORA-01110: data file 11: '/u1/oradata/prod/index/esis_indx.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 12 - see DBWR trace file

ORA-01110: data file 12: '/u1/oradata/prod/table/school_connects.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 13 - see DBWR trace file

ORA-01110: data file 13: '/u1/oradata/prod/table/esis_temp_data01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 14 - see DBWR trace file

ORA-01110: data file 14: '/u1/oradata/prod/table/esis_large_data01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 15 - see DBWR trace file

ORA-01110: data file 15: '/u1/oradata/prod/table/esis_data01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 16 - see DBWR trace file

ORA-01110: data file 16: '/u1/oradata/prod/table/tools01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

The work around that we have found is to copy all of the files off of that
mount point and then move them back.  We even tried stopping everything
and umounting/remounting the file system, but it still has the same
errors.

Oracle and Netapp are both pointing to other as the root of the problem.
But to me it sounds like an NFS locking issue.  Has anyone else
experienced this, or have any suggestions on a better work around that
having to copy the data files off and move them back?


(sorry if this is a duplicate)

Im assuming you have checked to be certain no firewall is blocking any
NFS ports.  If not, do that first.  At the very least, on the client run
rpcinfo -p <server>.

Check your NFS mount parameters.  Oracle can be very picky and the error
messages are usually not very helpful.  Here is what I've found works
(YMMV):

server:/etc/exports (rw,insecure,root_squash,no_subtree_check,sync)

client:/etc/fstab  user,tcp,rsize=32768,wsize=32768,hard,intr,nfsvers=3

If the system is RAC'ed add noac to the list

The rsize and wsize can be critical under heavy I/O.


Thanks
-Troy



--
Stephen Carville <stephen totalflood com>
Systems Engineer
Land America
1.626.667.1450 X1326




[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]