Oracle EBS的DB Server无法启动

冷备份系统后，然后重启DB Listener和DB没报什么错误，再起通过命令adstrtal.sh apps/apps来启动Apps Server的时候报错

[oracle@bej301441 scripts]$ adstrtal.sh apps/password
You are running adstrtal.sh version 120.15.12010000.3
adstrtal.sh: Database connection could not be established. Either the database is down or the APPS credentials supplied are wrong.
USAGE: adstrtal.sh <appsusername/appspassword>
adstrtal.sh <applications_username/applications_password> -secureapps
adstrtal.sh -nodbchk
adstrtal.sh: exiting with status 1

如果DB有问题的话，那么./adstrtal.sh apps/apps 是不会启动的。

解决过程：
然后检查DB的情况，尝试用sqlplus来连instance.

[oracle@bej301441 mc3yd213_bej301441]$ sqlplus apps/password@mc3yd213
SQL*Plus: Release 10.1.0.5.0 - Production on Sat Jun 30 03:18:39 2012
Copyright (c) 1982, 2005, Oracle. All rights reserved.
ERROR:
ORA-12537: TNS:connection closed
Enter user-name:

sqlplus不能连上instance，那么应该是数据库的问题。

ping服务器的IP

C:\Documents and Settings\ptian>ping 10.182.74.181
Pinging 10.182.74.181 with 32 bytes of data:
Reply from 10.182.74.181: bytes=32 time=229ms TTL=57
Reply from 10.182.74.181: bytes=32 time=230ms TTL=57
Reply from 10.182.74.181: bytes=32 time=230ms TTL=57
Reply from 10.182.74.181: bytes=32 time=228ms TTL=57
Ping statistics for 10.182.74.181:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 228ms, Maximum = 230ms, Average = 229ms
C:\Documents and Settings\ptian>

能够ping通，说明网络没问题

SELINUX=disabled，防火墙也是关闭的，不存在防火墙阻挡的问题。

[oracle@bej301441 mc3yd213_bej301441]$ vi /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - SELinux is fully disabled.
SELINUX=disabled
# SELINUXTYPE= type of policy in use. Possible values are:
# targeted - Only targeted network daemons are protected.
# strict - Full SELinux protection.
SELINUXTYPE=targeted

tnsping这个instance

C:\Documents and Settings\ptian>tnsping mc3yd213
TNS Ping Utility for 32-bit Windows: Version 10.1.0.2.0 - Production on 30-6月 -2012 18:14:29
Copyright (c) 1997, 2003, Oracle. All rights reserved.
Used parameter files:
D:\oracle\product\10.1.0\Db_1\NETWORK\ADMIN\sqlnet.ora
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = tcp) (HOST = myip)(PORT = 1522)) (CONNECT_DATA = (SID = mc3yd213)))
OK (770 msec)
C:\Documents and Settings\ptian>

这是说明服务器端DB Listener已经启动了。

接下来再次检查listener的状态

[oracle@bej301441 mc3yd213_bej301441]$ lsnrctl status mc3yd213
LSNRCTL for Linux: Version 10.1.0.5.0 - Production on 30-JUN-2012 03:50:09
Copyright (c) 1991, 2004, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=bej301441)(PORT=1522))(CONNECT_DATA=(SID=mc3yd213)))
STATUS of the LISTENER
------------------------
Alias mc3yd213
Version TNSLSNR for Linux: Version 11.1.0.7.0 - Production
Start Date 30-JUN-2012 03:09:47
Uptime 0 days 0 hr. 40 min. 21 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /u01/oracle/mc3yd213/db/tech_st/11.1.0/network/admin/mc3yd213_bej301441/listener.ora
Listener Log File /u01/oracle/mc3yd213/db/tech_st/11.1.0/log/diag/tnslsnr/bej301441/mc3yd213/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=myip)(PORT=1522)))
Services Summary...
Service "mc3yd213" has 1 instance(s).
Instance "mc3yd213", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

确认DB Listener以及启动

检查端口

[oracle@bej301441 mc3yd213_bej301441]$ netstat -an | grep 1522
tcp 0 0 0.0.0.0:1522 0.0.0.0:* LISTEN
tcp 0 0 10.182.74.33:1522 10.182.74.33:50530 TIME_WAIT
tcp 11541 0 ::ffff:10.182.74.33:23775 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 22601 0 ::ffff:10.182.74.33:47311 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 10061 0 ::ffff:10.182.74.33:60382 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 10061 0 ::ffff:10.182.74.33:60390 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 23081 0 ::ffff:10.182.74.33:37178 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 5501 0 ::ffff:10.182.74.33:47700 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 21611 0 ::ffff:10.182.74.33:12133 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 22601 0 ::ffff:10.182.74.33:47304 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 10131 0 ::ffff:10.182.74.33:23782 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 10071 0 ::ffff:10.182.74.33:60285 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 10061 0 ::ffff:10.182.74.33:60379 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 23081 0 ::ffff:10.182.74.33:37175 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 10071 0 ::ffff:10.182.74.33:60282 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 10071 0 ::ffff:10.182.74.33:60284 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 23081 0 ::ffff:10.182.74.33:37171 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 5391 0 ::ffff:10.182.74.33:47697 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 10061 0 ::ffff:10.182.74.33:60375 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 22601 0 ::ffff:10.182.74.33:47306 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 10061 0 ::ffff:10.182.74.33:60377 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 10061 0 ::ffff:10.182.74.33:60385 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 21611 0 ::ffff:10.182.74.33:12131 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 21551 0 ::ffff:10.182.74.33:12135 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 4351 0 ::ffff:10.182.74.33:60392 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 22601 0 ::ffff:10.182.74.33:47308 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 5971 0 ::ffff:10.182.74.33:29442 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 22601 0 ::ffff:10.182.74.33:47310 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 4271 0 ::ffff:10.182.74.33:60399 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 9811 0 ::ffff:10.182.74.33:60397 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 10071 0 ::ffff:10.182.74.33:23779 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 19661 0 ::ffff:10.182.74.33:57255 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 5381 0 ::ffff:10.182.74.33:15048 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 10131 0 ::ffff:10.182.74.33:23781 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 10071 0 ::ffff:10.182.74.33:60278 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 5791 0 ::ffff:10.182.74.33:29440 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 10071 0 ::ffff:10.182.74.33:60280 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 21611 0 ::ffff:10.182.74.33:12138 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 10061 0 ::ffff:10.182.74.33:60383 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 23081 0 ::ffff:10.182.74.33:37169 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 10071 0 ::ffff:10.182.74.33:60287 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 21611 0 ::ffff:10.182.74.33:12137 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 21611 0 ::ffff:10.182.74.33:12140 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 22601 0 ::ffff:10.182.74.33:47313 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 11541 0 ::ffff:10.182.74.33:23777 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 23031 0 ::ffff:10.182.74.33:37176 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 11541 0 ::ffff:10.182.74.33:23784 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 5851 0 ::ffff:10.182.74.33:29462 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 23031 0 ::ffff:10.182.74.33:37173 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 4271 0 ::ffff:10.182.74.33:60394 ::ffff:10.182.74.33:1522 CLOSE_WAIT
tcp 22581 0 ::ffff:10.182.74.33:47691 ::ffff:10.182.74.33:1522 CLOSE_WAIT

进程中的tns

[oracle@bej301441 mc3yd213_bej301441]$ ps -ef|grep tns
root 13 2 0 May24 ? 00:00:00 [netns]
oracle 13616 1 0 May27 ? 00:00:32 /u01/oracle/mc3yd213/apps/tech_st/10.1.2/bin/tnslsnr LISTENER -inherit
oracle 22183 1 0 02:55 ? 00:00:00 /u01/oracle/mc3yd213/apps/tech_st/10.1.2/bin/tnslsnr APPS_mc3yd213 -inherit
oracle 22486 1 0 03:09 ? 00:00:00 /u01/oracle/mc3yd213/db/tech_st/11.1.0/bin/tnslsnr mc3yd213 -inherit
oracle 28638 20816 0 03:51 pts/8 00:00:00 grep tns
[oracle@bej301441 mc3yd213_bej301441]$

/etc/hosts

[oracle@bej301441 mc3yd213_bej301441]$ cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
10.182.74.33 bej301441.xxx.com bej301441
10.182.74.181 bej301589.xxx.com
138.1.191.58 rinoid06.xxx.com
138.1.191.59 rinoid07.xxx.com

$TNS_ADMIN/listener.ora

[oracle@bej301441 mc3yd213_bej301441]$ cd $TNS_ADMIN
[oracle@bej301441 admin]$ cat listener.ora
# $Header: admk80ln_ux.sql 120.9 2007/03/09 04:31:05 sbandla ship $
# LISTENER.ORA For Oracle Applications
# This file is automatically generated
APPS_mc3yd213 =
(ADDRESS_LIST =
(ADDRESS= (PROTOCOL= TCP)(Host= host)(Port= 1627))
)
SID_LIST_APPS_mc3yd213 =
(SID_LIST =
( SID_DESC = ( SID_NAME = FNDSM )
( ORACLE_HOME = /u01/oracle/mc3yd213/apps/tech_st/10.1.2 )
( PROGRAM = /u01/oracle/mc3yd213/apps/apps_st/appl/fnd/12.0.0/bin/FNDSM )
( envs='MYAPPSORA=/u01/oracle/mc3yd213/apps/apps_st/appl/APPSmc3yd213_bej301441.env,PATH=/usr/bin:/usr/ccs/bin:/bin,FNDSM_SCRIPT=/u01/oracle/mc3yd213/inst/apps/mc3yd213_bej301441/admin/scripts/gsmstart.sh' )
)
( SID_DESC = ( SID_NAME = FNDFS )
( ORACLE_HOME = /u01/oracle/mc3yd213/apps/tech_st/10.1.2 )
( PROGRAM = /u01/oracle/mc3yd213/apps/apps_st/appl/fnd/12.0.0/bin/FNDFS )
( envs='EPC_DISABLED=TRUE,NLS_LANG=American_America.AL32UTF8,LD_LIBRARY_PATH=/u01/oracle/mc3yd213/apps/tech_st/10.1.2/lib32:/u01/oracle/mc3yd213/apps/tech_st/10.1.2/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/oracle/mc3yd213/apps/tech_st/10.1.2/jdk/jre/lib/i386:/u01/oracle/mc3yd213/apps/tech_st/10.1.2/jdk/jre/lib/i386/server:/u01/oracle/mc3yd213/apps/tech_st/10.1.2/jdk/jre/lib/i386/native_threads:/u01/oracle/mc3yd213/apps/apps_st/appl/sht/12.0.0/lib,SHLIB_PATH=/u01/oracle/mc3yd213/apps/tech_st/10.1.2/lib32:/u01/oracle/mc3yd213/apps/tech_st/10.1.2/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/oracle/mc3yd213/apps/tech_st/10.1.2/jdk/jre/lib/i386:/u01/oracle/mc3yd213/apps/tech_st/10.1.2/jdk/jre/lib/i386/server:/u01/oracle/mc3yd213/apps/tech_st/10.1.2/jdk/jre/lib/i386/native_threads:/u01/oracle/mc3yd213/apps/apps_st/appl/sht/12.0.0/lib,LIBPATH=/u01/oracle/mc3yd213/apps/tech_st/10.1.2/lib32:/u01/oracle/mc3yd213/apps/tech_st/10.1.2/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/oracle/mc3yd213/apps/tech_st/10.1.2/jdk/jre/lib/i386:/u01/oracle/mc3yd213/apps/tech_st/10.1.2/jdk/jre/lib/i386/server:/u01/oracle/mc3yd213/apps/tech_st/10.1.2/jdk/jre/lib/i386/native_threads:/u01/oracle/mc3yd213/apps/apps_st/appl/sht/12.0.0/lib,APPLFSTT=mc3yd213_BALANCE;mc3yd213_FO;mc3yd213,APPLFSWD=/u01/oracle/mc3yd213/inst/apps/mc3yd213_bej301441/appl/admin;/u01/oracle/mc3yd213/inst/apps/mc3yd213_bej301441/appltmp;/u01/oracle/mc3yd213/apps/apps_st/comn/webapps/oacore/html/oam/nonUix/launchMode/restricted' )
)
)
STARTUP_WAIT_TIME_APPS_mc3yd213 = 0
CONNECT_TIMEOUT_APPS_mc3yd213 = 10
TRACE_LEVEL_APPS_mc3yd213 = OFF
LOG_DIRECTORY_APPS_mc3yd213 = /u01/oracle/mc3yd213/inst/apps/mc3yd213_bej301441/logs/ora/10.1.2/network
LOG_FILE_APPS_mc3yd213 = APPS_mc3yd213
TRACE_DIRECTORY_APPS_mc3yd213 = /u01/oracle/mc3yd213/inst/apps/mc3yd213_bej301441/logs/ora/10.1.2/network
TRACE_FILE_APPS_mc3yd213 = APPS_mc3yd213
ADMIN_RESTRICTIONS_APPS_mc3yd213 = ON
IFILE = /u01/oracle/mc3yd213/inst/apps/mc3yd213_bej301441/ora/10.1.2/network/admin/mc3yd213_bej301441_listener_ifile.ora
SUBSCRIBE_FOR_NODE_DOWN_EVENT_APPS_mc3yd213 = OFF

[oracle@bej301441 admin]$

使用addbctl.sh启动数据库

Use addbctl.sh to start DB server.It tell me server is running...
[oracle@bej301441 mc3yd213_bej301441]$ addbctl.sh start immediate
You are running addbctl.sh version 120.1
Starting the database mc3yd213 ...
SQL*Plus: Release 11.1.0.7.0 - Production on Sat Jun 30 07:33:53 2012
Copyright (c) 1982, 2008, Oracle. All rights reserved.
Connected to an idle instance.
ORA-01081: cannot start already-running ORACLE - shut it down first
Disconnected
addbctl.sh: exiting with status 9
[oracle@bej301441 mc3yd213_bej301441]$
提示信息说，数据库已经启动了

尝试重启DB

addbctl.sh stopimmediate >addbctl.sh start发现数据库还是连不上。

解决方法

使用addbctl.sh stop abort来关闭数据库，再用addbctl.sh start启动。之前关闭DB的时候，没加abort项（用的是immediate），说明当时关闭数据库的时候，数据库并没有完全关闭，所以重启没有成功。

关闭项（NORMAL，IMMEDIATE，ABORT）：

SHUTDOWN NORMAL：不允许新的连接、等待会话结束、等待事务结束、做一个检查点并关闭数据文件。启动时不需要实例恢复。
SHUTDOWN IMMEDIATE：不允许新的连接、不等待会话结束、不等待事务结束、做一个检查点并关闭数据文件。没有结束的事务是自动rollback的。启动时不需要实例恢复。
SHUTDOWN ABORT：不允许新的连接、不等待会话结束、不等待事务结束、不做检查点且没有关闭数据文件。启动时自动进行实例恢复。

ps:我在OTN上发的问题帖：https://forums.oracle.com/forums/thread.jspa?threadID=2408941&tstart=0

转载请注明出处：http://blog.csdn.net/pan_tian/article/details/7705275

======EOF======