ORA-07445 [SIGBUS] [Object specific hardware error]错误一例

2010/09/09 by 3 Comments

一套Solaris上的9.2.0.7系统，实例意外终止，告警日志中出现以下记录:

Thu Sep 2 02:15:41 2010
Errors in file /u01/app/oracle/admin/preg063/bdump/preg063_smon_11391.trc:
ORA-07445: exception encountered: core dump [0000000101E05500] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CB3BF90] [] []
Thu Sep 2 02:15:48 2010
Errors in file /u01/app/oracle/admin/preg063/bdump/preg063_pmon_11379.trc:
ORA-00474: SMON process terminated with error
Thu Sep 2 02:15:48 2010
PMON: terminating instance due to error 474
Wed Sep 1 15:04:20 2010
Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_1772.trc:
ORA-07445: exception encountered: core dump [FFFFFFFF7F400980] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CA78000] [] []
Wed Sep 1 15:06:24 2010
Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_5316.trc:
ORA-07445: exception encountered: core dump [0000000101E10F94] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7C843DC8] [] []
Wed Sep 1 15:06:24 2010
Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_5207.trc:
ORA-07445: exception encountered: core dump [0000000101E10F94] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7C887A70] [] []
Wed Sep 1 15:06:24 2010
Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_28532.trc:
ORA-07445: exception encountered: core dump [0000000101E05500] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7C9B7FE8] [] []
Wed Sep 1 15:06:31 2010
Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_4392.trc:
ORA-07445: exception encountered: core dump [0000000100624600] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CA80000] [] []
Wed Sep 1 15:06:33 2010
Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_3748.trc:
ORA-07445: exception encountered: core dump [FFFFFFFF7F400980] [SIGBUS] [Object specific hardware error]


trace日志:

Dump file /u01/app/oracle/admin/preg063/bdump/preg063_smon_11391.trc
Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.7.0 - Production
ORACLE_HOME = /u01/app/oracle/product/920preg063
System name: SunOS

2010-09-02 02:15:41.917
*** SESSION ID:(7.1) 2010-09-02 02:15:33.090
Exception signal: 10 (SIGBUS), code: 3 (Object specific hardware error), addr: 0xffffffff7cb3bf90, PC: [0x101e05500, 0000000101E05500]
*** 2010-09-02 02:15:41.936
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [0000000101E05500] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CB3BF90] [] []
Current SQL statement for this session:
update sys.col_usage$ set equality_preds = equality_preds + decode(bitand(:flag,1),0,0,1), equijoin_preds = equijoin_preds + decode(bitand(:flag,2),0,0,1), nonequijoin_preds = nonequijoin_preds + decode(bitand(:flag,4),0,0,1), range_preds = range_preds + decode(bitand(:flag,8),0,0,1), like_preds = like_preds + decode(bitand(:flag,16),0,0,1), null_preds = null_preds + decode(bitand(:flag,32),0,0,1), timestamp = :time where obj# = :objn and intcol# = :coln
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp()+328 CALL ksedst() 00000000B ? 000000000 ?
000000000 ? 103357D68 ?
00000003E ?
FFFFFFFF7FFE3EE8 ?
ssexhd()+676 CALL ksedmp() 000103788 ? 103788000 ?
103788C68 ? 10378D000 ?
000102C00 ? 000000000 ?
sigacthandler()+44 PTR_CALL 0000000000000000 000103790 ?
FFFFFFFF7FFEAF80 ?
103790000 ? 10378DE30 ?
000000000 ? 103790DB0 ?
kghbshrt()+96 PTR_CALL 0000000000000000 00000000A ?
FFFFFFFF7FFEAF80 ?
FFFFFFFF7FFEACA0 ?
000000000 ? 00000BF78 ?
0B38F0000 ?
kghalo()+612 CALL kghbshrt() 103788EC8 ? 000000000 ?
10378C2E0 ?
FFFFFFFF7CB30018 ?
000000458 ? 080000000 ?
kghgex()+336 CALL kghalo() 000000000 ?
FFFFFFFF7CB30018 ?
103788EC8 ? 000001000 ?
FFFFFFFF7CB3E120 ?
C0B38F000000C3D1 ?
kghalf()+316 CALL kghgex() 000007C00 ? 000000440 ?
000000228 ? 103788EC8 ?
000007FFF ? 000000000 ?
kdb4chk()+216 PTR_CALL 0000000000000000 103788EC8 ? 000000178 ?
FFFFFFFF7CB3E180 ?
FFFFFFFF7CB3E150 ?
000000000 ? 102DA35D0 ?
kd4chk()+88 CALL kdb4chk() 3F5484014 ? 1007E7780 ?
000000000 ? 000000000 ?
1007E76C0 ? 000000000 ?
kcbchk()+244 PTR_CALL 0000000000000000 3F5484000 ? 000000000 ?
000002000 ? 000000000 ?
1007E76C0 ? 1007E7720 ?
kcoapl()+1288 CALL kcbchk() 3F5484000 ? 0001007E7 ?
1007E7000 ? 101BA4440 ?
000100400 ? 0001007E7 ?
kcbapl()+96 CALL kcoapl() 000000002 ? 000000000 ?
000000001 ? 3F5484000 ?
10388DA08 ? 3F5486000 ?
kcrfwr()+1716 CALL kcbapl() 10388D990 ? 3F5FA6200 ?
FFFFFFFF7FFEC3B0 ?
00010378E ? 000000004 ?
000000004 ?
kcbchg1()+2056 CALL kcrfwr() FFFFFFFF7FFEC638 ?
40E36C218 ? 10378F5E8 ?
FFFFFFFF7FFEC3B0 ?
10378F690 ? 000000002 ?
ktuchg()+1044 CALL kcbchg1() 000000000 ?
FFFFFFFF7FFECE08 ?
000000000 ? 10378B530 ?
000000000 ?
FFFFFFFF7FFEC730 ?
ktbchg2()+132

Dump file /u01/app/oracle/admin/preg063/udump/preg063_ora_10110.trc
Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.7.0 - Production
ORACLE_HOME = /u01/app/oracle/product/920preg063
System name: SunOS


2010-09-01 00:29:31.655
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [0000000101E10F94] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CAAC000] [] []
Current SQL statement for this session:
select lxOid,lxVal from lxString_c83b5454 where lxOid in (:va,:vb,:vc,:vd,:ve,:vf,:vg,:vh,:vi,:vj,:vk,:vl,:vm,:vn,:vo,:vp,:vq,:vr,:vs,:vt,:vu,:vv,:vw,:vx,:vy,:vz,:v26,:v27,:v28,:v29,:v30,:v31,:v32,:v33,:v34,:v35,:v36,:v37,:v38,:v39,:v40,:v41,:v42,:v43,:v44,:v45,:v46,:v47,:v48,:v49,:v50,:v51,:v52,:v53,:v54,:v55,:v56,:v57,:v58,:v59) and lxType=:v60
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp()+328 CALL ksedst() 00000000B ? 000000000 ?
000000000 ? 103357D68 ?
00000003E ?
FFFFFFFF7FFF3B98 ?
ssexhd()+676 CALL ksedmp() 000103788 ? 103788000 ?
103788C68 ? 10378D000 ?
000102C00 ? 000000000 ?
sigacthandler()+44 PTR_CALL 0000000000000000 000103790 ?
FFFFFFFF7FFFAC30 ?
103790000 ? 10378DE30 ?
000000000 ? 103790DB0 ?
kghalf()+436 PTR_CALL 0000000000000000 00000000A ?
FFFFFFFF7FFFAC30 ?
FFFFFFFF7FFFA950 ?
103788EC8 ? 0B38F0000 ?
B38F0000000000 ?
qerhjManageListElem CALL kghalf() 103788EC8 ?
ent()+116 10B38F0000000001 ?
FFFFFFFF7CC73418 ?
FFFFFFFF7CC73418 ?
FFFFFFFF7CAABFF8 ?
1034CC568 ?
qerhjadf()+248 CALL qerhjManageListElem FFFFFFFF7CBA2568 ?
ent() FFFFFFFF7CC733A8 ?
000000002 ? 000000049 ?
FFFFFFFF7CA41E9A ?
0000000AF ?
qerhjBuildHashTable CALL qerhjadf() FFFFFFFF7FFFB0F8 ?
()+4236 FFFFFFFF7C9D2DB0 ?
000000013 ?
FFFFFFFF7C9D2D90 ?
FFFFFFFF7CA41E9A ?
000000001 ?
qerhjFetch()+772 CALL qerhjBuildHashTable 000000000 ? 1038698D8 ?
() 000000040 ?
FFFFFFFF7C9B16A8 ?
000000001 ?
FFFFFFFF7CBA2630 ?
qervwFetch()+148 PTR_CALL 0000000000000000 43C80CBB0 ? 101624E80 ?
FFFFFFFF7FFFB338 ?
00000001E ? 000000008 ?
00000005F ?
kpofrws()+124 PTR_CALL 0000000000000000 43C80CB58 ? 100FEF620 ?
FFFFFFFF7FFFB940 ?
00000001E ? 000101400 ?
000101624 ?
opifch2()+1704 CALL kpofrws() 000000004 ? 000000004 ?
100FEF620 ?
FFFFFFFF7FFFB940 ?
102EC9160 ? 000000010 ?
opiall0()+3900 CALL opifch2() 10101B000 ? 000000006 ?

提交SR后，MOS认为可能是swap空间不足导致，建议检查系统日志:”Please check your OS logs for any errors during this time.Also please check swap space.
This error is sometimes recorded when swap space becomes full”。

系统日志显示在Sep 2出现过/tmp文件系统空间耗尽，swap空间不足的问题:

Aug 15 02:41:25 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded
Aug 15 02:41:25 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded
Aug 15 02:48:06 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded
Sep 2 02:15:42 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded
Sep 2 02:15:42 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded
Sep 2 02:15:42 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded

该内部错误通过增大swap最后解决了，that’s great!

Filed Under: Oracle Tagged With: , , , , ,

Comments

  1. admin says
    2010/09/09 at 22:47

    ORA-7445: exception encountered: core dump […] [SIGBUS] [Object specific hardware error]

    Applies to:

    Oracle Server – Enterprise Edition – Version: 8.1.7.4 to 11.2.0.1.0 – Release: 8.1.7 to 11.2
    Sun SPARC Sun OS
    *** Checked for relevance 29-Jul-2010 ***

    Symptoms

    The alert log shows an ORA-7445 error of the form:

    ORA-07445: exception encountered: core dump […] [SIGBUS] [Object specific hardware error]

    Cause

    This error indicates a lack of an operating system resource and has caused an Oracle process to terminate.

    Solution

    Check the operating system log files for any errors, and then resolve that error.

    The most common cause is a lack of swap space configured on the system. This can usually be identified from the operating system log file which shows something like:

    System message log (/var/adm/message):

    Apr 7 15:30:00 aemhrsPD1 genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 15415 (cron).

    In such cases either:

    1. Tune the database to ensure that limited swap space is not exhausted, or

    2. Increase the system swap space to three times the physical memory.

     

    回复
  2. admin says
    2010/09/09 at 22:48

    Hdr: 6474768 9.2.0.7.0 RDBMS 9.2.0.7.0 UNKNOWN PRODID-5 PORTID-23 ORA-7445
    Abstract: ORA-7445 [MEMSET] [OBJECT SPECIFIC HARDWARE ERROR]
    PROBLEM:
    ——–
    Following errors occurring during first week of the month.

    ORA-4030: out of process memory when trying to allocate 24956 bytes (pga
    heap,ksm stack)
    ORA-7445: exception encountered: core dump [memset()+320] [SIGBUS] [Object
    specific hardware
    error] [0xFE526000] [] []
    ORA-7445: exception encountered: core dump [ksmarfg()+628] [SIGBUS] [Object
    specific hardware
    error] [0xFE8E0000] [] []

    Same time message.log shows
    Oct 2 10:52:01 dmnlcdsi1 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File
    system full, swap
    space limit exceeded
    Oct 2 10:52:01 dmnlcdsi1 last message repeated 1 time
    Oct 2 10:52:01 dmnlcdsi1 genunix: [ID 470503 kern.warning] WARNING: Sorry,
    no swap space to grow
    stack for pid 13816 (ps)

    First time error reported on Aug 9th and then same problem on Sep-03
    In SR# 6468767.993, suggested to increase the swap size.
    Memory size is 32GB so increased the 64 GB. But error still occurs on Oct-02
    again.

    Needs to identify whether any oracle process or a query occupying most of the
    swap space

    DIAGNOSTIC ANALYSIS:
    ——————–

    WORKAROUND:
    ———–

    RELATED BUGS:
    ————-
    Bug 3981424

    REPRODUCIBILITY:
    —————-
    Problem occurs every month

    TEST CASE:
    ———-

    STACK TRACE:
    ————
    ksedmp ssexhd sigacthandler kghbshrt kghalo kghgex
    kghfnd kghalo kghgex kghfnd kghprmalo kghalp
    stsAllocFromSubheap smbalo smbput qergsRowP qerhjWalkHashBucket
    qerhjProbeHashTable
    qerhjWalkHashBucket qerhjProbeHashTable qerixFetchFastFullS can rwsfcd
    qerhjFetch
    rwsfcd qerhjFetch qergsFetch qerflFetchOutside qersoFetch kpofrws
    opifch2 opifch opipls opiodr rpidrus skgmstack

    SUPPORTING INFORMATION:
    ———————–
    Uploaded all the trace files, RDA and statspack report to bug.

    回复
  3. admin says
    2010/09/09 at 22:49

    ORA-07445: core dump [FFFFFFFF7F400AE8] [SIGBUS] [Object specific hardware error]

    Applies to:

    Oracle Server – Enterprise Edition – Version: 8.1.7.0 to 11.1.0.7 – Release: 8.1.7 to 11.1

    Generic UNIX

    Symptoms

    While performing a select, a core dump is generated in the alertlog showing the following error:

    ORA-07445: exception encountered: core dump [FFFFFFFF7F400AE8] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF73A9E000] [] []

    The Call Stack Trace in the ORA-7445 trace file contains:

    ksedmp ssexhd sigacthandler memset kghalo kghgex kghfnd kghalo kghgex …

    Cause

    The Call Stack Trace points to:

    Bug 6474768 – ORA-07445 [MEMSET] [OBJECT SPECIFIC HARDWARE ERROR] which was closed as not a bug

    In Bug 6474768  it was suggested that the error is caused by the swap space becoming full.

    This was further confirmed by checking the Operating System error log which also recorded the following message at the same time with the error ORA-7445:

    …. WARNING: Sorry, no swap space to grow stack for pid 13621 (oracle)

    Solution

    1. Increase swap space on the OS. A recommended value would be 2 – 3 x RAM size.

     

    回复

Comment

*

沪ICP备14014813号-2

沪公网安备 31010802001379号