Crash Dump Analyse with Scat Oracle Solaris

Unix / October 24, 2019 / By:

At this post I ‘ll provide  the  information how to  analyze crash dump  file for  Oracle Solaris.It helps  to find  out the  root  cause  of crash dump also  hang. If your system installed  properly then there  must be a separated  raw device which  size is as  much as  physical memory. Dump device size  is  related to physical memory size. It’s  important  to assign  dedicated dump device  no less than size  of physical memory.Otherwise,when system gets a  panic, memory dump won’t  be written  dump device  because  of lack  of  free size  in dump device.You ‘ll lose  last  memory snapshot to dig  root cause  of  problem.

First  of  all we  need to check configuratin  if system dump device  configured  properly.

#dumpadm

Dump content: kernel pages
Dump device: /dev/dsk/c1t0d0s1 (swap)
Savecore directory: /var/crash/TESTSERVER
Savecore enabled: yes
#swap -l
swapfile dev swaplo blocks free
/dev/dsk/c1t0d0s1 32,9 16 100664912 100664912
#prtconf |grep -i mem
Memory size: 32640 Megabytes

After  check system  dump device  configuration you can check if server  got a dump file  while panics.

#ls -ld /var/crash/TESTSERVER/vmcore*
-rw-r--r-- 1 root root 3519586304 Oct 29 16:57 /var/crash/TESTSERVER/vmcore.8

If  you ‘ll  make a test  on your test server  and you need  to a crash dump file,  apply  this  steps  to create a  dump  file.It’ll give panic and  get a snapshot  of  memory.

reboot  -d

Then  you need a server  which is same  architecture(Sparc or  X86 which depends  on your  server) and  SCAT  package  is  installed. If your  dump file generated  on a  sparc machine so  you need a sparc machine which  SCAT installed. Login  MOS and  download “Oracle Solaris Crash Analysis Tool”.The  patch  ID  is “p10364262_520_SOLARIS64”. Then install  SCAT.

#unzip p10364262_520_SOLARIS64.zip
Archive: p10364262_520_SOLARIS64.zip
inflating: SUNWscat5.2-GA-combined.pkg.gz
inflating: SUNWscat5.2-GA-i386.pkg.gz
inflating: SUNWscat5.2-GA-sparc.pkg.gz
inflating: license_agreement1.html
inflating: readme.html
root@TEST# gunzip SUNWscat5.2-GA-sparc.pkg.gz
root@TEST# uname -s
SunOS
root@TEST# uname -a
SunOS TEST 5.10 Generic_147147-26 sun4u sparc SUNW,Sun-Fire-V245
root@TEST# pkgadd -d /tmp/SUNWscat5.2-GA-sparc.pkg

While  installing package  you should  accept  installation.”Please enter your response to the license: ACCEPT”. After  installation  Scat  finishes then you need  to copy  vmcore.x file to  the  machine which  you installed SCAT. Start  analyze  with  this command .

# /opt/SUNWscat/bin/scat unix.3  /CASESUP/EXPECT/vmcore.11 

After  scat finish  operation run “kstat  xck”.

SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> analyze

SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> panic

Check this  logs; and  also  run “panic” to see  what happened why server  got panic.

SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> analyze

SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> panic

The strings  which you need to check  are;

  • Panic Strings:
    • panic string:   BAD TRAP: type=31 rp=2a107ee31e0 addr=70 mmu_fsr=0 occurred in module “pxfs” due to a NULL pointer dereference
  • Trap:
    • — trap data  type: 0x31 (data access MMU miss)  rp: 0x2a107ee31e0  —
  • Trap  Strings:
    • <trap>pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88()

After  you get  all dates  abaut  panic now you can  search  google  and  MOS for  more  informations and  bugs.

#/opt/SUNWscat/bin/scat unix.3  /CASESUP/EXPECT/vmcore.11</p>

  Solaris[TM] CAT 5.2 for Solaris 10 64-bit UltraSPARC
SV4990M, Aug 26 2009
Copyright © 2009 Sun Microsystems, Inc.  All rights reserved.
Patents Pending. Use is subject to license terms.
Sun Microsystems, Inc. has intellectual property rights relating to
technology embodied in the product that is described in this document.
In particular, and without limitation, these intellectual property
rights may include one or more of the U.S. patents listed at
http://www.sun.com/patents and one or more additional patents or
pending patent applications in the U.S. and in other countries.
U.S. Government Rights - Commercial software.  Government users
are subject to the Sun Microsystems, Inc. standard license agreement
and applicable provisions of the FAR and its supplements.
Use is subject to license terms.  Portions may be derived from
Berkeley BSD systems, licensed from U. of CA.
Sun, Sun Microsystems, the Sun logo, Java, Solaris, SunSpectrum and
SunTone are trademarks or registered trademarks of Sun Microsystems,
Inc. in the U.S. and other countries.
All SPARC trademarks are used under license and are trademarks or
registered trademarks of SPARC International, Inc. in the U.S. and
other countries.
Feedback regarding the tool should be sent to SolarisCAT_Feedback@Sun.COM
Visit the Solaris CAT blog at http://blogs.sun.com/SolarisCAT
opening /CASESUP/EXPECT/vmcore.11 ...dumphdr...symtab...core...done
loading core data: modules...symbols...CTF...done
core file:      /CASESUP/EXPECT/vmcore.11
user:           Super-User (root:0)
release:        5.10 (64-bit)
version:        Generic_127127-11
machine:        sun4v
node name:      TESTSERVER
hw_provider:    Sun_Microsystems
system type:    SUNW,SPARC-Enterprise-T5220 (UltraSPARC-T2)
hostid:         84ebae8a
dump_conflags:  0x10000 (DUMP_KERNEL) on /dev/dsk/c1t0d0s1(48G)
cluster_bflgs:  0x3 (CLUSTER_CONFIGURED|CLUSTER_BOOTED)
pxfs_software_mount_level: v1 (consolidated version)
current node:   1
time of crash:  Thu Oct 29 20:14:31 EET 2015
age of system:  3 hours 21 minutes 29.68 seconds
panic CPU:      60 (64 CPUs, 31.8G memory)
panic string:   BAD TRAP: type=31 rp=2a107ee31e0 addr=70 mmu_fsr=0 occurred in module "pxfs" due to a NULL pointer dereference

sanity checks: settings...
NOTE: /etc/system: module ge not loaded for "set ge:ge_intr_mode=0x833"
vmem...CPU...sysent...
WARNING: unknown module acctctl seen 4 times in sysent table
clock...
WARNING: hres_last_tick 4 minutes 19.874264183 seconds later
misc...done
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> kstat xck
loaded 35 kstat checks from /opt/SUNWscat/lib/kstat_xck
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> /opt/SUNWscat/lib/kstat_xck
/opt/SUNWscat/exec/scat510U: /opt/SUNWscat/lib/kstat_xck: cannot execute
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> analyze
core file:      /CASESUP/EXPECT/vmcore.11
user:           Super-User (root:0)
release:        5.10 (64-bit)
version:        Generic_127127-11
machine:        sun4v
node name:      TESTSERVER
hw_provider:    Sun_Microsystems
system type:    SUNW,SPARC-Enterprise-T5220 (UltraSPARC-T2)
hostid:         84ebae8a
dump_conflags:  0x10000 (DUMP_KERNEL) on /dev/dsk/c1t0d0s1(48G)
cluster_bflgs:  0x3 (CLUSTER_CONFIGURED|CLUSTER_BOOTED)
pxfs_software_mount_level: v1 (consolidated version)
current node:   1
time of crash:  Thu Oct 29 20:14:31 EET 2015
age of system:  3 hours 21 minutes 29.68 seconds
panic CPU:      60 (64 CPUs, 31.8G memory)
panic string:   BAD TRAP: type=31 rp=2a107ee31e0 addr=70 mmu_fsr=0 occurred in module "pxfs" due to a NULL pointer dereference
==== checking for trap information ====
CPU 60 had the panic
==== panic thread: 0x2a107ee3ca0 ==== CPU: 60 ====
==== panic kernel thread: 0x2a107ee3ca0  PID: 0  on CPU: 60  affinity CPU: 60 ====
cmd: sched
t_procp: 0x18594e0(proc_sched)
p_as: 0x185a8f8(kas)
zone: global
t_stk: 0x2a107ee3a90  sp: 0x18756e1  t_stkbase: 0x2a107ede000
t_pri: 60(SYS)  pctcpu: 0.000000
t_lwp: 0x0  psrset: 0  last CPU: 60
idle: 9000 ticks (1 minutes 30.00 seconds)
start: Thu Oct 29 17:55:01 2015
age: 8370 seconds (2 hours 19 minutes 30 seconds)
tstate: TS_ONPROC - thread is being run on a processor
tflg:   T_TALLOCSTK - thread structure allocated from stk
T_PANIC - thread initiated a system panic
tpflg:  none set
tsched: TS_LOAD - thread is in memory
TS_DONT_SWAP - thread/LWP should not be swapped
pflag:  SSYS - system resident process
pc:      unix:panicsys+0x48:   call     unix:setjmp
startpc: nfssrv:reaper_thread+0x0:   save       %sp, -0xb0, %sp
unix:panicsys+0x48(0x1096af8, 0x2a107ee2f88, 0x18760b0, 0x1, , , 0x9900001603, , , , , , , , 0x1096af8, 0x2a107ee2f88)
unix:vpanic_common+0x78(0x1096af8, 0x2a107ee2f88, 0x18a6800, 0x2a107ee2a81, 0x2a107ee3280, 0x4400001606)
unix:panic+0x1c(0x1096af8, 0x31, 0x2a107ee31e0, 0x70, 0x0, 0x30022dec1a8, 0x1831aa0)
unix:die+0x78(0x31, 0x2a107ee31e0, 0x70, 0x0)
unix:trap+0x9e4(0x2a107ee31e0, 0x70)
unix:ktl0+0x64()
-- trap data  type: 0x31 (data access MMU miss)  rp: 0x2a107ee31e0  --
addr: 0x70
pc:  0x7a61b944 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88:   ldub  [%l6 + 0x70], %l1
npc: 0x7a61b948 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x8c:   ldub  [%l3 + 0xd54], %l2
global:                       %g1         0x7a617cbc
%g2                0x3  %g3                0x3%g4      0x30051198280  %g5                0x4%g6               0x6d  %g7      0x2a107ee3ca0out:  %o0      0x300380d1850  %o1      0x2a107ee3ca4
%o2                  0  %o3                0x4%o4            0x70800  %o5      0x300380d1870%sp      0x2a107ee2a81  %o7         0x7a632b94loc:  %l0      0x3004a7f1bd0  %l1               0x13
%l2                0x1  %l3         0x7087c000%l4            0x7087c  %l5            0x70800%l6                  0  %l7             0x1000in:   %i0      0x300380d1850  %i1      0x3003d41f280
%i2             0x1000  %i3         0xffffef87%i4         0x7082c000  %i5            0x7082c%fp      0x2a107ee2b31  %i7          0x110be98<trap>pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88()
unix:cl_flk_state_transition_notify(0x3003d41f280, 0x3, 0x8) - frame recycled
genunix:flk_set_state+0x60(0x3003d41f280, 0x8, 0x0)
genunix:flk_delete_active_lock+0x58(0x3003d41f280, 0x0)
genunix:cl_flk_remove_locks_by_sysid+0xf0(0x10004)
pxfs:void pxfs_namespace_v1::fs_ii::remove_file_locks(int,CORBA::Environment&)+0xcc(0x30022cbe980?, 0x10004?)
pxfs:__SLIP.THUNK__P(, 0x10004, 0x2a107ee36a0) - frame recycled
pxfs:void pxfs_namespace_v1::pxvfs::remove_file_locks(int)+0xf4(0x10004)
pxfs:void nlm_remove_file_locks(int)+0x18(0x10004, 0x4, 0x10000, 0x1)
nfssrv:rfs4_lo_state_destroy+0x6c(0x3004a7f1c00)
nfssrv:rfs4_dbe_destroy+0x16c(0x3004a7f1b98?)
nfssrv:rfs4_dbe_reap+0x18c(0x300398c0000, 0x5a, 0x0)
nfssrv:reaper_thread+0x104(0x300398c0000, 0x0)
unix:thread_start+0x4()
-- end of kernel thread's stack --
==== analyzing panic thread stack for trap frames ====
==== using trap() frame 1 @ 0x2a107ee2fe0, rp(%i0): 0x2a107ee31e0 ====
type(%l2): 0x31 (data access MMU miss)
pc:  0x7a61b944 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88:   ldub  [%l6 + 0x70], %l1
npc: 0x7a61b948 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x8c:   ldub  [%l3 + 0xd54], %l2

global:                         %g1 pxfs:cl_flk_state_transition_notify+0x0
%g2                0x3  %g3                0x3%g4      0x30051198280  %g5                0x4%g6               0x6d  %g7      0x2a107ee3ca0out:    %o0      0x300380d1850  %o1      0x2a107ee3ca4
%o2                  0  %o3                0x4%o4            0x70800  %o5      0x300380d1870%sp      0x2a107ee2a81  %o7 pxfs:void flk_state_transition_notify(lock_descriptor*,int,int)+0x130loc:    %l0      0x3004a7f1bd0  %l1               0x13
%l2                0x1  %l3 pxfs(data):        0x7087c000%l4            0x7087c  %l5            0x70800%l6                  0  %l7             0x1000in:     %i0      0x300380d1850  %i1      0x3003d41f280
%i2             0x1000  %i3         0xffffef87%i4 cl_dcs(data):PXFS_GREEN+0x0 %i5            0x7082c%fp      0x2a107ee2b31  %i7 genunix:flk_set_state+0x60
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x60:      sra        %o7, 0x0, %o4
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x64:      add        %g3, 0x96f, %o1
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x68:      sra        %l0, 0x0, %o3
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x6c:      sra        %o5, 0x0, %o5
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x70:      call       cl_load:void dbg_print_buf::dbprintf(char*,...)
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x74:      or %g0, %i0, %o2   ( mov   %i0, %o2 )

pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x78:   1: sethi      %hi(0x70800) %l5        ( set   0x70800, %l5 )
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x7c:      ldx        [%i0 + 0x30], %l6
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x80:      or %l5, 0x7c, %l4 (0x7087c)
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x84:      sllx       %l4, 0xc, %l3
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88:      ldub       [%l6 + 0x70], %l1
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x8c:      ldub       [%l3 + 0xd54], %l2
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x90:      orcc       %l1, %l2, %g0
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x94:      bne,pn     %icc, pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xc8 (2f)
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x98:      or %g0, %i0, %o0   ( mov   %i0, %o0 )

pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x9c:      call       pxfs:repl_pxfs::lock_info_t*fobj_ii::get_lock_info(lock_descriptor*)
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xa0:      or %g0, %i1, %o1   ( mov   %i1, %o1 )
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xa4:      or %g0,  0x1, %i1  ( mov   0x1, %i1 )
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xa8:      or %g0, %o0, %l7   ( mov   %o0, %l7 )
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xac:      stw        %i1, [%o0 + 0x60]
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xb0:      or %g0,  0x10, %o0 ( mov   0x10, %o0 )
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> panic
panic on CPU 60
panic string:   BAD TRAP: type=31 rp=2a107ee31e0 addr=70 mmu_fsr=0 occurred in module "pxfs" due to a NULL pointer dereference
==== panic kernel thread: 0x2a107ee3ca0  PID: 0  on CPU: 60  affinity CPU: 60 ====
cmd: sched
t_procp: 0x18594e0(proc_sched)
p_as: 0x185a8f8(kas)
zone: global
t_stk: 0x2a107ee3a90  sp: 0x18756e1  t_stkbase: 0x2a107ede000
t_pri: 60(SYS)  pctcpu: 0.000000
t_lwp: 0x0  psrset: 0  last CPU: 60
idle: 9000 ticks (1 minutes 30.00 seconds)
start: Thu Oct 29 17:55:01 2015
age: 8370 seconds (2 hours 19 minutes 30 seconds)
tstate: TS_ONPROC - thread is being run on a processor
tflg:   T_TALLOCSTK - thread structure allocated from stk
T_PANIC - thread initiated a system panic
tpflg:  none set
tsched: TS_LOAD - thread is in memory
TS_DONT_SWAP - thread/LWP should not be swapped
pflag:  SSYS - system resident process

pc:      unix:panicsys+0x48:   call     unix:setjmp
startpc: nfssrv:reaper_thread+0x0:   save       %sp, -0xb0, %sp

unix:panicsys+0x48(0x1096af8, 0x2a107ee2f88, 0x18760b0, 0x1, , , 0x9900001603, , , , , , , , 0x1096af8, 0x2a107ee2f88)
unix:vpanic_common+0x78(0x1096af8, 0x2a107ee2f88, 0x18a6800, 0x2a107ee2a81, 0x2a107ee3280, 0x4400001606)
unix:panic+0x1c(0x1096af8, 0x31, 0x2a107ee31e0, 0x70, 0x0, 0x30022dec1a8, 0x1831aa0)
unix:die+0x78(0x31, 0x2a107ee31e0, 0x70, 0x0)
unix:trap+0x9e4(0x2a107ee31e0, 0x70)
unix:ktl0+0x64()
-- trap data  type: 0x31 (data access MMU miss)  rp: 0x2a107ee31e0  --
addr: 0x70
pc:  0x7a61b944 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88:   ldub  [%l6 + 0x70], %l1
npc: 0x7a61b948 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x8c:   ldub  [%l3 + 0xd54], %l2
global:                       %g1         0x7a617cbc
%g2                0x3  %g3                0x3%g4      0x30051198280  %g5                0x4%g6               0x6d  %g7      0x2a107ee3ca0out:  %o0      0x300380d1850  %o1      0x2a107ee3ca4
%o2                  0  %o3                0x4%o4            0x70800  %o5      0x300380d1870%sp      0x2a107ee2a81  %o7         0x7a632b94loc:  %l0      0x3004a7f1bd0  %l1               0x13
%l2                0x1  %l3         0x7087c000%l4            0x7087c  %l5            0x70800%l6                  0  %l7             0x1000in:   %i0      0x300380d1850  %i1      0x3003d41f280
%i2             0x1000  %i3         0xffffef87%i4         0x7082c000  %i5            0x7082c%fp      0x2a107ee2b31  %i7          0x110be98<trap>pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88()
unix:cl_flk_state_transition_notify(0x3003d41f280, 0x3, 0x8) - frame recycled
genunix:flk_set_state+0x60(0x3003d41f280, 0x8, 0x0)
genunix:flk_delete_active_lock+0x58(0x3003d41f280, 0x0)
genunix:cl_flk_remove_locks_by_sysid+0xf0(0x10004)
pxfs:void pxfs_namespace_v1::fs_ii::remove_file_locks(int,CORBA::Environment&)+0xcc(0x30022cbe980?, 0x10004?)
pxfs:__SLIP.THUNK__P(, 0x10004, 0x2a107ee36a0) - frame recycled
pxfs:void pxfs_namespace_v1::pxvfs::remove_file_locks(int)+0xf4(0x10004)
pxfs:void nlm_remove_file_locks(int)+0x18(0x10004, 0x4, 0x10000, 0x1)
nfssrv:rfs4_lo_state_destroy+0x6c(0x3004a7f1c00)
nfssrv:rfs4_dbe_destroy+0x16c(0x3004a7f1b98?)
nfssrv:rfs4_dbe_reap+0x18c(0x300398c0000, 0x5a, 0x0)
nfssrv:reaper_thread+0x104(0x300398c0000, 0x0)
unix:thread_start+0x4()
-- end of kernel thread's stack --
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> ^C

<p style="text-align: left;">
Run  help command get  full list  of  commands  which available on  SCAT.
</p>

SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> help
Initial Investigation:
analyze   coreinfo  msgbuf  panic  stack  stat
toolinfo

General Commands:
analyze  clockinfo  coreinfo  cyclic  demangle  eckstat
environ  exit       help      intr    kstat     modinfo
msgbuf   stat       symbols   taskq   ttrace    tunables

Memory Dump/Display Commands:
bigdump  buf   flip  kseg    mdump  nvlist
pdump    pkma  r     rd      rd16   rd32
rd64     rd8   rdb   rdc     rdd    rdf
rdh      rdi   rdl   rdq     rds    rdw
sdump    seg   skma  strsum  vmem   wr

Data Conversion Commands:
2base  2dec  2double  2float  2hex       2string
2time  bits  calc     decode  deltatime  pid2paddr
size

Process/Thread Information:
classtbl  panic    pid2paddr  proc  sig   thread
tlist     upcount

Stacks/Traps:
findaddr_stack  frame  regs  stack  trap

Searching Memory/Core:
findport  findval  stack  tlist

File and Filesystem Information:
autofs   bigdump  buf   cgcheck  dnlc   findfiles
getpath  inode    nfs   node     tmpfs  vfssw
vfstype

CPU and Dispatch Queues:
callout  cpu   dispq  disptbl

Memory Information:
anon  iommu   ipc      kma       kseg  kstat
map   memerr  meminfo  page      pkma  resvswap
seg   sfmmu   skma     swapinfo  vmem  whatis
xmmu

Network/IPC:
findport  ifconf  ipc     ndd   nfs   nstat
pdump     pkma    stream

Type Database Commands:
ctf    sarray  savl   sdump  shash   skma
slist  slistt  stree  stype  typedb

Lock Display Commands:
cv      lck   mutex  rwlock  sema  sleepq
tstile

Device Information Commands:
dev   qlcfc

System Files/Tunables:
etcsystem  msgbuf  name2major  path2inst  tunables  vfstab

Disassembler Commands:
codepath  dis   rdi   sim

Data Structure Consistency Checks:
anon  callout  cgcheck  dispq    dnlc  kma
seg   sfmmu    symbols  upcount  vmem

Miscellaneous Commands:
clust  door  legal     namelist  pool  project
pty    rctl  sanitize  svm       task  toolinfo
trans  zfs   zone

Tool Configuration:
base  color  refclock  scat_version  scatenv  writeUsage: help [command]
help -? regular_expression
help -g [<group>]
help all

I added some of  controls like ;
  • Memory  usage
  • Process  information
  • Memory  Errors

You can check help  output  for  more  details.

SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> memerr
errorq pci_target_queue @ 0x300014d0040:
dispatched:    0
dropped:       0
logged:        0
reserved:      0
reserve_fail:  0
committed:     0
commit_fail:   0
cancelled:     0

errorq ce_queue @ 0x300014d07c0:
dispatched:    0
dropped:       0
logged:        0
reserved:      0
reserve_fail:  0
committed:     0
commit_fail:   0
cancelled:     0

errorq ue_queue @ 0x300014d0a40:
dispatched:    0
dropped:       0
logged:        0
reserved:      0
reserve_fail:  0
committed:     0
commit_fail:   0
cancelled:     0

errorq fm_ereport_queue @ 0x300014d1440:
dispatched:    0
dropped:       0
logged:        0
reserved:      0
reserve_fail:  0
committed:     0
commit_fail:   0
cancelled:     0
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> meminfo
pages        bytes
physinstalled               4177920  34225520640 (31.8G)
physmem                     4172567  34181668864 (31.8G)
total_pages                 4069191  33334812672 (31G)

freemem                     3171275  25979084800 (24.1G)
avefree                     3169686  25966067712 (24.1G)
avefree30                   3170063  25969156096 (24.1G)
availrmem (nonswapable)     3788115  31032238080 (28.9G)
availrmem_initial           4069191  33334812672 (31G)
swapfs_minfree               521570   4272701440 (3.97G)
sw_pending_size                                0 (0)

lotsfree                      63581    520855552 (496M)
desfree                       31790    260423680 (248M)
minfree                       15895    130211840 (124M)
throttlefree                  15895    130211840 (124M)

pp_kernel(calculated)        377838   3095248896 (2.88G)
obp_pages                      3270     26787840 (25.5M)
pages_locked                    339      2777088 (2.64M)

kcage_freemem                 21381    175153152 (167M)
kcage_needfree                    0            0 (0)
kcage_lotsfree                15895    130211840 (124M)
kcage_desfree                  7947     65101824 (62M)
kcage_minfree                  7947     65101824 (62M)
kcage_throttlefree             3973     32546816 (31M)

shared memory (SM)                        852048 (832K)
intimate SM (ISM)                              0 (0)
dynamic ISM (DISM)                             0 (0)
locked DISM                       0            0 (0)
total locked SM                                0 (0) (0.00% of memory)
spt_used (ISM)                    0            0 (0)
segspt_minfree                    0            0 (0)

anoninfo: (physical == disk-backed)
ani_max - total reservable physical swap                6291557 pages (48G)
ani_free - unallocated physical and memory              5837739 pages (44.5G)
ani_phys_resv - reserved physical                        492932 pages (3.76G)
ani_mem_resv - reserved memory                                0 pages (0)
ani_locked_swap - swap locked in reserved mem swap            0 pages (0)

initial virtual swap available for reservation            9839178 pages (75G)
ani_max + MAX((availrmem_initial - swapfs_minfree), 0)
current virtual swap available for reservation            9065170 pages (69.1G)
(ani_max - ani_phys_resv) + MAX((availrmem - swapfs_minfree), 0)

swap device                    pages           free
/dev/dsk/c1t0d0s1      6291557 (48G)   6291557 (48G)

tmpfs:
tmount          size  mount point
0x600408b3208  2.04M  /etc/svc/volatile
0x60049434718  65.2M  /tmp
0x60049571310   168K  /var/run

ramdisk: (none)
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> proc
addr       PID    PPID   RUID/UID     size      RSS     swresv   time  command
============= ====== ====== ========== ========== ======== ======== ====== =========
0x30022e41920  25301   1121          0    6316032  5210112   884736     10 /usr/lib/ssh/sshd
0x300574646b8  24710    704          0   12509184  7168000  1736704     63 /usr/sfw/sbin/smbd -D
0x6005e3970b0  23371  23369  151611447    5079040  3760128   991232     48 /usr/lib/ssh/sftp-server
0x300369eda38  23369   5898  151611447    1425408  1368064   196608      1 sh -c /usr/lib/ssh/sftp-server
0x300598df2f8  20312  20310 2129141062    3776512  1703936   229376      2 -sh
0x30024c16570  20310  19983 2129141062   15261696  4882432  1572864      1 /usr/lib/ssh/sshd
0x3002856d9e0  19983   1121          0   13377536  7675904  1572864     24 /usr/lib/ssh/sshd
0x300285659e8  21504  21503 1667533448    4882432  2768896   794624      4 /usr/lib/ssh/sftp-server
0x30022e3c0f8  21503  14653 1667533448    1425408  1368064   196608      1 sh -c /usr/lib/ssh/sftp-server
0x30022e36548  19126    887          0   13737984 10321920  1900544     45 /usr/sbin/rpc.metacld
0x3002cba01f0  14877   6123          0   16105472 10625024  2572288    252 /usr/ecc/exec/mstragent -s
0x3002204c090  13079   4214        100    5644288  3227648   655360      5 bash
0x30040457a78   8463  18478        100  126468096 90005504 29532160    153 siebsess 22 1 /global/sbsiebcrmcc/siebsrvr/admin/ENT_CCPROD.TESTSUNCL.shm 0 enu
0x300220340a8   7095   4628          0    2736128  2555904  1187840    188 vi /var/adm/messages
0x30001cb04a8   4628   4583          0    3301376  2924544   614400      7 bash
0x30037c5fa50   4583   4535          0    1458176  1433600   204800     14 -sh
0x30024c0a588   4535   4530          0    1458176  1433600   204800     13 -sh
0x30051fbd2c8   4530  23788 1548235000    1458176  1417216   204800      2 -sh
0x3002204d8c0   4214  10519        100    5644288  3219456   655360      6 bash
addr       PID    PPID   RUID/UID     size      RSS     swresv   time  command
============= ====== ====== ========== ========== ======== ======== ====== =========
0x30040452250   4124  23861          0    3301376  2924544   614400     12 bash
0x300220524d0  29319  29318          0    8331264  5677056  1097728      7 /opt/SUNWscnfs/bin/nfs_probe -R nfs1-server-rs -T SUNW.nfs:3.2 -G nfs1-rg
0x300220380a0  29318   7167          0    1425408  1376256   196608      4 /bin/sh -c /opt/SUNWscnfs/bin/nfs_probe -R nfs1-server-rs -T SUNW.nfs:3.2 -G nf
0x600443be018  29316   7167          0    9576448  7028736  2334720     25 /opt/SUNWscnfs/bin/nfs_daemons_probe -R nfs1-server-rs -T SUNW.nfs:3.2 -G nfs1-
0x3003ac46678  29258      1          1    2531328  2146304   745472      4 /usr/lib/nfs/lockd
0x300285841a0  29249      1          1    6078464  2637824  1146880      2 /usr/lib/nfs/statd -p /global/nfs1/SUNW.nfs -a 10.4.18.163
0x30027cc3a08  29158      1          1    2834432  2572288  1015808  11230 /usr/lib/nfs/nfsd -s /global/nfs1/SUNW.nfs
0x30022e5c0d8  29134      1          0    4349952  2899968  1138688      2 /usr/lib/nfs/mountd
0x30022e440e8  23861  23794          0    3301376  2916352   614400     12 bash
0x30027cc21d8  23794  23790          0    1458176  1433600   204800     14 -sh
0x3003ac45a60  23790  23788 1548235000    1458176  1409024   204800      2 -sh
0x30001cac4b0  23788  23743 1548235000   10706944  4505600  1540096     52 /usr/lib/ssh/sshd
Tagged In:

I'm a IT Infrastructure and Operations Architect with extensive experience and administration skills and works for Turk Telekom. I provide hardware and software support for the IT Infrastructure and Operations tasks.

205 Total Posts
Follow Me