At this post I ‘ll provide the information how to analyze crash dump file for Oracle Solaris.It helps to find out the root cause of crash dump also hang. If your system installed properly then there must be a separated raw device which size is as much as physical memory. Dump device size is related to physical memory size. It’s important to assign dedicated dump device no less than size of physical memory.Otherwise,when system gets a panic, memory dump won’t be written dump device because of lack of free size in dump device.You ‘ll lose last memory snapshot to dig root cause of problem.
First of all we need to check configuratin if system dump device configured properly.
#dumpadm Dump content: kernel pages Dump device: /dev/dsk/c1t0d0s1 (swap) Savecore directory: /var/crash/TESTSERVER Savecore enabled: yes #swap -l swapfile dev swaplo blocks free /dev/dsk/c1t0d0s1 32,9 16 100664912 100664912 #prtconf |grep -i mem Memory size: 32640 Megabytes
After check system dump device configuration you can check if server got a dump file while panics.
#ls -ld /var/crash/TESTSERVER/vmcore* -rw-r--r-- 1 root root 3519586304 Oct 29 16:57 /var/crash/TESTSERVER/vmcore.8
If you ‘ll make a test on your test server and you need to a crash dump file, apply this steps to create a dump file.It’ll give panic and get a snapshot of memory.
reboot -d Then you need a server which is same architecture(Sparc or X86 which depends on your server) and SCAT package is installed. If your dump file generated on a sparc machine so you need a sparc machine which SCAT installed. Login MOS and download “Oracle Solaris Crash Analysis Tool”.The patch ID is “p10364262_520_SOLARIS64”. Then install SCAT.
#unzip p10364262_520_SOLARIS64.zip Archive: p10364262_520_SOLARIS64.zip inflating: SUNWscat5.2-GA-combined.pkg.gz inflating: SUNWscat5.2-GA-i386.pkg.gz inflating: SUNWscat5.2-GA-sparc.pkg.gz inflating: license_agreement1.html inflating: readme.html root@TEST# gunzip SUNWscat5.2-GA-sparc.pkg.gz root@TEST# uname -s SunOS root@TEST# uname -a SunOS TEST 5.10 Generic_147147-26 sun4u sparc SUNW,Sun-Fire-V245 root@TEST# pkgadd -d /tmp/SUNWscat5.2-GA-sparc.pkg
While installing package you should accept installation.”Please enter your response to the license: ACCEPT”. After installation Scat finishes then you need to copy vmcore.x file to the machine which you installed SCAT. Start analyze with this command .
# /opt/SUNWscat/bin/scat unix.3 /CASESUP/EXPECT/vmcore.11
After scat finish operation run “kstat xck”.
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> analyze SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> panic
Check this logs; and also run “panic” to see what happened why server got panic.
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> analyze SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> panic
The strings which you need to check are;
- Panic Strings:
- panic string: BAD TRAP: type=31 rp=2a107ee31e0 addr=70 mmu_fsr=0 occurred in module “pxfs” due to a NULL pointer dereference
- Trap:
- — trap data type: 0x31 (data access MMU miss) rp: 0x2a107ee31e0 —
- Trap Strings:
- <trap>pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88()
After you get all dates abaut panic now you can search google and MOS for more informations and bugs.
#/opt/SUNWscat/bin/scat unix.3 /CASESUP/EXPECT/vmcore.11</p> Solaris[TM] CAT 5.2 for Solaris 10 64-bit UltraSPARC SV4990M, Aug 26 2009 Copyright © 2009 Sun Microsystems, Inc. All rights reserved. Patents Pending. Use is subject to license terms. Sun Microsystems, Inc. has intellectual property rights relating to technology embodied in the product that is described in this document. In particular, and without limitation, these intellectual property rights may include one or more of the U.S. patents listed at http://www.sun.com/patents and one or more additional patents or pending patent applications in the U.S. and in other countries. U.S. Government Rights - Commercial software. Government users are subject to the Sun Microsystems, Inc. standard license agreement and applicable provisions of the FAR and its supplements. Use is subject to license terms. Portions may be derived from Berkeley BSD systems, licensed from U. of CA. Sun, Sun Microsystems, the Sun logo, Java, Solaris, SunSpectrum and SunTone are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. and other countries. Feedback regarding the tool should be sent to SolarisCAT_Feedback@Sun.COM Visit the Solaris CAT blog at http://blogs.sun.com/SolarisCAT opening /CASESUP/EXPECT/vmcore.11 ...dumphdr...symtab...core...done loading core data: modules...symbols...CTF...done core file: /CASESUP/EXPECT/vmcore.11 user: Super-User (root:0) release: 5.10 (64-bit) version: Generic_127127-11 machine: sun4v node name: TESTSERVER hw_provider: Sun_Microsystems system type: SUNW,SPARC-Enterprise-T5220 (UltraSPARC-T2) hostid: 84ebae8a dump_conflags: 0x10000 (DUMP_KERNEL) on /dev/dsk/c1t0d0s1(48G) cluster_bflgs: 0x3 (CLUSTER_CONFIGURED|CLUSTER_BOOTED) pxfs_software_mount_level: v1 (consolidated version) current node: 1 time of crash: Thu Oct 29 20:14:31 EET 2015 age of system: 3 hours 21 minutes 29.68 seconds panic CPU: 60 (64 CPUs, 31.8G memory) panic string: BAD TRAP: type=31 rp=2a107ee31e0 addr=70 mmu_fsr=0 occurred in module "pxfs" due to a NULL pointer dereference sanity checks: settings... NOTE: /etc/system: module ge not loaded for "set ge:ge_intr_mode=0x833" vmem...CPU...sysent... WARNING: unknown module acctctl seen 4 times in sysent table clock... WARNING: hres_last_tick 4 minutes 19.874264183 seconds later misc...done SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> kstat xck loaded 35 kstat checks from /opt/SUNWscat/lib/kstat_xck SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> /opt/SUNWscat/lib/kstat_xck /opt/SUNWscat/exec/scat510U: /opt/SUNWscat/lib/kstat_xck: cannot execute SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> analyze core file: /CASESUP/EXPECT/vmcore.11 user: Super-User (root:0) release: 5.10 (64-bit) version: Generic_127127-11 machine: sun4v node name: TESTSERVER hw_provider: Sun_Microsystems system type: SUNW,SPARC-Enterprise-T5220 (UltraSPARC-T2) hostid: 84ebae8a dump_conflags: 0x10000 (DUMP_KERNEL) on /dev/dsk/c1t0d0s1(48G) cluster_bflgs: 0x3 (CLUSTER_CONFIGURED|CLUSTER_BOOTED) pxfs_software_mount_level: v1 (consolidated version) current node: 1 time of crash: Thu Oct 29 20:14:31 EET 2015 age of system: 3 hours 21 minutes 29.68 seconds panic CPU: 60 (64 CPUs, 31.8G memory) panic string: BAD TRAP: type=31 rp=2a107ee31e0 addr=70 mmu_fsr=0 occurred in module "pxfs" due to a NULL pointer dereference ==== checking for trap information ==== CPU 60 had the panic ==== panic thread: 0x2a107ee3ca0 ==== CPU: 60 ==== ==== panic kernel thread: 0x2a107ee3ca0 PID: 0 on CPU: 60 affinity CPU: 60 ==== cmd: sched t_procp: 0x18594e0(proc_sched) p_as: 0x185a8f8(kas) zone: global t_stk: 0x2a107ee3a90 sp: 0x18756e1 t_stkbase: 0x2a107ede000 t_pri: 60(SYS) pctcpu: 0.000000 t_lwp: 0x0 psrset: 0 last CPU: 60 idle: 9000 ticks (1 minutes 30.00 seconds) start: Thu Oct 29 17:55:01 2015 age: 8370 seconds (2 hours 19 minutes 30 seconds) tstate: TS_ONPROC - thread is being run on a processor tflg: T_TALLOCSTK - thread structure allocated from stk T_PANIC - thread initiated a system panic tpflg: none set tsched: TS_LOAD - thread is in memory TS_DONT_SWAP - thread/LWP should not be swapped pflag: SSYS - system resident process pc: unix:panicsys+0x48: call unix:setjmp startpc: nfssrv:reaper_thread+0x0: save %sp, -0xb0, %sp unix:panicsys+0x48(0x1096af8, 0x2a107ee2f88, 0x18760b0, 0x1, , , 0x9900001603, , , , , , , , 0x1096af8, 0x2a107ee2f88) unix:vpanic_common+0x78(0x1096af8, 0x2a107ee2f88, 0x18a6800, 0x2a107ee2a81, 0x2a107ee3280, 0x4400001606) unix:panic+0x1c(0x1096af8, 0x31, 0x2a107ee31e0, 0x70, 0x0, 0x30022dec1a8, 0x1831aa0) unix:die+0x78(0x31, 0x2a107ee31e0, 0x70, 0x0) unix:trap+0x9e4(0x2a107ee31e0, 0x70) unix:ktl0+0x64() -- trap data type: 0x31 (data access MMU miss) rp: 0x2a107ee31e0 -- addr: 0x70 pc: 0x7a61b944 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88: ldub [%l6 + 0x70], %l1 npc: 0x7a61b948 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x8c: ldub [%l3 + 0xd54], %l2 global: %g1 0x7a617cbc %g2 0x3 %g3 0x3%g4 0x30051198280 %g5 0x4%g6 0x6d %g7 0x2a107ee3ca0out: %o0 0x300380d1850 %o1 0x2a107ee3ca4 %o2 0 %o3 0x4%o4 0x70800 %o5 0x300380d1870%sp 0x2a107ee2a81 %o7 0x7a632b94loc: %l0 0x3004a7f1bd0 %l1 0x13 %l2 0x1 %l3 0x7087c000%l4 0x7087c %l5 0x70800%l6 0 %l7 0x1000in: %i0 0x300380d1850 %i1 0x3003d41f280 %i2 0x1000 %i3 0xffffef87%i4 0x7082c000 %i5 0x7082c%fp 0x2a107ee2b31 %i7 0x110be98<trap>pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88() unix:cl_flk_state_transition_notify(0x3003d41f280, 0x3, 0x8) - frame recycled genunix:flk_set_state+0x60(0x3003d41f280, 0x8, 0x0) genunix:flk_delete_active_lock+0x58(0x3003d41f280, 0x0) genunix:cl_flk_remove_locks_by_sysid+0xf0(0x10004) pxfs:void pxfs_namespace_v1::fs_ii::remove_file_locks(int,CORBA::Environment&)+0xcc(0x30022cbe980?, 0x10004?) pxfs:__SLIP.THUNK__P(, 0x10004, 0x2a107ee36a0) - frame recycled pxfs:void pxfs_namespace_v1::pxvfs::remove_file_locks(int)+0xf4(0x10004) pxfs:void nlm_remove_file_locks(int)+0x18(0x10004, 0x4, 0x10000, 0x1) nfssrv:rfs4_lo_state_destroy+0x6c(0x3004a7f1c00) nfssrv:rfs4_dbe_destroy+0x16c(0x3004a7f1b98?) nfssrv:rfs4_dbe_reap+0x18c(0x300398c0000, 0x5a, 0x0) nfssrv:reaper_thread+0x104(0x300398c0000, 0x0) unix:thread_start+0x4() -- end of kernel thread's stack -- ==== analyzing panic thread stack for trap frames ==== ==== using trap() frame 1 @ 0x2a107ee2fe0, rp(%i0): 0x2a107ee31e0 ==== type(%l2): 0x31 (data access MMU miss) pc: 0x7a61b944 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88: ldub [%l6 + 0x70], %l1 npc: 0x7a61b948 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x8c: ldub [%l3 + 0xd54], %l2 global: %g1 pxfs:cl_flk_state_transition_notify+0x0 %g2 0x3 %g3 0x3%g4 0x30051198280 %g5 0x4%g6 0x6d %g7 0x2a107ee3ca0out: %o0 0x300380d1850 %o1 0x2a107ee3ca4 %o2 0 %o3 0x4%o4 0x70800 %o5 0x300380d1870%sp 0x2a107ee2a81 %o7 pxfs:void flk_state_transition_notify(lock_descriptor*,int,int)+0x130loc: %l0 0x3004a7f1bd0 %l1 0x13 %l2 0x1 %l3 pxfs(data): 0x7087c000%l4 0x7087c %l5 0x70800%l6 0 %l7 0x1000in: %i0 0x300380d1850 %i1 0x3003d41f280 %i2 0x1000 %i3 0xffffef87%i4 cl_dcs(data):PXFS_GREEN+0x0 %i5 0x7082c%fp 0x2a107ee2b31 %i7 genunix:flk_set_state+0x60 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x60: sra %o7, 0x0, %o4 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x64: add %g3, 0x96f, %o1 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x68: sra %l0, 0x0, %o3 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x6c: sra %o5, 0x0, %o5 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x70: call cl_load:void dbg_print_buf::dbprintf(char*,...) pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x74: or %g0, %i0, %o2 ( mov %i0, %o2 ) pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x78: 1: sethi %hi(0x70800) %l5 ( set 0x70800, %l5 ) pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x7c: ldx [%i0 + 0x30], %l6 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x80: or %l5, 0x7c, %l4 (0x7087c) pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x84: sllx %l4, 0xc, %l3 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88: ldub [%l6 + 0x70], %l1 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x8c: ldub [%l3 + 0xd54], %l2 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x90: orcc %l1, %l2, %g0 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x94: bne,pn %icc, pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xc8 (2f) pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x98: or %g0, %i0, %o0 ( mov %i0, %o0 ) pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x9c: call pxfs:repl_pxfs::lock_info_t*fobj_ii::get_lock_info(lock_descriptor*) pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xa0: or %g0, %i1, %o1 ( mov %i1, %o1 ) pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xa4: or %g0, 0x1, %i1 ( mov 0x1, %i1 ) pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xa8: or %g0, %o0, %l7 ( mov %o0, %l7 ) pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xac: stw %i1, [%o0 + 0x60] pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xb0: or %g0, 0x10, %o0 ( mov 0x10, %o0 ) SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> panic panic on CPU 60 panic string: BAD TRAP: type=31 rp=2a107ee31e0 addr=70 mmu_fsr=0 occurred in module "pxfs" due to a NULL pointer dereference ==== panic kernel thread: 0x2a107ee3ca0 PID: 0 on CPU: 60 affinity CPU: 60 ==== cmd: sched t_procp: 0x18594e0(proc_sched) p_as: 0x185a8f8(kas) zone: global t_stk: 0x2a107ee3a90 sp: 0x18756e1 t_stkbase: 0x2a107ede000 t_pri: 60(SYS) pctcpu: 0.000000 t_lwp: 0x0 psrset: 0 last CPU: 60 idle: 9000 ticks (1 minutes 30.00 seconds) start: Thu Oct 29 17:55:01 2015 age: 8370 seconds (2 hours 19 minutes 30 seconds) tstate: TS_ONPROC - thread is being run on a processor tflg: T_TALLOCSTK - thread structure allocated from stk T_PANIC - thread initiated a system panic tpflg: none set tsched: TS_LOAD - thread is in memory TS_DONT_SWAP - thread/LWP should not be swapped pflag: SSYS - system resident process pc: unix:panicsys+0x48: call unix:setjmp startpc: nfssrv:reaper_thread+0x0: save %sp, -0xb0, %sp unix:panicsys+0x48(0x1096af8, 0x2a107ee2f88, 0x18760b0, 0x1, , , 0x9900001603, , , , , , , , 0x1096af8, 0x2a107ee2f88) unix:vpanic_common+0x78(0x1096af8, 0x2a107ee2f88, 0x18a6800, 0x2a107ee2a81, 0x2a107ee3280, 0x4400001606) unix:panic+0x1c(0x1096af8, 0x31, 0x2a107ee31e0, 0x70, 0x0, 0x30022dec1a8, 0x1831aa0) unix:die+0x78(0x31, 0x2a107ee31e0, 0x70, 0x0) unix:trap+0x9e4(0x2a107ee31e0, 0x70) unix:ktl0+0x64() -- trap data type: 0x31 (data access MMU miss) rp: 0x2a107ee31e0 -- addr: 0x70 pc: 0x7a61b944 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88: ldub [%l6 + 0x70], %l1 npc: 0x7a61b948 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x8c: ldub [%l3 + 0xd54], %l2 global: %g1 0x7a617cbc %g2 0x3 %g3 0x3%g4 0x30051198280 %g5 0x4%g6 0x6d %g7 0x2a107ee3ca0out: %o0 0x300380d1850 %o1 0x2a107ee3ca4 %o2 0 %o3 0x4%o4 0x70800 %o5 0x300380d1870%sp 0x2a107ee2a81 %o7 0x7a632b94loc: %l0 0x3004a7f1bd0 %l1 0x13 %l2 0x1 %l3 0x7087c000%l4 0x7087c %l5 0x70800%l6 0 %l7 0x1000in: %i0 0x300380d1850 %i1 0x3003d41f280 %i2 0x1000 %i3 0xffffef87%i4 0x7082c000 %i5 0x7082c%fp 0x2a107ee2b31 %i7 0x110be98<trap>pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88() unix:cl_flk_state_transition_notify(0x3003d41f280, 0x3, 0x8) - frame recycled genunix:flk_set_state+0x60(0x3003d41f280, 0x8, 0x0) genunix:flk_delete_active_lock+0x58(0x3003d41f280, 0x0) genunix:cl_flk_remove_locks_by_sysid+0xf0(0x10004) pxfs:void pxfs_namespace_v1::fs_ii::remove_file_locks(int,CORBA::Environment&)+0xcc(0x30022cbe980?, 0x10004?) pxfs:__SLIP.THUNK__P(, 0x10004, 0x2a107ee36a0) - frame recycled pxfs:void pxfs_namespace_v1::pxvfs::remove_file_locks(int)+0xf4(0x10004) pxfs:void nlm_remove_file_locks(int)+0x18(0x10004, 0x4, 0x10000, 0x1) nfssrv:rfs4_lo_state_destroy+0x6c(0x3004a7f1c00) nfssrv:rfs4_dbe_destroy+0x16c(0x3004a7f1b98?) nfssrv:rfs4_dbe_reap+0x18c(0x300398c0000, 0x5a, 0x0) nfssrv:reaper_thread+0x104(0x300398c0000, 0x0) unix:thread_start+0x4() -- end of kernel thread's stack -- SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> ^C <p style="text-align: left;"> Run help command get full list of commands which available on SCAT. </p> SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> help Initial Investigation: analyze coreinfo msgbuf panic stack stat toolinfo General Commands: analyze clockinfo coreinfo cyclic demangle eckstat environ exit help intr kstat modinfo msgbuf stat symbols taskq ttrace tunables Memory Dump/Display Commands: bigdump buf flip kseg mdump nvlist pdump pkma r rd rd16 rd32 rd64 rd8 rdb rdc rdd rdf rdh rdi rdl rdq rds rdw sdump seg skma strsum vmem wr Data Conversion Commands: 2base 2dec 2double 2float 2hex 2string 2time bits calc decode deltatime pid2paddr size Process/Thread Information: classtbl panic pid2paddr proc sig thread tlist upcount Stacks/Traps: findaddr_stack frame regs stack trap Searching Memory/Core: findport findval stack tlist File and Filesystem Information: autofs bigdump buf cgcheck dnlc findfiles getpath inode nfs node tmpfs vfssw vfstype CPU and Dispatch Queues: callout cpu dispq disptbl Memory Information: anon iommu ipc kma kseg kstat map memerr meminfo page pkma resvswap seg sfmmu skma swapinfo vmem whatis xmmu Network/IPC: findport ifconf ipc ndd nfs nstat pdump pkma stream Type Database Commands: ctf sarray savl sdump shash skma slist slistt stree stype typedb Lock Display Commands: cv lck mutex rwlock sema sleepq tstile Device Information Commands: dev qlcfc System Files/Tunables: etcsystem msgbuf name2major path2inst tunables vfstab Disassembler Commands: codepath dis rdi sim Data Structure Consistency Checks: anon callout cgcheck dispq dnlc kma seg sfmmu symbols upcount vmem Miscellaneous Commands: clust door legal namelist pool project pty rctl sanitize svm task toolinfo trans zfs zone Tool Configuration: base color refclock scat_version scatenv writeUsage: help [command] help -? regular_expression help -g [<group>] help all I added some of controls like ;
- Memory usage
- Process information
- Memory Errors
You can check help output for more details.
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> memerr
errorq pci_target_queue @ 0x300014d0040:
dispatched: 0
dropped: 0
logged: 0
reserved: 0
reserve_fail: 0
committed: 0
commit_fail: 0
cancelled: 0
errorq ce_queue @ 0x300014d07c0:
dispatched: 0
dropped: 0
logged: 0
reserved: 0
reserve_fail: 0
committed: 0
commit_fail: 0
cancelled: 0
errorq ue_queue @ 0x300014d0a40:
dispatched: 0
dropped: 0
logged: 0
reserved: 0
reserve_fail: 0
committed: 0
commit_fail: 0
cancelled: 0
errorq fm_ereport_queue @ 0x300014d1440:
dispatched: 0
dropped: 0
logged: 0
reserved: 0
reserve_fail: 0
committed: 0
commit_fail: 0
cancelled: 0
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> meminfo
pages bytes
physinstalled 4177920 34225520640 (31.8G)
physmem 4172567 34181668864 (31.8G)
total_pages 4069191 33334812672 (31G)
freemem 3171275 25979084800 (24.1G)
avefree 3169686 25966067712 (24.1G)
avefree30 3170063 25969156096 (24.1G)
availrmem (nonswapable) 3788115 31032238080 (28.9G)
availrmem_initial 4069191 33334812672 (31G)
swapfs_minfree 521570 4272701440 (3.97G)
sw_pending_size 0 (0)
lotsfree 63581 520855552 (496M)
desfree 31790 260423680 (248M)
minfree 15895 130211840 (124M)
throttlefree 15895 130211840 (124M)
pp_kernel(calculated) 377838 3095248896 (2.88G)
obp_pages 3270 26787840 (25.5M)
pages_locked 339 2777088 (2.64M)
kcage_freemem 21381 175153152 (167M)
kcage_needfree 0 0 (0)
kcage_lotsfree 15895 130211840 (124M)
kcage_desfree 7947 65101824 (62M)
kcage_minfree 7947 65101824 (62M)
kcage_throttlefree 3973 32546816 (31M)
shared memory (SM) 852048 (832K)
intimate SM (ISM) 0 (0)
dynamic ISM (DISM) 0 (0)
locked DISM 0 0 (0)
total locked SM 0 (0) (0.00% of memory)
spt_used (ISM) 0 0 (0)
segspt_minfree 0 0 (0)
anoninfo: (physical == disk-backed)
ani_max - total reservable physical swap 6291557 pages (48G)
ani_free - unallocated physical and memory 5837739 pages (44.5G)
ani_phys_resv - reserved physical 492932 pages (3.76G)
ani_mem_resv - reserved memory 0 pages (0)
ani_locked_swap - swap locked in reserved mem swap 0 pages (0)
initial virtual swap available for reservation 9839178 pages (75G)
ani_max + MAX((availrmem_initial - swapfs_minfree), 0)
current virtual swap available for reservation 9065170 pages (69.1G)
(ani_max - ani_phys_resv) + MAX((availrmem - swapfs_minfree), 0)
swap device pages free
/dev/dsk/c1t0d0s1 6291557 (48G) 6291557 (48G)
tmpfs:
tmount size mount point
0x600408b3208 2.04M /etc/svc/volatile
0x60049434718 65.2M /tmp
0x60049571310 168K /var/run
ramdisk: (none)
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> proc
addr PID PPID RUID/UID size RSS swresv time command
============= ====== ====== ========== ========== ======== ======== ====== =========
0x30022e41920 25301 1121 0 6316032 5210112 884736 10 /usr/lib/ssh/sshd
0x300574646b8 24710 704 0 12509184 7168000 1736704 63 /usr/sfw/sbin/smbd -D
0x6005e3970b0 23371 23369 151611447 5079040 3760128 991232 48 /usr/lib/ssh/sftp-server
0x300369eda38 23369 5898 151611447 1425408 1368064 196608 1 sh -c /usr/lib/ssh/sftp-server
0x300598df2f8 20312 20310 2129141062 3776512 1703936 229376 2 -sh
0x30024c16570 20310 19983 2129141062 15261696 4882432 1572864 1 /usr/lib/ssh/sshd
0x3002856d9e0 19983 1121 0 13377536 7675904 1572864 24 /usr/lib/ssh/sshd
0x300285659e8 21504 21503 1667533448 4882432 2768896 794624 4 /usr/lib/ssh/sftp-server
0x30022e3c0f8 21503 14653 1667533448 1425408 1368064 196608 1 sh -c /usr/lib/ssh/sftp-server
0x30022e36548 19126 887 0 13737984 10321920 1900544 45 /usr/sbin/rpc.metacld
0x3002cba01f0 14877 6123 0 16105472 10625024 2572288 252 /usr/ecc/exec/mstragent -s
0x3002204c090 13079 4214 100 5644288 3227648 655360 5 bash
0x30040457a78 8463 18478 100 126468096 90005504 29532160 153 siebsess 22 1 /global/sbsiebcrmcc/siebsrvr/admin/ENT_CCPROD.TESTSUNCL.shm 0 enu
0x300220340a8 7095 4628 0 2736128 2555904 1187840 188 vi /var/adm/messages
0x30001cb04a8 4628 4583 0 3301376 2924544 614400 7 bash
0x30037c5fa50 4583 4535 0 1458176 1433600 204800 14 -sh
0x30024c0a588 4535 4530 0 1458176 1433600 204800 13 -sh
0x30051fbd2c8 4530 23788 1548235000 1458176 1417216 204800 2 -sh
0x3002204d8c0 4214 10519 100 5644288 3219456 655360 6 bash
addr PID PPID RUID/UID size RSS swresv time command
============= ====== ====== ========== ========== ======== ======== ====== =========
0x30040452250 4124 23861 0 3301376 2924544 614400 12 bash
0x300220524d0 29319 29318 0 8331264 5677056 1097728 7 /opt/SUNWscnfs/bin/nfs_probe -R nfs1-server-rs -T SUNW.nfs:3.2 -G nfs1-rg
0x300220380a0 29318 7167 0 1425408 1376256 196608 4 /bin/sh -c /opt/SUNWscnfs/bin/nfs_probe -R nfs1-server-rs -T SUNW.nfs:3.2 -G nf
0x600443be018 29316 7167 0 9576448 7028736 2334720 25 /opt/SUNWscnfs/bin/nfs_daemons_probe -R nfs1-server-rs -T SUNW.nfs:3.2 -G nfs1-
0x3003ac46678 29258 1 1 2531328 2146304 745472 4 /usr/lib/nfs/lockd
0x300285841a0 29249 1 1 6078464 2637824 1146880 2 /usr/lib/nfs/statd -p /global/nfs1/SUNW.nfs -a 10.4.18.163
0x30027cc3a08 29158 1 1 2834432 2572288 1015808 11230 /usr/lib/nfs/nfsd -s /global/nfs1/SUNW.nfs
0x30022e5c0d8 29134 1 0 4349952 2899968 1138688 2 /usr/lib/nfs/mountd
0x30022e440e8 23861 23794 0 3301376 2916352 614400 12 bash
0x30027cc21d8 23794 23790 0 1458176 1433600 204800 14 -sh
0x3003ac45a60 23790 23788 1548235000 1458176 1409024 204800 2 -sh
0x30001cac4b0 23788 23743 1548235000 10706944 4505600 1540096 52 /usr/lib/ssh/sshd
