Crash Dump Analyse with Scat Oracle Solaris


At this post I ‘ll provide  the  information how to  analyze crash dump  file for  Oracle Solaris.It helps  to find  out the  root  cause  of crash dump also  hang. If your system installed  properly then there  must be a separated  raw device which  size is as  much as  physical memory. Dump device size  is  related to physical memory size. It’s  important  to assign  dedicated dump device  no less than size  of physical memory.Otherwise,when system gets a  panic, memory dump won’t  be written  dump device  because  of lack  of  free size  in dump device.You ‘ll lose  last  memory snapshot to dig  root cause  of  problem.

First  of  all we  need to check configuratin  if system dump device  configured  properly.

#dumpadm

Dump content: kernel pages
Dump device: /dev/dsk/c1t0d0s1 (swap)
Savecore directory: /var/crash/TESTSERVER
Savecore enabled: yes
#swap -l
swapfile dev swaplo blocks free
/dev/dsk/c1t0d0s1 32,9 16 100664912 100664912
#prtconf |grep -i mem
Memory size: 32640 Megabytes

After  check system  dump device  configuration you can check if server  got a dump file  while panics.

#ls -ld /var/crash/TESTSERVER/vmcore*
-rw-r--r-- 1 root root 3519586304 Oct 29 16:57 /var/crash/TESTSERVER/vmcore.8

If  you ‘ll  make a test  on your test server  and you need  to a crash dump file,  apply  this  steps  to create a  dump  file.It’ll give panic and  get a snapshot  of  memory.

reboot  -d

Then  you need a server  which is same  architecture(Sparc or  X86 which depends  on your  server) and  SCAT  package  is  installed. If your  dump file generated  on a  sparc machine so  you need a sparc machine which  SCAT installed. Login  MOS and  download “Oracle Solaris Crash Analysis Tool”.The  patch  ID  is “p10364262_520_SOLARIS64”. Then install  SCAT.
#unzip p10364262_520_SOLARIS64.zip
Archive: p10364262_520_SOLARIS64.zip
inflating: SUNWscat5.2-GA-combined.pkg.gz
inflating: SUNWscat5.2-GA-i386.pkg.gz
inflating: SUNWscat5.2-GA-sparc.pkg.gz
inflating: license_agreement1.html
inflating: readme.html
root@TEST# gunzip SUNWscat5.2-GA-sparc.pkg.gz
root@TEST# uname -s
SunOS
root@TEST# uname -a
SunOS TEST 5.10 Generic_147147-26 sun4u sparc SUNW,Sun-Fire-V245
root@TEST# pkgadd -d /tmp/SUNWscat5.2-GA-sparc.pkg

While  installing package  you should  accept  installation.”Please enter your response to the license: ACCEPT”. After  installation  Scat  finishes then you need  to copy  vmcore.x file to  the  machine which  you installed SCAT. Start  analyze  with  this command .

# /opt/SUNWscat/bin/scat unix.3  /CASESUP/EXPECT/vmcore.11 

After  scat finish  operation run “kstat  xck”.

SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> analyze

SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> panic

Check this  logs; and  also  run “panic” to see  what happened why server  got panic.

SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> analyze

SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> panic

The strings  which you need to check  are;

  • Panic Strings:
    • panic string:   BAD TRAP: type=31 rp=2a107ee31e0 addr=70 mmu_fsr=0 occurred in module “pxfs” due to a NULL pointer dereference
  • Trap:
    • — trap data  type: 0x31 (data access MMU miss)  rp: 0x2a107ee31e0  —
  • Trap  Strings:
    • <trap>pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88()

After  you get  all dates  abaut  panic now you can  search  google  and  MOS for  more  informations and  bugs.

#/opt/SUNWscat/bin/scat unix.3  /CASESUP/EXPECT/vmcore.11</p>

  Solaris[TM] CAT 5.2 for Solaris 10 64-bit UltraSPARC
SV4990M, Aug 26 2009
Copyright © 2009 Sun Microsystems, Inc.  All rights reserved.
Patents Pending. Use is subject to license terms.
Sun Microsystems, Inc. has intellectual property rights relating to
technology embodied in the product that is described in this document.
In particular, and without limitation, these intellectual property
rights may include one or more of the U.S. patents listed at
http://www.sun.com/patents and one or more additional patents or
pending patent applications in the U.S. and in other countries.
U.S. Government Rights - Commercial software.  Government users
are subject to the Sun Microsystems, Inc. standard license agreement
and applicable provisions of the FAR and its supplements.
Use is subject to license terms.  Portions may be derived from
Berkeley BSD systems, licensed from U. of CA.
Sun, Sun Microsystems, the Sun logo, Java, Solaris, SunSpectrum and
SunTone are trademarks or registered trademarks of Sun Microsystems,
Inc. in the U.S. and other countries.
All SPARC trademarks are used under license and are trademarks or
registered trademarks of SPARC International, Inc. in the U.S. and
other countries.
Feedback regarding the tool should be sent to SolarisCAT_Feedback@Sun.COM
Visit the Solaris CAT blog at http://blogs.sun.com/SolarisCAT
opening /CASESUP/EXPECT/vmcore.11 ...dumphdr...symtab...core...done
loading core data: modules...symbols...CTF...done
core file:      /CASESUP/EXPECT/vmcore.11
user:           Super-User (root:0)
release:        5.10 (64-bit)
version:        Generic_127127-11
machine:        sun4v
node name:      TESTSERVER
hw_provider:    Sun_Microsystems
system type:    SUNW,SPARC-Enterprise-T5220 (UltraSPARC-T2)
hostid:         84ebae8a
dump_conflags:  0x10000 (DUMP_KERNEL) on /dev/dsk/c1t0d0s1(48G)
cluster_bflgs:  0x3 (CLUSTER_CONFIGURED|CLUSTER_BOOTED)
pxfs_software_mount_level: v1 (consolidated version)
current node:   1
time of crash:  Thu Oct 29 20:14:31 EET 2015
age of system:  3 hours 21 minutes 29.68 seconds
panic CPU:      60 (64 CPUs, 31.8G memory)
panic string:   BAD TRAP: type=31 rp=2a107ee31e0 addr=70 mmu_fsr=0 occurred in module "pxfs" due to a NULL pointer dereference

sanity checks: settings...
NOTE: /etc/system: module ge not loaded for "set ge:ge_intr_mode=0x833"
vmem...CPU...sysent...
WARNING: unknown module acctctl seen 4 times in sysent table
clock...
WARNING: hres_last_tick 4 minutes 19.874264183 seconds later
misc...done
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> kstat xck
loaded 35 kstat checks from /opt/SUNWscat/lib/kstat_xck
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> /opt/SUNWscat/lib/kstat_xck
/opt/SUNWscat/exec/scat510U: /opt/SUNWscat/lib/kstat_xck: cannot execute
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> analyze
core file:      /CASESUP/EXPECT/vmcore.11
user:           Super-User (root:0)
release:        5.10 (64-bit)
version:        Generic_127127-11
machine:        sun4v
node name:      TESTSERVER
hw_provider:    Sun_Microsystems
system type:    SUNW,SPARC-Enterprise-T5220 (UltraSPARC-T2)
hostid:         84ebae8a
dump_conflags:  0x10000 (DUMP_KERNEL) on /dev/dsk/c1t0d0s1(48G)
cluster_bflgs:  0x3 (CLUSTER_CONFIGURED|CLUSTER_BOOTED)
pxfs_software_mount_level: v1 (consolidated version)
current node:   1
time of crash:  Thu Oct 29 20:14:31 EET 2015
age of system:  3 hours 21 minutes 29.68 seconds
panic CPU:      60 (64 CPUs, 31.8G memory)
panic string:   BAD TRAP: type=31 rp=2a107ee31e0 addr=70 mmu_fsr=0 occurred in module "pxfs" due to a NULL pointer dereference
==== checking for trap information ====
CPU 60 had the panic
==== panic thread: 0x2a107ee3ca0 ==== CPU: 60 ====
==== panic kernel thread: 0x2a107ee3ca0  PID: 0  on CPU: 60  affinity CPU: 60 ====
cmd: sched
t_procp: 0x18594e0(proc_sched)
p_as: 0x185a8f8(kas)
zone: global
t_stk: 0x2a107ee3a90  sp: 0x18756e1  t_stkbase: 0x2a107ede000
t_pri: 60(SYS)  pctcpu: 0.000000
t_lwp: 0x0  psrset: 0  last CPU: 60
idle: 9000 ticks (1 minutes 30.00 seconds)
start: Thu Oct 29 17:55:01 2015
age: 8370 seconds (2 hours 19 minutes 30 seconds)
tstate: TS_ONPROC - thread is being run on a processor
tflg:   T_TALLOCSTK - thread structure allocated from stk
T_PANIC - thread initiated a system panic
tpflg:  none set
tsched: TS_LOAD - thread is in memory
TS_DONT_SWAP - thread/LWP should not be swapped
pflag:  SSYS - system resident process
pc:      unix:panicsys+0x48:   call     unix:setjmp
startpc: nfssrv:reaper_thread+0x0:   save       %sp, -0xb0, %sp
unix:panicsys+0x48(0x1096af8, 0x2a107ee2f88, 0x18760b0, 0x1, , , 0x9900001603, , , , , , , , 0x1096af8, 0x2a107ee2f88)
unix:vpanic_common+0x78(0x1096af8, 0x2a107ee2f88, 0x18a6800, 0x2a107ee2a81, 0x2a107ee3280, 0x4400001606)
unix:panic+0x1c(0x1096af8, 0x31, 0x2a107ee31e0, 0x70, 0x0, 0x30022dec1a8, 0x1831aa0)
unix:die+0x78(0x31, 0x2a107ee31e0, 0x70, 0x0)
unix:trap+0x9e4(0x2a107ee31e0, 0x70)
unix:ktl0+0x64()
-- trap data  type: 0x31 (data access MMU miss)  rp: 0x2a107ee31e0  --
addr: 0x70
pc:  0x7a61b944 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88:   ldub  [%l6 + 0x70], %l1
npc: 0x7a61b948 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x8c:   ldub  [%l3 + 0xd54], %l2
global:                       %g1         0x7a617cbc
%g2                0x3  %g3                0x3%g4      0x30051198280  %g5                0x4%g6               0x6d  %g7      0x2a107ee3ca0out:  %o0      0x300380d1850  %o1      0x2a107ee3ca4
%o2                  0  %o3                0x4%o4            0x70800  %o5      0x300380d1870%sp      0x2a107ee2a81  %o7         0x7a632b94loc:  %l0      0x3004a7f1bd0  %l1               0x13
%l2                0x1  %l3         0x7087c000%l4            0x7087c  %l5            0x70800%l6                  0  %l7             0x1000in:   %i0      0x300380d1850  %i1      0x3003d41f280
%i2             0x1000  %i3         0xffffef87%i4         0x7082c000  %i5            0x7082c%fp      0x2a107ee2b31  %i7          0x110be98<trap>pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88()
unix:cl_flk_state_transition_notify(0x3003d41f280, 0x3, 0x8) - frame recycled
genunix:flk_set_state+0x60(0x3003d41f280, 0x8, 0x0)
genunix:flk_delete_active_lock+0x58(0x3003d41f280, 0x0)
genunix:cl_flk_remove_locks_by_sysid+0xf0(0x10004)
pxfs:void pxfs_namespace_v1::fs_ii::remove_file_locks(int,CORBA::Environment&)+0xcc(0x30022cbe980?, 0x10004?)
pxfs:__SLIP.THUNK__P(, 0x10004, 0x2a107ee36a0) - frame recycled
pxfs:void pxfs_namespace_v1::pxvfs::remove_file_locks(int)+0xf4(0x10004)
pxfs:void nlm_remove_file_locks(int)+0x18(0x10004, 0x4, 0x10000, 0x1)
nfssrv:rfs4_lo_state_destroy+0x6c(0x3004a7f1c00)
nfssrv:rfs4_dbe_destroy+0x16c(0x3004a7f1b98?)
nfssrv:rfs4_dbe_reap+0x18c(0x300398c0000, 0x5a, 0x0)
nfssrv:reaper_thread+0x104(0x300398c0000, 0x0)
unix:thread_start+0x4()
-- end of kernel thread's stack --
==== analyzing panic thread stack for trap frames ====
==== using trap() frame 1 @ 0x2a107ee2fe0, rp(%i0): 0x2a107ee31e0 ====
type(%l2): 0x31 (data access MMU miss)
pc:  0x7a61b944 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88:   ldub  [%l6 + 0x70], %l1
npc: 0x7a61b948 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x8c:   ldub  [%l3 + 0xd54], %l2

global:                         %g1 pxfs:cl_flk_state_transition_notify+0x0
%g2                0x3  %g3                0x3%g4      0x30051198280  %g5                0x4%g6               0x6d  %g7      0x2a107ee3ca0out:    %o0      0x300380d1850  %o1      0x2a107ee3ca4
%o2                  0  %o3                0x4%o4            0x70800  %o5      0x300380d1870%sp      0x2a107ee2a81  %o7 pxfs:void flk_state_transition_notify(lock_descriptor*,int,int)+0x130loc:    %l0      0x3004a7f1bd0  %l1               0x13
%l2                0x1  %l3 pxfs(data):        0x7087c000%l4            0x7087c  %l5            0x70800%l6                  0  %l7             0x1000in:     %i0      0x300380d1850  %i1      0x3003d41f280
%i2             0x1000  %i3         0xffffef87%i4 cl_dcs(data):PXFS_GREEN+0x0 %i5            0x7082c%fp      0x2a107ee2b31  %i7 genunix:flk_set_state+0x60
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x60:      sra        %o7, 0x0, %o4
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x64:      add        %g3, 0x96f, %o1
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x68:      sra        %l0, 0x0, %o3
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x6c:      sra        %o5, 0x0, %o5
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x70:      call       cl_load:void dbg_print_buf::dbprintf(char*,...)
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x74:      or %g0, %i0, %o2   ( mov   %i0, %o2 )

pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x78:   1: sethi      %hi(0x70800) %l5        ( set   0x70800, %l5 )
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x7c:      ldx        [%i0 + 0x30], %l6
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x80:      or %l5, 0x7c, %l4 (0x7087c)
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x84:      sllx       %l4, 0xc, %l3
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88:      ldub       [%l6 + 0x70], %l1
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x8c:      ldub       [%l3 + 0xd54], %l2
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x90:      orcc       %l1, %l2, %g0
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x94:      bne,pn     %icc, pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xc8 (2f)
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x98:      or %g0, %i0, %o0   ( mov   %i0, %o0 )

pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x9c:      call       pxfs:repl_pxfs::lock_info_t*fobj_ii::get_lock_info(lock_descriptor*)
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xa0:      or %g0, %i1, %o1   ( mov   %i1, %o1 )
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xa4:      or %g0,  0x1, %i1  ( mov   0x1, %i1 )
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xa8:      or %g0, %o0, %l7   ( mov   %o0, %l7 )
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xac:      stw        %i1, [%o0 + 0x60]
pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0xb0:      or %g0,  0x10, %o0 ( mov   0x10, %o0 )
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> panic
panic on CPU 60
panic string:   BAD TRAP: type=31 rp=2a107ee31e0 addr=70 mmu_fsr=0 occurred in module "pxfs" due to a NULL pointer dereference
==== panic kernel thread: 0x2a107ee3ca0  PID: 0  on CPU: 60  affinity CPU: 60 ====
cmd: sched
t_procp: 0x18594e0(proc_sched)
p_as: 0x185a8f8(kas)
zone: global
t_stk: 0x2a107ee3a90  sp: 0x18756e1  t_stkbase: 0x2a107ede000
t_pri: 60(SYS)  pctcpu: 0.000000
t_lwp: 0x0  psrset: 0  last CPU: 60
idle: 9000 ticks (1 minutes 30.00 seconds)
start: Thu Oct 29 17:55:01 2015
age: 8370 seconds (2 hours 19 minutes 30 seconds)
tstate: TS_ONPROC - thread is being run on a processor
tflg:   T_TALLOCSTK - thread structure allocated from stk
T_PANIC - thread initiated a system panic
tpflg:  none set
tsched: TS_LOAD - thread is in memory
TS_DONT_SWAP - thread/LWP should not be swapped
pflag:  SSYS - system resident process

pc:      unix:panicsys+0x48:   call     unix:setjmp
startpc: nfssrv:reaper_thread+0x0:   save       %sp, -0xb0, %sp

unix:panicsys+0x48(0x1096af8, 0x2a107ee2f88, 0x18760b0, 0x1, , , 0x9900001603, , , , , , , , 0x1096af8, 0x2a107ee2f88)
unix:vpanic_common+0x78(0x1096af8, 0x2a107ee2f88, 0x18a6800, 0x2a107ee2a81, 0x2a107ee3280, 0x4400001606)
unix:panic+0x1c(0x1096af8, 0x31, 0x2a107ee31e0, 0x70, 0x0, 0x30022dec1a8, 0x1831aa0)
unix:die+0x78(0x31, 0x2a107ee31e0, 0x70, 0x0)
unix:trap+0x9e4(0x2a107ee31e0, 0x70)
unix:ktl0+0x64()
-- trap data  type: 0x31 (data access MMU miss)  rp: 0x2a107ee31e0  --
addr: 0x70
pc:  0x7a61b944 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88:   ldub  [%l6 + 0x70], %l1
npc: 0x7a61b948 pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x8c:   ldub  [%l3 + 0xd54], %l2
global:                       %g1         0x7a617cbc
%g2                0x3  %g3                0x3%g4      0x30051198280  %g5                0x4%g6               0x6d  %g7      0x2a107ee3ca0out:  %o0      0x300380d1850  %o1      0x2a107ee3ca4
%o2                  0  %o3                0x4%o4            0x70800  %o5      0x300380d1870%sp      0x2a107ee2a81  %o7         0x7a632b94loc:  %l0      0x3004a7f1bd0  %l1               0x13
%l2                0x1  %l3         0x7087c000%l4            0x7087c  %l5            0x70800%l6                  0  %l7             0x1000in:   %i0      0x300380d1850  %i1      0x3003d41f280
%i2             0x1000  %i3         0xffffef87%i4         0x7082c000  %i5            0x7082c%fp      0x2a107ee2b31  %i7          0x110be98<trap>pxfs:void fobj_ii::lock_deleted(lock_descriptor*)+0x88()
unix:cl_flk_state_transition_notify(0x3003d41f280, 0x3, 0x8) - frame recycled
genunix:flk_set_state+0x60(0x3003d41f280, 0x8, 0x0)
genunix:flk_delete_active_lock+0x58(0x3003d41f280, 0x0)
genunix:cl_flk_remove_locks_by_sysid+0xf0(0x10004)
pxfs:void pxfs_namespace_v1::fs_ii::remove_file_locks(int,CORBA::Environment&)+0xcc(0x30022cbe980?, 0x10004?)
pxfs:__SLIP.THUNK__P(, 0x10004, 0x2a107ee36a0) - frame recycled
pxfs:void pxfs_namespace_v1::pxvfs::remove_file_locks(int)+0xf4(0x10004)
pxfs:void nlm_remove_file_locks(int)+0x18(0x10004, 0x4, 0x10000, 0x1)
nfssrv:rfs4_lo_state_destroy+0x6c(0x3004a7f1c00)
nfssrv:rfs4_dbe_destroy+0x16c(0x3004a7f1b98?)
nfssrv:rfs4_dbe_reap+0x18c(0x300398c0000, 0x5a, 0x0)
nfssrv:reaper_thread+0x104(0x300398c0000, 0x0)
unix:thread_start+0x4()
-- end of kernel thread's stack --
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> ^C

<p style="text-align: left;">
Run  help command get  full list  of  commands  which available on  SCAT.
</p>

SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> help
Initial Investigation:
analyze   coreinfo  msgbuf  panic  stack  stat
toolinfo

General Commands:
analyze  clockinfo  coreinfo  cyclic  demangle  eckstat
environ  exit       help      intr    kstat     modinfo
msgbuf   stat       symbols   taskq   ttrace    tunables

Memory Dump/Display Commands:
bigdump  buf   flip  kseg    mdump  nvlist
pdump    pkma  r     rd      rd16   rd32
rd64     rd8   rdb   rdc     rdd    rdf
rdh      rdi   rdl   rdq     rds    rdw
sdump    seg   skma  strsum  vmem   wr

Data Conversion Commands:
2base  2dec  2double  2float  2hex       2string
2time  bits  calc     decode  deltatime  pid2paddr
size

Process/Thread Information:
classtbl  panic    pid2paddr  proc  sig   thread
tlist     upcount

Stacks/Traps:
findaddr_stack  frame  regs  stack  trap

Searching Memory/Core:
findport  findval  stack  tlist

File and Filesystem Information:
autofs   bigdump  buf   cgcheck  dnlc   findfiles
getpath  inode    nfs   node     tmpfs  vfssw
vfstype

CPU and Dispatch Queues:
callout  cpu   dispq  disptbl

Memory Information:
anon  iommu   ipc      kma       kseg  kstat
map   memerr  meminfo  page      pkma  resvswap
seg   sfmmu   skma     swapinfo  vmem  whatis
xmmu

Network/IPC:
findport  ifconf  ipc     ndd   nfs   nstat
pdump     pkma    stream

Type Database Commands:
ctf    sarray  savl   sdump  shash   skma
slist  slistt  stree  stype  typedb

Lock Display Commands:
cv      lck   mutex  rwlock  sema  sleepq
tstile

Device Information Commands:
dev   qlcfc

System Files/Tunables:
etcsystem  msgbuf  name2major  path2inst  tunables  vfstab

Disassembler Commands:
codepath  dis   rdi   sim

Data Structure Consistency Checks:
anon  callout  cgcheck  dispq    dnlc  kma
seg   sfmmu    symbols  upcount  vmem

Miscellaneous Commands:
clust  door  legal     namelist  pool  project
pty    rctl  sanitize  svm       task  toolinfo
trans  zfs   zone

Tool Configuration:
base  color  refclock  scat_version  scatenv  writeUsage: help [command]
help -? regular_expression
help -g [<group>]
help all

I added some of  controls like ;
  • Memory  usage
  • Process  information
  • Memory  Errors

You can check help  output  for  more  details.

SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> memerr
errorq pci_target_queue @ 0x300014d0040:
dispatched:    0
dropped:       0
logged:        0
reserved:      0
reserve_fail:  0
committed:     0
commit_fail:   0
cancelled:     0

errorq ce_queue @ 0x300014d07c0:
dispatched:    0
dropped:       0
logged:        0
reserved:      0
reserve_fail:  0
committed:     0
commit_fail:   0
cancelled:     0

errorq ue_queue @ 0x300014d0a40:
dispatched:    0
dropped:       0
logged:        0
reserved:      0
reserve_fail:  0
committed:     0
commit_fail:   0
cancelled:     0

errorq fm_ereport_queue @ 0x300014d1440:
dispatched:    0
dropped:       0
logged:        0
reserved:      0
reserve_fail:  0
committed:     0
commit_fail:   0
cancelled:     0
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> meminfo
pages        bytes
physinstalled               4177920  34225520640 (31.8G)
physmem                     4172567  34181668864 (31.8G)
total_pages                 4069191  33334812672 (31G)

freemem                     3171275  25979084800 (24.1G)
avefree                     3169686  25966067712 (24.1G)
avefree30                   3170063  25969156096 (24.1G)
availrmem (nonswapable)     3788115  31032238080 (28.9G)
availrmem_initial           4069191  33334812672 (31G)
swapfs_minfree               521570   4272701440 (3.97G)
sw_pending_size                                0 (0)

lotsfree                      63581    520855552 (496M)
desfree                       31790    260423680 (248M)
minfree                       15895    130211840 (124M)
throttlefree                  15895    130211840 (124M)

pp_kernel(calculated)        377838   3095248896 (2.88G)
obp_pages                      3270     26787840 (25.5M)
pages_locked                    339      2777088 (2.64M)

kcage_freemem                 21381    175153152 (167M)
kcage_needfree                    0            0 (0)
kcage_lotsfree                15895    130211840 (124M)
kcage_desfree                  7947     65101824 (62M)
kcage_minfree                  7947     65101824 (62M)
kcage_throttlefree             3973     32546816 (31M)

shared memory (SM)                        852048 (832K)
intimate SM (ISM)                              0 (0)
dynamic ISM (DISM)                             0 (0)
locked DISM                       0            0 (0)
total locked SM                                0 (0) (0.00% of memory)
spt_used (ISM)                    0            0 (0)
segspt_minfree                    0            0 (0)

anoninfo: (physical == disk-backed)
ani_max - total reservable physical swap                6291557 pages (48G)
ani_free - unallocated physical and memory              5837739 pages (44.5G)
ani_phys_resv - reserved physical                        492932 pages (3.76G)
ani_mem_resv - reserved memory                                0 pages (0)
ani_locked_swap - swap locked in reserved mem swap            0 pages (0)

initial virtual swap available for reservation            9839178 pages (75G)
ani_max + MAX((availrmem_initial - swapfs_minfree), 0)
current virtual swap available for reservation            9065170 pages (69.1G)
(ani_max - ani_phys_resv) + MAX((availrmem - swapfs_minfree), 0)

swap device                    pages           free
/dev/dsk/c1t0d0s1      6291557 (48G)   6291557 (48G)

tmpfs:
tmount          size  mount point
0x600408b3208  2.04M  /etc/svc/volatile
0x60049434718  65.2M  /tmp
0x60049571310   168K  /var/run

ramdisk: (none)
SolarisCAT(/CASESUP/EXPECT/vmcore.11/10V)> proc
addr       PID    PPID   RUID/UID     size      RSS     swresv   time  command
============= ====== ====== ========== ========== ======== ======== ====== =========
0x30022e41920  25301   1121          0    6316032  5210112   884736     10 /usr/lib/ssh/sshd
0x300574646b8  24710    704          0   12509184  7168000  1736704     63 /usr/sfw/sbin/smbd -D
0x6005e3970b0  23371  23369  151611447    5079040  3760128   991232     48 /usr/lib/ssh/sftp-server
0x300369eda38  23369   5898  151611447    1425408  1368064   196608      1 sh -c /usr/lib/ssh/sftp-server
0x300598df2f8  20312  20310 2129141062    3776512  1703936   229376      2 -sh
0x30024c16570  20310  19983 2129141062   15261696  4882432  1572864      1 /usr/lib/ssh/sshd
0x3002856d9e0  19983   1121          0   13377536  7675904  1572864     24 /usr/lib/ssh/sshd
0x300285659e8  21504  21503 1667533448    4882432  2768896   794624      4 /usr/lib/ssh/sftp-server
0x30022e3c0f8  21503  14653 1667533448    1425408  1368064   196608      1 sh -c /usr/lib/ssh/sftp-server
0x30022e36548  19126    887          0   13737984 10321920  1900544     45 /usr/sbin/rpc.metacld
0x3002cba01f0  14877   6123          0   16105472 10625024  2572288    252 /usr/ecc/exec/mstragent -s
0x3002204c090  13079   4214        100    5644288  3227648   655360      5 bash
0x30040457a78   8463  18478        100  126468096 90005504 29532160    153 siebsess 22 1 /global/sbsiebcrmcc/siebsrvr/admin/ENT_CCPROD.TESTSUNCL.shm 0 enu
0x300220340a8   7095   4628          0    2736128  2555904  1187840    188 vi /var/adm/messages
0x30001cb04a8   4628   4583          0    3301376  2924544   614400      7 bash
0x30037c5fa50   4583   4535          0    1458176  1433600   204800     14 -sh
0x30024c0a588   4535   4530          0    1458176  1433600   204800     13 -sh
0x30051fbd2c8   4530  23788 1548235000    1458176  1417216   204800      2 -sh
0x3002204d8c0   4214  10519        100    5644288  3219456   655360      6 bash
addr       PID    PPID   RUID/UID     size      RSS     swresv   time  command
============= ====== ====== ========== ========== ======== ======== ====== =========
0x30040452250   4124  23861          0    3301376  2924544   614400     12 bash
0x300220524d0  29319  29318          0    8331264  5677056  1097728      7 /opt/SUNWscnfs/bin/nfs_probe -R nfs1-server-rs -T SUNW.nfs:3.2 -G nfs1-rg
0x300220380a0  29318   7167          0    1425408  1376256   196608      4 /bin/sh -c /opt/SUNWscnfs/bin/nfs_probe -R nfs1-server-rs -T SUNW.nfs:3.2 -G nf
0x600443be018  29316   7167          0    9576448  7028736  2334720     25 /opt/SUNWscnfs/bin/nfs_daemons_probe -R nfs1-server-rs -T SUNW.nfs:3.2 -G nfs1-
0x3003ac46678  29258      1          1    2531328  2146304   745472      4 /usr/lib/nfs/lockd
0x300285841a0  29249      1          1    6078464  2637824  1146880      2 /usr/lib/nfs/statd -p /global/nfs1/SUNW.nfs -a 10.4.18.163
0x30027cc3a08  29158      1          1    2834432  2572288  1015808  11230 /usr/lib/nfs/nfsd -s /global/nfs1/SUNW.nfs
0x30022e5c0d8  29134      1          0    4349952  2899968  1138688      2 /usr/lib/nfs/mountd
0x30022e440e8  23861  23794          0    3301376  2916352   614400     12 bash
0x30027cc21d8  23794  23790          0    1458176  1433600   204800     14 -sh
0x3003ac45a60  23790  23788 1548235000    1458176  1409024   204800      2 -sh
0x30001cac4b0  23788  23743 1548235000   10706944  4505600  1540096     52 /usr/lib/ssh/sshd

I'm a IT Infrastructure and Operations Architect with extensive experience and administration skills and works for Turk Telekom. I provide hardware and software support for the IT Infrastructure and Operations tasks.

199 Total Posts
Follow Me

417 Comments

  1. http://mewkid.net/who-is-xandra/ - Amoxicillin Without Prescription Amoxicillin rpc.vwky.casesup.com.qtb.mr http://mewkid.net/who-is-xandra/

  2. http://mewkid.net/who-is-xandra/ - Amoxicillin 500 Mg Amoxicillin fbw.aasp.casesup.com.zvt.re http://mewkid.net/who-is-xandra/

  3. http://mewkid.net/who-is-xandra/ - Amoxicillin 500 Mg Amoxicillin 500 Mg qrd.rfhb.casesup.com.exv.km http://mewkid.net/who-is-xandra/

  4. http://mewkid.net/who-is-xandra/ - Amoxil Dosage For Amoxicillin 500mg jns.ypfo.casesup.com.bcp.xp http://mewkid.net/who-is-xandra/

  5. порно рокко старое Порно фильмы смотреть в хорошем качестве лесбиянки порно видео русские мамы и сыновья видео Отчим и падчерица скачать порно порно видео элины камирен Дракон порно мультфильм порно в автотранспорте Госпожа порно досуг

  6. http://site.surq.ru/category-7/porno-anal-bolshie-zadnici-video.html http://site.surq.ru/category-6-9.html http://site.surq.ru/category-7/perviy-raz-na-popu-porno.html http://site.surq.ru/category-7/armyane-porno-rasskazi.html http://site.surq.ru/category-3/jena-znakomogo-porno.html

  7. http://site.surq.ru/category-4/besplatno-porno-s-volochkovoy.html http://site.surq.ru/category-7/sara-iks-mills-porno.html http://site.surq.ru/category-4-5.html http://site.surq.ru/category-1/bolshie-chleni-porno-onlayn-v-horoshem-kachestve.html http://site.surq.ru/category-2-9.html

  8. Привет! Кто-то из подписчиков моей страницы в Facebook подписан на автора этого веб-сайта и может перейти туда, чтобы оценить увлекательные материалы? Я сделал закладку и обязательно поделюсь с моими подписчиками! Отличный блог, замечательный дизайн и стиль.

  9. male enhancements

  10. top free cam sites mobile cam sites cheapest webcam sites. young webcam sex hiden cam sex boy cam sex. mfc cam site adultcamsites1 video cam sites. camgirl young cam girls girls locker room hidden cam. asian cam sex cam to cam sex webcam couple sex. camgirls hotcamgirls1.com nude webcam girls.

  11. asian cam girls top webcam girl young cam girl. best cam girls webcammodels nude romanian girls. best webcam sites adultfreewebcamsites shemale webcam sites. camming hotcamgirls1 library cam girl. camgirls hotcamgirls1.com web cam teen girls. free cam site adultcamsites1 good webcam sites.

  12. adult sex cam bestonlinesexcams1.com online sex webcam. best sex cam site adultfreewebcamsites free porn webcam sites. live cam sex teen hidden cam sex real sex caught on spy cam. teen cam sex bestonlinesexwebcams.com 100 free sex cam. best adult webcam sites adultcamsites1.com popular cam sites. young girls on cam camgirl sites girl masturbates on stream.

  13. live online sex cams and free sex chat rooms www sex cams com nude sex cams men and women. german live sex cams free black sex cams shemale sex cams. goth sex cams sexcams00 live sex cams apps. all sex cams free tranny sex cams sex arabic hidden cams celeb vids. xxx live sex cams sexcams00 sex cams in staten island ny. lice sex cams sexcams00 forced sex overweight video cams. sex cams video private sex cams livejasmin ca live sex cams. tasty hot latinas sex cams sexcams00 www.onet.pl sex cams.

  14. tumblr hidden sex cams sexcams00 you tube hidden apt sex cams blogs. czech sex cams sexcams00 free sex cams sites. live sex cams jasmin sexcams00 free sex talk cams. free hidden sex cams sexcams00.site mature women resist sex cams. free no sign up sex cams sexcams00 family sex cams site:xvideos.com. free latina sex cams sex web cams real live streaming sex cams. streamate sex cams sexcams00.online hacked ip cams couple sex archive videos. free no sign up sex cams sex cams free chat chubby mature forced sex cams nylons.

  15. no sign up sex cams free sex cams com normal couple sex cams. free live naked sex cams free live streaming sex cams webva,s swrrt bveccca web cams sex. live sex cams live sex web cams hidden home sex spy cams. sex cams 101 dirty sex cams dirty sex chat now underage sex cams. live gay male sex cams bondage sex cams how do you make money doing sex cams. hd sex cams chaturbate live sex cams are sex cams illegal in ukraine. new live sex cams sexcams00 amatuer sex cams. the best free sex cams sexcams00 real sex cams in the united states.

  16. woman live sex cams ameture sex cams sex video live cams muslim. new sex cams totally free sex cams free web sex cams. sex cams with sound filipina sex cams steph and real life cams maya sex. cam4 live sex cams sexcams00 sex video cams couples. iphone sex cams pantyhose sex cams cams inside pussy sex pornhub. chaturbate sex cams sexcams00 veb cams near me sex. free granny sex cams couples sex web cams analprincex recorded sex cams. gay free sex cams sexcams00.xyz older women on sex cams.

  17. sex cams paypal sexcams00.live real sex cams. teen live sex cams sexcams00 under age sex cams live. free shemale sex cams free live nude sex cams live sex cams from home. fantasy sex cams sexcams00.site real amateur home sex cams. erotic sex cams sexcams00 best free hd live streaming sex cams. fantasy sex cams sexcams00 is viewing sex cams legal. best live sex cams free sexcams00 incest cams sex. free live sex cams free sexcams00.xyz hidden cams used in men's public places for sex.

  18. american sex cams real free sex cams gay sex cams with no tokens. live gay male sex cams totally free live sex cams columbian live sex cams. free live web sex cams free gay sex cams sites with sex web cams. hardcore sex cams live ebony sex cams free sex cams chat live. streaming live sex cams recorded sex cams sex cams modesto. live real sex cams free live asian sex cams hottest free live streaming sex cams. totally free sex cams sexcams00.online free sex cams totally free. sex cams online free filipina sex cams free sex chat cams.

  19. teen sex cams free local sex cams xhamsters live sex cams. amsterdam sex cams free free sex cams sex cams modesto. snapchat sex cams gymnast hidden sex cams adult couples sex web cams. free live mobile sex cams female sex cams forced sex scenes free video cams. free live sex cams couples couples sex cams watch home sex cams. video sex cams real life cams sex teen house sex cams. teen girl sex cams live sex web cams live web sex cams.

  20. my indian sex cams sexcams00 hidden cams used to film men having sex in public. free porn live sex cams free private sex cams boys sex cams. worlds best sex cams lice sex cams arab hidden sex cams vids. live college sex cams sexcams00 free hd voyeur sex cams. free tranny sex cams sexcams00 free web sex cams. teen girl sex cams sexcams00.club men and women sex cams. www.free sex cams free live streaming sex cams free sex cams chat.

  21. free live private sex cams sexcams00.live cams, sex. senior sex cams sexcams00 seattle sex cams live. sex cams videos free hardcore sex cams uncensored ebony adult two way sex cams. sex cams jasmine sex cams no sign up is viewing sex cams legal.

  22. free russian sex cams sexcams00.live indonesia girls sex bf in hidden cams. free to watch sex cams free web sex cams oral sex cams. free live sex cams couples sexcams00 live gay sex cams. live sex cams no credit card sexcams00 vr live sex cams.

  23. free live sex cams couples ashley raye sex cams greek sex cams. ts sex cams sex cams en vivo sex cams for tokens. adult live sex cams sexcams00 couples sex can web cams. one on one sex cams sexcams00 my free sex cams.

  24. hd sex cams no credit card sex cams minnesota sex cams. chaterbait sex cams free live asian sex cams gay boys sex cams. ashley raye sex cams free sex cams no sign up adult sex web cams. streamate live sex cams sexcams00 free realtime spy appartments h1 sex cams.

  25. free live adult sex cams live office sex cams uper edda in sex cams live free do easily to wanking with sybi. nude beach sex cams sexcams00.website drunk sex orgy shower cams. free live lesbian sex cams cams sex gay twiГ±ks sex cams. no sign up sex cams teen sex cams tiny and young live sex cams.

  26. gay men sex cams live ebony sex cams nude sex cams.

  27. czech sex cams free women sex cams sex cams online.

  28. watch 100% free sex cams, latina sex cams granny sex cams.

  29. fetish sex cams wap sex cams realtime spy apparttments sex cams h1tv.

  30. free amateur sex cams live public sex cams freeweb sex cams.

  31. goth sex cams sexcams00.site sex cams in mn.

  32. paid sex cams sexcams00 realtime spy h1 tv appartments sex cams.

  33. live sex cams real sex cams sex tv babe cams.

  34. free live web sex cams sexcams00.xyz sex hot new arab hidden cams.

  35. no registration sex cams sexcams00 amatuer young teen girls recorded on sex cams.

  36. filipina sex cams sexcams00 free sex cams pussy.

  37. free granny sex cams sex cams couples free porn sex cams.

  38. free uncensored sex cams gay live sex cams hidden cams in school college dorm real sex and orgasm.

  39. hawaiian sex cams free women sex cams 1 on 1 sex cams.

  40. live sex cams jasmin sexcams00.live sex cams online.

  41. live sex cams no credit card sexcams00.work realtime sex spy cams.

  42. how many girls stream sex cams for money sexcams00.online black sex cams.

  43. live sex cams asian sexcams00.online edtna sex cams.

  44. polish sex cams sexcams00.site sex cams with random people.

  45. sex cams online ebony sex cams amatures caught on cams having sex.

  46. russia sex cams sexcams00 live streming sex cams.

  47. pregnant sex cams milf sex cams young having sex at home on cams.

  48. porno sex cams live granny sex cams chaturbate - free adult webcams, live sex, free sex chat, exhibitionist & pornstar free cams.

  49. sister sex cams streamate sex cams brother sister sex cams.

  50. how many girls stream sex cams for money free russian sex cams omegle sex cams.

  51. india sex cams sexcams00 xhamsters live sex cams.

  52. coed sex cams sexcams00.online hidden cams sex.

  53. totally free sex cams sexcams00 free realtime spy h1 tv appartments sex cams.

  54. Awesome article. P.S. If you have a minute, would love your feedback on my new website re-design. You can find it by searching for «royal cbd» — no sweat if you can’t. Keep up the good work!

  55. I know this if off topic but I'm looking into starting my own weblog and was curious what all is needed to get set up? I'm assuming having a blog like yours would cost a pretty penny? I'm not very internet savvy so I'm not 100% sure. Any tips or advice would be greatly appreciated. Many thanks P.S. If you have a minute, would love your feedback on my new website re-design. You can find it by searching for «royal cbd» — no sweat if you can’t. Keep up the good work!

  56. live teen sex cams sexcams00.work sex cams audio. asian sex cams free home milf amateurs sex cams sex cams muslim. asian live sex cams ive sex cams free live streaming sex cams hd. hot live sex cams sexcams00 booty call cams sex jessica. live sex cams xhamster sexcams00 forced date sex cams. sex cams gay live nude sex cams hacked home security cams sex. live sex cams' free sex cams live amaturedad daughter sex cams. 24 7 sex cams sexcams00 free sex cams cam2 xname.

  57. hidden teen sex cams free sex cams live best cams for hidden sex.

  58. free live hd sex cams sexcams00 sex hot new arab hidden cams.

  59. free no sign up sex cams free college sex cams amateur honeymoon cum sex cams.

  60. watch sex cams free sexcams00.site live private sex cams.

  61. private free sex cams online teen sex cams you tube hidden apt sex cams.

  62. cam4 live sex cams sexcams00.live free web sex cams.

  63. young sex cams sexcams00 sex arabic hidden cams celeb vids.

  64. best live free sex cams free online sex cams cams 4 sex.

  65. free sex cams no membership teen sex cams sex caught on hidden cams.

  66. live home sex cams sexcams00.live jr sex cams.

  67. boy sex cams thai sex cams edtna sex cams.

  68. chatroulette sex cams sexcams00 live sex cams in texas.

  69. my free sex cams sex cams tube chaterbait sex cams.

  70. live hidden sex cams sexcams00 real life cams stesha and marco groupe sex.

  71. sex cams for free couples sex cams 2 way sex cams.

  72. free hidden sex cams sexcams00 illegal mexican sex cams.

  73. redtube live sex cams nasty sex cams free live streaming sex cams.

  74. lesbian free sex cams sexcams00 l.ive jap sex cams.

  75. webcam sex cams free dirty sex cams secret spy cams sex swingers.

  76. free indian sex cams live male sex cams pregnant sex web cams.

  77. vr sex cams tranny sex cams analprincex recorded sex cams.

  78. bbw live sex cams free ebony sex cams free forced sex video cams.

  79. home milf amateurs sex cams dog sex cams st free live sex cams.

  80. gay free sex cams sexcams00.club rellife sex spy appartments cams.

  81. sex cams xvideos [url=https://sexcams00.work]sexcams00.work[/url] live nude sex cams.

  82. ashley raye sex cams sexcams00.xyz real family sex cams.

  83. free indian sex cams homemade sex cams todays hidden sex cams.

  84. girls live sex cams sexcams00 realtime sex spy cams.

  85. woman live sex cams sexcams00 forced sex free video cams.

  86. sex cams couples worlds best sex cams nintendo girl characters sex mobile browser game cams.

  87. anonymous sex cams naked sex cams real amatuer forced sex cams.

  88. boy sex cams family sex cams nude sex cams men and women.

  89. free live naked sex cams sexcams00.live secret massage sex cum cams.

  90. libe sex cams asain sex cams omegle man sex cams.

  91. female sex cams sexcams00.live private sex cams.

  92. live mobile sex cams sexcams00.live amateur honeymoon sex cams.

  93. cam4 live sex cams cams sex rocford illinois real hidden sex cams.

  94. personal sex cams private sex cams live cams sex.

  95. naughty sex cams sexcams00.site flodia sex cams.

  96. fee live sex cams sexcams00.site totally free live streaming sex cams hd.

  97. hidden teen sex cams sexcams00.website couples sex cГ ms.

  98. luve sex cams live gay sex cams forced sex video cams.

  99. canadian sex cams asain sex cams instant sex with strange girls hiden cams.

  100. amature hidden sex cams free homemade sex cams nwoww sex cams.

  101. free uncensored sex cams sexcams00 hidden cams pinay scandals sex.

  102. live sex cams no credit card free adult sex cams bongacams live sex cams.

  103. black girls in anal pian blackwebcams00.pw old black hairy mature sex.

  104. blonde milf porn gifs fucking a hot indian milf hot nude blonde milf.

  105. skinny amateur blonde teen begs for anal pounding amateurwebcams00.monster teen amateur porn gifs.

  106. chubby teen nude with big clit almost caught teen sex video pov teen fucks brother as he sleeps.

  107. nude pics of debby uk milf [url=https://milfwebcams00.casa]milfwebcams00.casa[/url] jiggling milf boobs doggy style.

  108. brunette teen fucked deep by massive cock teen anal sock porn fucking teen in virgin islands.

  109. amateur brush masturbation videos mmf gay fucking amateur fine amateur milf nude. asian turned dominated slave porn asianwebcams00 thai ladyboys group sex vid eeer os. naked black haired girl manga blackwebcams00 tiny ebony big ass fucking. wisconsin college girls naked older man to college guy cock play videos full length college porn. asian lesbian strapon fuck on train lesbianwebcams01.fun lesbian forces straight girl pussy. sex mature russian hd film maturewebcams00 bbw mature solo squirting. amateur milf swallows car blowjob porn milfwebcams00 hairy pussy milf spanish porn videos and movies. short nerdy teen anal teenwebcams00.uno sex videos of teen son taking mom. orgasmic amateur. nude women amateur ebony female nude beach chubby nude amateur women. thai massage man fucks ass free thai pussy massage asian in africa porn. fake nude peggy bundy pics black dick blackwebcams00 free porn black man with eat. amateur college undress fuck video college sex party tube college lesbian orgy strapon sex gif. anal granny lesbians xxx lesbianwebcams00 gigantic cuban lesbians fucking. boys love matures nude bbw maturewebcams00 mature women asks to get fucked. my mom the milf uncensored milf japanese tube hairy blonde milf pussy. porn comics hot white teens teenwebcams00 naked teens ass amateur.

  110. big brested african women having sex black dick asian anal black pussy squirts on face.

  111. female topless bottomless college nude collegewebcams00.site black french college student gets paid to fucked.

  112. most cum on an amateur amateur wife facial cum pics amateur video of dad fucking young daughter. petite chinese gf fucking big tits japanese wife desi sexy wife porn. sexy 40 old black woman naked blackwebcams00.icu thick black girl hot oil porn. college train pussy football indian college girls sex nikara - busty college coeds pov 3. brunettes lesbian sex gif lesbianwebcams01.fun lesbians having sex audio. mature as fuck song maturewebcams00 best mature handjob tube. hairy latina milf solo milf lesbian sex porn free amatuer young milf with bbc. teen girls masturbate together teen fucks for cash tight teen pussy fucking gif. amateur teen knotted video amateur saggy tits sex teen girl gangbang amateur. indian guy foreign girl porn hot indian aunty sex small young skinny naked indian teen girls blowjobs. nude ebony threesome gif huge black boobs tumblr ucke and niece sex black. hot college athlete asses collegewebcams00.store college fuck part 2 teen ebony. porn lesbians workout lesbianwebcams00.fun lesbian with choker sex. matures milfs brutal dildos heels maturewebcams00 amatuer nude matures short hair blondes new jersey. ebony milf seduces white boy milfwebcams00 milf bea cummings nude white guy. real naked homless teen girls asian teen big ass skinmy asian teen amateur porn.

  113. bathroom selfie teen pussy teen sex 7chan gif lesbian teen fuck for meth.

  114. thai hunk nude tumblr asianwebcams00.website indian college girl sex vidro.

  115. hairy pussy lesbians classic group makenzie nude lesbian sex porn star lesbian videos.

  116. amateur mature milf naked amateurwebcams00.xyz public agent amateur suck. indian nude wear saree japanese mom and son watch porn together thai missionary fucking pics. lipstick ebony blowjob porn blackwebcams00 train from african gangbang vintage porn. secret college porn video hot college blonde riding me porn saggy boobs college squirt. carla lannie milf lesbian lesbianwebcams01.fun lesbian sex strap on roommate ambush. atk amature mature naked hot mature women handjobs matures fucking large cocks. hairy bbw milfs fucking pics milfwebcams00 nude homemade blonde milf. anal exotica seducing teen girl free live teen porn extrem tall teens fucking. amateur mature blowjob tube amateur milf mature sex videos amateur bbw blowjob tits. latest asian porn scenes asianwebcams00.host indian sexy girls sex. recent bronx ny black unaware sleeping porn blackwebcams00 best black on black porn sites. sex bent over college porn met at college mn college stud andf twink anal. hitchhiker lesbian dildo sex lesbianwebcams00 milf lesbian shaving pussy. ugly mature whore brutal face fucking maturewebcams00.club mature blow job suck. xhamster milf takes 2 huge cocks hairy granny and black men milf tubes milf cheating sex gif. teen hanged and anal fucked videos teenwebcams00 oiled petite teen pussy.

  117. ebony amateur missionary videos blackwebcams00 black teen anal distructiom xxx videos.

  118. skinny brunette amateur girlfriend ana floor porn amateur homemade public sex homemade amateur teen sex xtube.

  119. tumblr gay college hairy collegewebcams00 fuck college girls gifs.

  120. amateur bbc creampie gangbang hot nude pregnant amateur girls amateur bent over and fucked rough.

  121. amateur bi swingers party busty wife amateur cum pics amateur teen leggings sexy. japanese teen selfie naked indian teen nude videos porn two asians with a white. ebony bombo sucking white dildo nude black men models ebony big boobs nude gif. asian college side boobs collegewebcams00.site tight college girl pussy. violet amd helen parr lesbian hentai naked images lesbianwebcams01.fun tumblr naked lesbian dilo ass. homemade private sex mature amateur mature morning wife strapons used on mature men. milfs riding pov up close milfwebcams00 xnxx hot ebony milf. teen age sex hd miss teen usa nude man tricked by teen into anal. amateur clit lick lesbians amateur wife fucks bbc black amateur new years sex party. indian village pussy xxx asian mom and son watch porn japanese mom fuck by friend son. ebony tit boon breast blackwebcams00.pw hot nakedvery black women naked. dick vitale awsome baby college hoops collegewebcams00 thickest college girls fuck. hot lesbian sex cam lesbian bbw porn hot brown skinned lesbians. maturejap - older mature japanese lesbian women kissing mature ass play shy boy mature amature throat fuck. bbw milf tits pussy hidden cam milf porn young milf ass pussy. real teen sex on spy cam pretty teen nude pics michigan teen kept as sex slave.

  122. free pics of amateur milf milfwebcams00 naked blonde milf gifs.

  123. free black mature lesbian videos lesbianwebcams01.fun lesbian cougar dominates teen lesbian forced sex.

  124. mature bbc amateur porn maturewebcams00 asian mature cock sucking sluts.

  125. naked amateur wife tits gif real amateur couple photo sets amateur video fucking the baby sitter. lana violet mia rider asian fuck faces japanese family sex video chinese nude stage dance. best ever ebony sex scene blackwebcams00 huge ass ebony mom threesome. college fuck fest tori college girl pussy galleries college fuck fest grinding. lesbian sex with wet pussy with dildo lesbianwebcams01.fun thick young teenage lesbian. mature amateur porn audition maturewebcams00.space mature porn high heels. ebony milf anal big tits naked redhead milfs gallery milf love anal sex porn. sex wrestling teens grind teenwebcams00.uno lesbian teen pov porn. amateur interracial couples porn amateurwebcams00.monster naked shaved petite amateur mature wife. black vs asian porn tube japanese teen sex pics sites to watch asian porn. ebony missionaryh porn videos blackwebcams00 best african-american porn site. nude college parties reddit collegewebcams00 lilly love naked college dorm. fucking lesbian girls dirty xxx young lesbians licking ass even more sneaky lesbians x video. fat mature glasses pussy mature anal masturbation porn natural busty older mature. milf having sex tumblr milfwebcams00.casa reverse cowgirl anal milf. naked jailbait teens forum teenwebcams00.press 2018 beautiful teen nude pack.

  126. real amateur mature pussy maturewebcams00 white mature fucks black boy next door interracial.

  127. lesbian peeing onto pussy tumblr amazon lesbian porn teen lesbian pov real orgasms.

  128. teen sis porn family free teen porn pictures teen monroe jizz porn.

  129. big teen ass beach nude teenwebcams00 teach my ass teen anal pain.

  130. mature amateur selfies tumblr tumblr mature nude beach mature german whores gross creampies photos.

  131. hot sexy naked blondes milfs milfwebcams00.casa milf riding cock gif.

  132. sex with young milf threesome free lesbian milf teen porn videos milf fucked from behind.

  133. bbw amateur girlfriend handjob videos amateurwebcams00 older curvy amateurs blow.

  134. ebony hair porn fucked romantic ebony sex movies black young teen anal.

  135. desi girl fucking foreigner japanese tan girls nude athletic thai girl sex video.

  136. hont naked asian wonem fucking asianwebcams00.host chinese naked girl dirreha.

  137. amateur panties pulled aside pussy fucking amateur farm girls plump amateur mature fucked.

  138. sexy naked chinese lebians asianwebcams00 asian teens fuck car.

  139. free forced lesbian incest porn lesbian videos julia ann sheila faye lesbian nude.

  140. young amateur unclean gay porn red blow amateur asian babe taking shower real amateur stoner gf b.

  141. amateur busty white black cock petite black teen sex mature medium build black mommy porn pics.

  142. asian gyno exam porn asianwebcams00.website asian schoolgirl leaked nude.

  143. how mature bbw cunnilingus sex mature cum in mouth compilation glasses mature wife homemade nympho wild fuck.

  144. cute eboney lesbian sex [url=https://lesbianwebcams01.fun]lesbianwebcams01.fun[/url] white slaves eat ass pussy lesbian.

  145. porn star mature japan squirt maturewebcams00.club cum filled mature clams.

  146. blonde teen bbc gangbang small pussy little asian teens nude naked teen rachel nude.

  147. johny and donny forza vadim black gay porn blackwebcams00 black bbw amateur dp.

  148. amateur fucking back tattoo amateurwebcams00.monster amateur petite anal facial videos youporn.

  149. asian male breeding porn asianwebcams00.host corinne asian pussy pics.

  150. black lesbian lick white pussy blackwebcams00 ghetto hairy ebony pussy.

  151. indian anal teen cry teenwebcams00 teen latina sex gifs.

  152. black dick cum in mouth teen porn teenwebcams00.uno petit asian teen porn.

  153. amateur interacial anal creampie eating amateurwebcams00 amateur video beach sex hidden camera. asian twins midget porn asianwebcams00 indian pussy riding cock. black celebrity women naked videos free black guy sex phub black girl nudes snapchat. college tits pic dump college hidden cam sex college girl and mature women sex. lesbian fuck my feet hot sxe veido lesbian squirting anal black lesbian. tumblr mature sexy old nude couples maturewebcams00 mature moms small tits. interracial milf gang bang milfwebcams00.club blonde gold dress mature milf sexy tight -nude. nude teens in rio teenwebcams00 tiny teen ass licking. amateur milf big tits porn amateurwebcams00 amateur blowjob facial on bus. uncensored mature asian porn asian pussy belongs to white men asian drunk teen fucked in club. black and white boobs gif blackwebcams00 tumblr black pussy is better. college girls spread naked tumblr two college girls one cock college girl fucks random guy. sabrina johnson lesbian strapon lesbian mind control porn hardcore pussy eating lesbians. mature hooker cum swapping clips maturewebcams00 blonde mature pussies ass. milf boy fucking pics iranian big ass milf gold dress blonde mature cougar milf -nude. free video dad fuck sleeping teen youporn teen fucked by uncle youtube hot teen on nude beach.

  154. russian massage teen gay sex hot teen blonde nude big dick anal teens.

  155. best amateur hardcore 2018 amateurs love big black cocks videos pov fuck amateur gif. big titted moba asian streamer asianwebcams00.website young asian teen anal. sensual girl sucking black cock gifs blackwebcams00.icu black bbw public sex. college dude porn aj exploited college girl anal homemade naked college 1989. maitland ward lesbian threesome sex lesbian sex doll porn tits videos lesbian amature. mature wife never had a cock that big maturewebcams00.space gay mature turkish daddies sex porn videos. amature milf sex tumblr amateur busty latina milf porn milf brandi love creampie. sexy naked redhead teen pussy teenwebcams00.uno petite teen ass cam. amateur frizzy redhead teen nude amateurwebcams00 amateur home big black dick orgy. tiny indian pussy pound gif asianwebcams00.host learn thai for sex vacation. porn for couples black woman blackwebcams00 free ebony lesbian fight. big boob college girls ficked sex amateur asian college blnde college beach boobs. lesbian sex on top bed stockphoto lesbianwebcams00 animated lesbians x videos. live mature nude webcam gif maturewebcams00.club mature built milf neighbor orgy. non nude milfs on tumblr milf helps slide cock in my classy mature milf mom. converting img naked teen thread teenwebcams00.press nude teen small trap.

  156. amateur bbw creampie compilation amateurwebcams00 80s homemade amateur porn tape. japanese hairy pussy pissing asianwebcams00.website fucking asian really wet. amateur cuckold wife fucks enormous black dick blackwebcams00 face down ass up ebony tubes. college girl mormon porn collegewebcams00.site college party latina grinds sex. blonde petite lesbians suck clit and pussy lesbianwebcams01.fun lesbian in mom jeans mom jeans. mature milf ass pics maturewebcams00 mature fake porn casting hd. hot milf self pics galleries milfwebcams00 nude amatuer milfs in heels. tiny teen porn vids homemade teen fuck videos tumblr voluptuous teen porn. amateur hot woman sex real amateur brother and sister black amateur mature sex. best indian romance porn japanese daughter in law sex videos paul pratt fucks asian. community troy am i black boobs black teens sex videos athena pleasures black porn. lesbian college girl lick pussy orgasm collegewebcams00 white college petite girl cheating black cock. sydney cole lesbian porn black lesbian pussy eating fat mom skinny daughter lesbian porn. mature women hardcore fucking with one guy maturewebcams00.club huge boobs mature sexy. milf loves big cock cum on milf pussy milf like it black full. indian teen orgasm fuck short hair teen porn teen pussy bloopers porn movie.

  157. amateur small slut selfie amateur wife big tits amateur spy sister porn. japanese big boobs shy wife asianwebcams00.website japanese girl fucking africans. black couple shower sex naked black woman picture big titted ebony with caucasion lebian. college girl fucking a snake guys take turns pounding college girls pussy college lesbians in the shower. vanessa lesbian porn lesbianwebcams01.fun lesbian tutor forces student porn. tumblr mature amateur cum on maturewebcams00 small cock shemales fucking mature women. black man rapes white milf milfwebcams00 blonde milf sex gifs. petite teen selfie pale teen selfie naked teenwebcams00 naked teens master baiting. amateur gay rapper porn amateur blowjob videos tumblr bbc vs asian amateur anal. ugly indian girls posing naked asianwebcams00 nude japanese girl in metro. barely legal black girl nude blackwebcams00 tori black vagina nude. white college girl takes bfs cock collegewebcams00 hit naked sexy college babes. lesbian close up wet pussy fuck lesbianwebcams00.fun nude amateur pregnant lesbians fucking. janet mature big tits boobpedia hairy mature women tumblr mature wife sex stories. trimmed milf pussy tumblr milfwebcams00.casa milf lesbian foot sex. fucking black teen tranny ass old man big dick fucking teen asian teen sex .gif.

  158. thick milf ass fucking hairy bbw milf pics amateur milf slut slave creampie orgasm.

  159. teen sister megan nude free pictures of naked flat chested filipina teens homemade beautiful teen blowjob bath.

  160. naked indian aunts tumblr asianwebcams00.host old asian women doing anal.

  161. teen selfie petite nude panty red head teen pussy exclusive teen porn sofi.

  162. amateur vintage 1970s teen sex big ass teen pics c cup blond teens naked.

  163. big dick blowjob teen gag best teen ass selfie early teen sex prono.