Tracing

Summary of various tracing and debugging utilities

Differences between AIX truss, the OSS strace and OpenBSD ktrace

OpenBSD ktrace details

ktrace works similarly to the AIX trace where the output goes to a file which is a BINARY log and can be read with 'kdump'.

$ ktrace -adp 31337 && kdump

$ ktrace -ad -f ls.ktrace /usr/bin/ls

Special uses

Disable tracing on all user owned processes, and, if executed by root, all processes in the system.

$ ktrace -C

Enable (disable) tracing on all processes in the process group

$ ktrace -G PGID

Enable (disable) tracing on the indicated process ID

$ ktrace -p PID

AIX truss details

truss manual (AIX 6.1)

-D

0.0000:        execve("/usr/bin/pwd", 0x2FF22C10, 0x20012BE8)  argc: 1

0.0302:        sbrk(0x00000000)                 = 0x2000104C

0.0006:        vmgetinfo(0x2FF21600, 7, 16)     = 0

0.0005:        sbrk(0x00000000)                 = 0x2000104C

0.0004:        sbrk(0x00000004)                 = 0x2000104C

0.0005:        __libc_sbrk(0x00000000)          = 0x20001050

...

strace details

strace manual

-T

write(7, "\0\0\1\273Port\t\t\t\t22\n\nListenAddress\t\t\t"..., 451) = 451 <0.000017>

close(7)                                = 0 <0.000013>

close(8)                                = 0 <0.000013>

close(4)                                = 0 <0.000014>

-r

     0.000096 write(7, "\0\0\1\273Port\t\t\t\t22\n\nListenAddress\t\t\t"..., 451) = 451

     0.000203 close(7)                  = 0

     0.000062 close(8)                  = 0

     0.000051 close(4)                  = 0

-e trace=

Syscall,...
file
process
network
signal
ipc
desc

...

Core dumps

Linux

Core file size

ulimit -c 0 # no core file, turn off core handlers

ulimit -c 1 # no core file

ulimit -c 100 # core file size = 100 x 1k blocks

ulimit -c unlimited

-H - hard limit (cannot be increased later)

-S - soft limit (recommended)

General

/etc/profile (or /etc/profile.d/)

/etc/security/limits.conf - used by the PAM pam_limits module

Core file name: sysctl kernel.core_pattern

SUSE/openSUSE

/etc/profile - ulimit commands, commented out by default

/etc/initscript - legacy but supported

SUSE/openSUSE with sysvinit: /etc/sysconfig/ulimit - SOFTCORELIMIT, HARDCORELIMIT

openSUSE with systemd: /etc/systemd/system.conf - LimitCORE

Analyzing the core file in GDB

# gdb /path/to/binary /path/to/core

Information

man bash (ulimit)

man systemd.exec

Application Core Files in SUSE Linux (27 Oct 2005)

HowTo: Debug Crashed Linux Application Core Files Like A Pro (3 Jun 2010)

Collecting and analyzing Linux kernel crashes - Kdump

Kdump on openSUSE 11.2

AIX specific

General - Application coredumps

Get/Set system-wide coredump directory

# syscorepath -g

# syscorepath -p /path/to/dumpdir

System dump

Device used by the kernel to generate a dump during system crash. Configuring system dump makes sense when you want to be able to pass it to IBM software support for crash analysis.

Dump devices should be separate logical volumes of type sysdump.

It is recommended NOT to mirror them even if the other LVs are in mirror. There should be separate dump LVs on each mirror disk.

# mklv -t sysdump -y dump0lv rootvg 1 hdisk0

List system dump devices

# sysdumpdev -l

Set primary system dump to dump0lv

# sysdumpdev -P -p /dev/dump0lv

Estimate current dump size

# /usr/lib/ras/dumpcheck -p

The dump LVs must be resized with extendlv.

Show/Enable full core dump

# lsattr -El sys0 -a fullcore -F value

# chdev -l sys0 -a fullcore=true

Network

Examples of using Iperf to diagnose network issues (by Symantec)