strace
strace is a diagnostic, debugging and instructional userspace utility for Linux. It is used to monitor and tamper with interactions between processes and the Linux kernel, which include system calls, signal deliveries, and changes of process state.
System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them.
The operation of strace is made possible by the kernel feature known as ptrace.
Some of the features
Attach to an already running process.
$ strace -p 26380 strace: Process 26380 attached ...
Count time, calls, and errors for each system call.
$ strace -c ls > /dev/null % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 89.76 0.008016 4 1912 getdents 8.71 0.000778 0 11778 lstat 0.81 0.000072 0 8894 write 0.60 0.000054 0 943 open 0.11 0.000010 0 942 close 0.00 0.000000 0 1 read 0.00 0.000000 0 944 fstat 0.00 0.000000 0 8 mmap 0.00 0.000000 0 4 mprotect 0.00 0.000000 0 1 munmap 0.00 0.000000 0 7 brk 0.00 0.000000 0 3 3 access 0.00 0.000000 0 1 execve 0.00 0.000000 0 1 sysinfo 0.00 0.000000 0 1 arch_prctl ------ ----------- ----------- --------- --------- ---------------- 100.00 0.008930 25440 3 total