It is packaged in Debian and Ubuntu, but those versions miss some key features. As perf-tools consists of shell scripts (no compilation necessary), I recommend using the GitHub version directly:
git clone https://github.com/brendangregg/perf-tools.git
Two tools that are included are
opensnoop, which trace new
program executions and
open() calls across the whole system.
$ sudo ./execsnoop TIME PID PPID ARGS 21:12:56 22898 15674 ls --color=auto -la 21:12:56 22899 15674 git rev-parse --is-inside-work-tree 21:12:56 22900 15674 git rev-parse --git-dir ... $ sudo ./opensnoop Tracing open()s. Ctrl-C to end. COMM PID FD FILE opensnoop 22924 0x3 /etc/ld.so.cache gawk 22924 0x3 /usr/lib/locale/locale-archive top 15555 0x8 /proc/1/stat ...
Maybe the most interesting tool is
uprobe. It's magic: it traces function
calls in arbitrary user-space programs. With debugging symbols available,
it can trace practically every function in a program. Without them, it can trace
exported functions or arbitrary code locations (specified by raw address).
It can also trace library code, e.g. libc). Having these possibilities on
a production system without any prior setup is staggering.
$ sudo user/uprobe -F -l /tmp/a.out | grep quicksort _Z9quicksortN9__gnu_cxx17__normal_iteratorIPiSt6vectorIiSaIiEEEES5_ $ sudo user/uprobe -F p:/tmp/a.out:_Z9quicksortN9__gnu_cxx17__normal_iteratorIPiSt6vectorIiSaIiEEEES5_ Tracing uprobe _Z9quicksort[snip] (p:_Z9quicksort[snip] /tmp/a.out:0x8ba). Ctrl-C to end. a.out-23171  d... 1860355.891238: _Z9quicksort[snip]: (0x80488ba) a.out-23171  d... 1860355.891353: _Z9quicksort[snip]: (0x80488ba) ...
(To demangle the C++ function names, use the
perf-tools really shows the power of the Linux perf/ftrace infrastructure, and make it usable for the broad masses. There are several other tools that analyze latency and cache hit rates, trace kernel functions, and much more. To finally have such functionality in Linux is fabulous!