Ftrace Favorites Cheat Sheet - Fun Commands to Try with Ftrace

When you see the ftrace presentation or read through the presentations, you get an overload. It's just too much at once. That's why I decided to write up this little cheat sheet for ftrace with commands from Steve's presentations.

Reset everything:

echo | tee set_* 

echo 0 | tee tracing_enabled tracing_max_latency

Trace fsyncs:
trace-cmd record -g '*fsync*' -p function_graph -s 10000 -b 10000
Makes a nice small log only of the fsyncs on your system. Good to check what's waking up your hard disk. The low frequency (-s) effects your system less and the large buffer (-b) makes sure no tracing data is lost.

Trace somemodule:
echo '*somemodulename*' > set_ftrace_filter
e.g. echo '*ext4*' > set_ftrace_filter
or echo :mod:e1000e > set_ftrace_filter

Switch tracing on/off when a certain function is called:
echo bad_inode_create:traceoff > set_ftrace_filter

Check IRQ Latencies:
echo do_IRQ > set_ftrace_filter
echo function_graph > current_tracer
echo 1 > events/irq/irq_handler_entry/enable

Check for code areas where IRQs are disabled
echo irqsoff > current_tracer

Trance a certain function, e.g. sys_read:
echo sys_read > set_graph_function

Enable events:
echo 1 > events/enable -> all echo
or echo 1 > events/sched/enable -> just some,
sched > set_event -> the same command
"available_events shows what events are available"

Further Filtering on an by event basis:
echo "prev_state == 0" > events/sched/sched_switch/filter

Stack Tracing:
echo 1 > /proc/sys/kernel/stack_tracer_enabled
cat stack_trace

Don't trace locks:
echo '*lock*' > set_ftrace_notrace

Ftrace on oops:
echo 1 > /proc/sys/kernel/ftrace_dump_on_oops
sysrq-z -> get the dump

Trace a command:
trace-cmd record -o fgraph-events.dat -e sched -p function_graph \
ls -ltr /usr > /dev/null
"-f : filter
-O : option
-g : same as echoing into set_graph_function
-l : same as echoing into set_ftrace_filter
-n : same as echoing into set_ftrace_notrace"

Trace via network:
trace-cmd listen -p 5678 -d
trace-cmd record -N host:5678 -e all

Disable all tracing:
trace-cmd reset

And if that's all unclear or not enough, try to catch one of Steve's presentations and check his slides.

And if you wonder why in his slides he likes to use 'cat | head' instead of just 'head', here's his answer to that: Most of these tools are not efficient with non-linear reads. So cat is slower to type, but works faster as it forces a linear read.

And feel free to post your favorites!