Blame view
Documentation/admin-guide/dynamic-debug-howto.rst
13 KB
7d4e3517b Documentation/dyn... |
1 2 |
Dynamic debug +++++++++++++ |
86151fdf3 dynamic debug: up... |
3 4 5 |
Introduction ============ |
29e36c9ff dynamic_debug: up... |
6 |
This document describes how to use the dynamic debug (dyndbg) feature. |
86151fdf3 dynamic debug: up... |
7 |
|
29e36c9ff dynamic_debug: up... |
8 9 |
Dynamic debug is designed to allow you to dynamically enable/disable kernel code to obtain additional kernel information. Currently, if |
7d4e3517b Documentation/dyn... |
10 11 |
``CONFIG_DYNAMIC_DEBUG`` is set, then all ``pr_debug()``/``dev_dbg()`` and ``print_hex_dump_debug()``/``print_hex_dump_bytes()`` calls can be dynamically |
7a555613e dynamic_debug: dy... |
12 |
enabled per-callsite. |
7d4e3517b Documentation/dyn... |
13 14 |
If ``CONFIG_DYNAMIC_DEBUG`` is not set, ``print_hex_dump_debug()`` is just shortcut for ``print_hex_dump(KERN_DEBUG)``. |
7a555613e dynamic_debug: dy... |
15 |
|
7d4e3517b Documentation/dyn... |
16 17 18 |
For ``print_hex_dump_debug()``/``print_hex_dump_bytes()``, format string is its ``prefix_str`` argument, if it is constant string; or ``hexdump`` in case ``prefix_str`` is build dynamically. |
86151fdf3 dynamic debug: up... |
19 20 |
Dynamic debug has even more useful features: |
29e36c9ff dynamic_debug: up... |
21 22 |
* Simple query language allows turning on and off debugging statements by matching any combination of 0 or 1 of: |
86151fdf3 dynamic debug: up... |
23 24 25 26 27 28 |
- source filename - function name - line number (including ranges of line numbers) - module name - format string |
7d4e3517b Documentation/dyn... |
29 |
* Provides a debugfs control file: ``<debugfs>/dynamic_debug/control`` |
29e36c9ff dynamic_debug: up... |
30 31 |
which can be read to display the complete list of known debug statements, to help guide you |
86151fdf3 dynamic debug: up... |
32 33 |
Controlling dynamic debug Behaviour |
a648ec05b Dynamic Debug: In... |
34 |
=================================== |
86151fdf3 dynamic debug: up... |
35 |
|
7d4e3517b Documentation/dyn... |
36 |
The behaviour of ``pr_debug()``/``dev_dbg()`` are controlled via writing to a |
29e36c9ff dynamic_debug: up... |
37 38 39 |
control file in the 'debugfs' filesystem. Thus, you must first mount the debugfs filesystem, in order to make use of this feature. Subsequently, we refer to the control file as: |
7d4e3517b Documentation/dyn... |
40 41 |
``<debugfs>/dynamic_debug/control``. For example, if you want to enable printing from source file ``svcsock.c``, line 1603 you simply do:: |
86151fdf3 dynamic debug: up... |
42 |
|
7d4e3517b Documentation/dyn... |
43 |
nullarbor:~ # echo 'file svcsock.c line 1603 +p' > |
86151fdf3 dynamic debug: up... |
44 |
<debugfs>/dynamic_debug/control |
7d4e3517b Documentation/dyn... |
45 |
If you make a mistake with the syntax, the write will fail thus:: |
86151fdf3 dynamic debug: up... |
46 |
|
7d4e3517b Documentation/dyn... |
47 |
nullarbor:~ # echo 'file svcsock.c wtf 1 +p' > |
86151fdf3 dynamic debug: up... |
48 |
<debugfs>/dynamic_debug/control |
7d4e3517b Documentation/dyn... |
49 |
-bash: echo: write error: Invalid argument |
86151fdf3 dynamic debug: up... |
50 51 |
Viewing Dynamic Debug Behaviour |
7d4e3517b Documentation/dyn... |
52 |
=============================== |
86151fdf3 dynamic debug: up... |
53 |
|
29e36c9ff dynamic_debug: up... |
54 |
You can view the currently configured behaviour of all the debug |
7d4e3517b Documentation/dyn... |
55 |
statements via:: |
86151fdf3 dynamic debug: up... |
56 |
|
7d4e3517b Documentation/dyn... |
57 58 59 60 61 62 63 |
nullarbor:~ # cat <debugfs>/dynamic_debug/control # filename:lineno [module]function flags format /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:323 [svcxprt_rdma]svc_rdma_cleanup =_ "SVCRDMA Module Removed, deregister RPC RDMA transport\012" /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:341 [svcxprt_rdma]svc_rdma_init =_ "\011max_inline : %d\012" /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:340 [svcxprt_rdma]svc_rdma_init =_ "\011sq_depth : %d\012" /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:338 [svcxprt_rdma]svc_rdma_init =_ "\011max_requests : %d\012" ... |
86151fdf3 dynamic debug: up... |
64 65 66 |
You can also apply standard Unix text manipulation filters to this |
7d4e3517b Documentation/dyn... |
67 |
data, e.g.:: |
86151fdf3 dynamic debug: up... |
68 |
|
7d4e3517b Documentation/dyn... |
69 70 |
nullarbor:~ # grep -i rdma <debugfs>/dynamic_debug/control | wc -l 62 |
86151fdf3 dynamic debug: up... |
71 |
|
7d4e3517b Documentation/dyn... |
72 73 |
nullarbor:~ # grep -i tcp <debugfs>/dynamic_debug/control | wc -l 42 |
86151fdf3 dynamic debug: up... |
74 |
|
29e36c9ff dynamic_debug: up... |
75 76 |
The third column shows the currently enabled flags for each debug statement callsite (see below for definitions of the flags). The |
7d4e3517b Documentation/dyn... |
77 78 |
default value, with no flags enabled, is ``=_``. So you can view all the debug statement callsites with any non-default flags:: |
86151fdf3 dynamic debug: up... |
79 |
|
7d4e3517b Documentation/dyn... |
80 81 82 |
nullarbor:~ # awk '$3 != "=_"' <debugfs>/dynamic_debug/control # filename:lineno [module]function flags format /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svcsock.c:1603 [sunrpc]svc_send p "svc_process: st_sendto returned %d\012" |
86151fdf3 dynamic debug: up... |
83 84 85 86 87 |
Command Language Reference ========================== At the lexical level, a command comprises a sequence of words separated |
7d4e3517b Documentation/dyn... |
88 |
by spaces or tabs. So these are all equivalent:: |
86151fdf3 dynamic debug: up... |
89 |
|
31fc93d5f dynamic-debug-how... |
90 |
nullarbor:~ # echo -n 'file svcsock.c line 1603 +p' > |
86151fdf3 dynamic debug: up... |
91 |
<debugfs>/dynamic_debug/control |
31fc93d5f dynamic-debug-how... |
92 |
nullarbor:~ # echo -n ' file svcsock.c line 1603 +p ' > |
86151fdf3 dynamic debug: up... |
93 |
<debugfs>/dynamic_debug/control |
7d4e3517b Documentation/dyn... |
94 |
nullarbor:~ # echo -n 'file svcsock.c line 1603 +p' > |
86151fdf3 dynamic debug: up... |
95 |
<debugfs>/dynamic_debug/control |
85f7f6c0e dynamic_debug: pr... |
96 |
Command submissions are bounded by a write() system call. |
7d4e3517b Documentation/dyn... |
97 98 |
Multiple commands can be written together, separated by ``;`` or `` ``:: |
86151fdf3 dynamic debug: up... |
99 |
|
85f7f6c0e dynamic_debug: pr... |
100 101 |
~# echo "func pnpacpi_get_resources +p; func pnp_assign_mem +p" \ > <debugfs>/dynamic_debug/control |
86151fdf3 dynamic debug: up... |
102 |
|
7d4e3517b Documentation/dyn... |
103 |
If your query set is big, you can batch them too:: |
86151fdf3 dynamic debug: up... |
104 |
|
85f7f6c0e dynamic_debug: pr... |
105 |
~# cat query-batch-file > <debugfs>/dynamic_debug/control |
86151fdf3 dynamic debug: up... |
106 |
|
7d4e3517b Documentation/dyn... |
107 108 109 |
A another way is to use wildcard. The match rule support ``*`` (matches zero or more characters) and ``?`` (matches exactly one character).For example, you can match all usb drivers:: |
8f073bd0d dynamic-debug-how... |
110 111 |
~# echo "file drivers/usb/* +p" > <debugfs>/dynamic_debug/control |
86151fdf3 dynamic debug: up... |
112 |
At the syntactical level, a command comprises a sequence of match |
7d4e3517b Documentation/dyn... |
113 |
specifications, followed by a flags change specification:: |
86151fdf3 dynamic debug: up... |
114 |
|
7d4e3517b Documentation/dyn... |
115 |
command ::= match-spec* flags-spec |
86151fdf3 dynamic debug: up... |
116 |
|
29e36c9ff dynamic_debug: up... |
117 |
The match-spec's are used to choose a subset of the known pr_debug() |
86151fdf3 dynamic debug: up... |
118 119 |
callsites to which to apply the flags-spec. Think of them as a query with implicit ANDs between each pair. Note that an empty list of |
29e36c9ff dynamic_debug: up... |
120 |
match-specs will select all debug statement callsites. |
86151fdf3 dynamic debug: up... |
121 |
|
29e36c9ff dynamic_debug: up... |
122 123 |
A match specification comprises a keyword, which controls the attribute of the callsite to be compared, and a value to compare |
7d4e3517b Documentation/dyn... |
124 125 126 127 128 129 130 |
against. Possible keywords are::: match-spec ::= 'func' string | 'file' string | 'module' string | 'format' string | 'line' line-range |
86151fdf3 dynamic debug: up... |
131 |
|
7d4e3517b Documentation/dyn... |
132 133 134 135 |
line-range ::= lineno | '-'lineno | lineno'-' | lineno'-'lineno |
86151fdf3 dynamic debug: up... |
136 |
|
7d4e3517b Documentation/dyn... |
137 138 139 140 141 142 |
lineno ::= unsigned-int .. note:: ``line-range`` cannot contain space, e.g. "1-30" is valid range but "1 - 30" is not. |
86151fdf3 dynamic debug: up... |
143 |
|
86151fdf3 dynamic debug: up... |
144 145 146 147 148 |
The meanings of each keyword are: func The given string is compared against the function name |
7d4e3517b Documentation/dyn... |
149 |
of each callsite. Example:: |
86151fdf3 dynamic debug: up... |
150 |
|
7d4e3517b Documentation/dyn... |
151 |
func svc_tcp_accept |
86151fdf3 dynamic debug: up... |
152 153 |
file |
2b6783191 dynamic_debug: ad... |
154 155 |
The given string is compared against either the full pathname, the src-root relative pathname, or the basename of the source file of |
7d4e3517b Documentation/dyn... |
156 |
each callsite. Examples:: |
86151fdf3 dynamic debug: up... |
157 |
|
7d4e3517b Documentation/dyn... |
158 159 160 |
file svcsock.c file kernel/freezer.c file /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svcsock.c |
86151fdf3 dynamic debug: up... |
161 162 163 164 |
module The given string is compared against the module name of each callsite. The module name is the string as |
7d4e3517b Documentation/dyn... |
165 166 |
seen in ``lsmod``, i.e. without the directory or the ``.ko`` suffix and with ``-`` changed to ``_``. Examples:: |
86151fdf3 dynamic debug: up... |
167 |
|
7d4e3517b Documentation/dyn... |
168 169 |
module sunrpc module nfsd |
86151fdf3 dynamic debug: up... |
170 171 172 173 174 175 |
format The given string is searched for in the dynamic debug format string. Note that the string does not need to match the entire format, only some part. Whitespace and other special characters can be escaped using C octal character |
7d4e3517b Documentation/dyn... |
176 |
escape ``\ooo`` notation, e.g. the space character is ``\040``. |
9898abb3d Dynamic debug: al... |
177 |
Alternatively, the string can be enclosed in double quote |
7d4e3517b Documentation/dyn... |
178 179 |
characters (``"``) or single quote characters (``'``). Examples:: |
86151fdf3 dynamic debug: up... |
180 |
|
7d4e3517b Documentation/dyn... |
181 182 183 184 185 |
format svcrdma: // many of the NFS/RDMA server pr_debugs format readahead // some pr_debugs in the readahead cache format nfsd:\040SETATTR // one way to match a format with whitespace format "nfsd: SETATTR" // a neater way to match a format with whitespace format 'nfsd: SETATTR' // yet another way to match a format with whitespace |
86151fdf3 dynamic debug: up... |
186 187 188 |
line The given line number or range of line numbers is compared |
7d4e3517b Documentation/dyn... |
189 |
against the line number of each ``pr_debug()`` callsite. A single |
86151fdf3 dynamic debug: up... |
190 191 192 193 |
line number matches the callsite line number exactly. A range of line numbers matches any callsite between the first and last line number inclusive. An empty first number means the first line in the file, an empty line number means the |
7d4e3517b Documentation/dyn... |
194 |
last number in the file. Examples:: |
86151fdf3 dynamic debug: up... |
195 |
|
7d4e3517b Documentation/dyn... |
196 197 198 199 |
line 1603 // exactly line 1603 line 1600-1605 // the six lines from line 1600 to line 1605 line -1605 // the 1605 lines from line 1 to line 1605 line 1600- // all lines from line 1600 to the end of the file |
86151fdf3 dynamic debug: up... |
200 201 202 |
The flags specification comprises a change operation followed by one or more flag characters. The change operation is one |
7d4e3517b Documentation/dyn... |
203 |
of the characters:: |
86151fdf3 dynamic debug: up... |
204 |
|
29e36c9ff dynamic_debug: up... |
205 206 207 |
- remove the given flags + add the given flags = set the flags to the given flags |
86151fdf3 dynamic debug: up... |
208 |
|
7d4e3517b Documentation/dyn... |
209 |
The flags are:: |
86151fdf3 dynamic debug: up... |
210 |
|
29e36c9ff dynamic_debug: up... |
211 212 213 214 215 216 |
p enables the pr_debug() callsite. f Include the function name in the printed message l Include line number in the printed message m Include module name in the printed message t Include thread ID in messages not generated from interrupt context _ No flags are set. (Or'd with others on input) |
7d4e3517b Documentation/dyn... |
217 |
For ``print_hex_dump_debug()`` and ``print_hex_dump_bytes()``, only ``p`` flag |
7a555613e dynamic_debug: dy... |
218 |
have meaning, other flags ignored. |
7d4e3517b Documentation/dyn... |
219 |
For display, the flags are preceded by ``=`` |
29e36c9ff dynamic_debug: up... |
220 |
(mnemonic: what the flags are currently equal to). |
86151fdf3 dynamic debug: up... |
221 |
|
7d4e3517b Documentation/dyn... |
222 223 |
Note the regexp ``^[-+=][flmpt_]+$`` matches a flags specification. To clear all flags at once, use ``=_`` or ``-flmpt``. |
86151fdf3 dynamic debug: up... |
224 |
|
a648ec05b Dynamic Debug: In... |
225 |
|
29e36c9ff dynamic_debug: up... |
226 |
Debug messages during Boot Process |
a648ec05b Dynamic Debug: In... |
227 |
================================== |
29e36c9ff dynamic_debug: up... |
228 229 |
To activate debug messages for core code and built-in modules during the boot process, even before userspace and debugfs exists, use |
7d4e3517b Documentation/dyn... |
230 231 |
``dyndbg="QUERY"``, ``module.dyndbg="QUERY"``, or ``ddebug_query="QUERY"`` (``ddebug_query`` is obsoleted by ``dyndbg``, and deprecated). QUERY follows |
29e36c9ff dynamic_debug: up... |
232 233 |
the syntax described above, but must not exceed 1023 characters. Your bootloader may impose lower limits. |
7d4e3517b Documentation/dyn... |
234 |
These ``dyndbg`` params are processed just after the ddebug tables are |
29e36c9ff dynamic_debug: up... |
235 236 237 |
processed, as part of the arch_initcall. Thus you can enable debug messages in all code run after this arch_initcall via this boot parameter. |
a648ec05b Dynamic Debug: In... |
238 |
|
7d4e3517b Documentation/dyn... |
239 |
On an x86 system for example ACPI enablement is a subsys_initcall and:: |
29e36c9ff dynamic_debug: up... |
240 |
dyndbg="file ec.c +p" |
7d4e3517b Documentation/dyn... |
241 |
|
a648ec05b Dynamic Debug: In... |
242 243 244 245 |
will show early Embedded Controller transactions during ACPI setup if your machine (typically a laptop) has an Embedded Controller. PCI (or other devices) initialization also is a hot candidate for using this boot parameter for debugging purposes. |
7d4e3517b Documentation/dyn... |
246 |
If ``foo`` module is not built-in, ``foo.dyndbg`` will still be processed at |
29e36c9ff dynamic_debug: up... |
247 |
boot time, without effect, but will be reprocessed when module is |
7d4e3517b Documentation/dyn... |
248 |
loaded later. ``dyndbg_query=`` and bare ``dyndbg=`` are only processed at |
29e36c9ff dynamic_debug: up... |
249 250 251 252 253 |
boot. Debug Messages at Module Initialization Time ============================================ |
7d4e3517b Documentation/dyn... |
254 255 256 |
When ``modprobe foo`` is called, modprobe scans ``/proc/cmdline`` for ``foo.params``, strips ``foo.``, and passes them to the kernel along with params given in modprobe args or ``/etc/modprob.d/*.conf`` files, |
29e36c9ff dynamic_debug: up... |
257 |
in the following order: |
7d4e3517b Documentation/dyn... |
258 259 260 261 262 263 |
1. parameters given via ``/etc/modprobe.d/*.conf``:: options foo dyndbg=+pt options foo dyndbg # defaults to +p 2. ``foo.dyndbg`` as given in boot args, ``foo.`` is stripped and passed:: |
29e36c9ff dynamic_debug: up... |
264 |
|
7d4e3517b Documentation/dyn... |
265 |
foo.dyndbg=" func bar +p; func buz +mp" |
29e36c9ff dynamic_debug: up... |
266 |
|
7d4e3517b Documentation/dyn... |
267 |
3. args to modprobe:: |
29e36c9ff dynamic_debug: up... |
268 |
|
7d4e3517b Documentation/dyn... |
269 270 271 272 |
modprobe foo dyndbg==pmf # override previous settings These ``dyndbg`` queries are applied in order, with last having final say. This allows boot args to override or modify those from ``/etc/modprobe.d`` |
29e36c9ff dynamic_debug: up... |
273 274 |
(sensible, since 1 is system wide, 2 is kernel or boot specific), and modprobe args to override both. |
7d4e3517b Documentation/dyn... |
275 276 277 |
In the ``foo.dyndbg="QUERY"`` form, the query must exclude ``module foo``. ``foo`` is extracted from the param-name, and applied to each query in ``QUERY``, and only 1 match-spec of each type is allowed. |
29e36c9ff dynamic_debug: up... |
278 |
|
7d4e3517b Documentation/dyn... |
279 |
The ``dyndbg`` option is a "fake" module parameter, which means: |
29e36c9ff dynamic_debug: up... |
280 281 282 |
- modules do not need to define it explicitly - every module gets it tacitly, whether they use pr_debug or not |
7d4e3517b Documentation/dyn... |
283 284 |
- it doesn't appear in ``/sys/module/$module/parameters/`` To see it, grep the control file, or inspect ``/proc/cmdline.`` |
29e36c9ff dynamic_debug: up... |
285 |
|
7d4e3517b Documentation/dyn... |
286 287 288 |
For ``CONFIG_DYNAMIC_DEBUG`` kernels, any settings given at boot-time (or enabled by ``-DDEBUG`` flag during compilation) can be disabled later via the sysfs interface if the debug messages are no longer needed:: |
29e36c9ff dynamic_debug: up... |
289 290 |
echo "module module_name -p" > <debugfs>/dynamic_debug/control |
a648ec05b Dynamic Debug: In... |
291 |
|
86151fdf3 dynamic debug: up... |
292 293 |
Examples ======== |
7d4e3517b Documentation/dyn... |
294 295 296 297 |
:: // enable the message at line 1603 of file svcsock.c nullarbor:~ # echo -n 'file svcsock.c line 1603 +p' > |
86151fdf3 dynamic debug: up... |
298 |
<debugfs>/dynamic_debug/control |
7d4e3517b Documentation/dyn... |
299 300 |
// enable all the messages in file svcsock.c nullarbor:~ # echo -n 'file svcsock.c +p' > |
86151fdf3 dynamic debug: up... |
301 |
<debugfs>/dynamic_debug/control |
7d4e3517b Documentation/dyn... |
302 303 |
// enable all the messages in the NFS server module nullarbor:~ # echo -n 'module nfsd +p' > |
86151fdf3 dynamic debug: up... |
304 |
<debugfs>/dynamic_debug/control |
7d4e3517b Documentation/dyn... |
305 306 |
// enable all 12 messages in the function svc_process() nullarbor:~ # echo -n 'func svc_process +p' > |
86151fdf3 dynamic debug: up... |
307 |
<debugfs>/dynamic_debug/control |
7d4e3517b Documentation/dyn... |
308 309 |
// disable all 12 messages in the function svc_process() nullarbor:~ # echo -n 'func svc_process -p' > |
86151fdf3 dynamic debug: up... |
310 |
<debugfs>/dynamic_debug/control |
9898abb3d Dynamic debug: al... |
311 |
|
7d4e3517b Documentation/dyn... |
312 313 |
// enable messages for NFS calls READ, READLINK, READDIR and READDIR+. nullarbor:~ # echo -n 'format "nfsd: READ" +p' > |
9898abb3d Dynamic debug: al... |
314 |
<debugfs>/dynamic_debug/control |
29e36c9ff dynamic_debug: up... |
315 |
|
7d4e3517b Documentation/dyn... |
316 317 |
// enable messages in files of which the paths include string "usb" nullarbor:~ # echo -n '*usb* +p' > <debugfs>/dynamic_debug/control |
8f073bd0d dynamic-debug-how... |
318 |
|
7d4e3517b Documentation/dyn... |
319 320 |
// enable all messages nullarbor:~ # echo -n '+p' > <debugfs>/dynamic_debug/control |
29e36c9ff dynamic_debug: up... |
321 |
|
7d4e3517b Documentation/dyn... |
322 323 |
// add module, function to all enabled messages nullarbor:~ # echo -n '+mf' > <debugfs>/dynamic_debug/control |
29e36c9ff dynamic_debug: up... |
324 |
|
7d4e3517b Documentation/dyn... |
325 326 327 328 329 330 331 332 |
// boot-args example, with newlines and comments for readability Kernel command line: ... // see whats going on in dyndbg=value processing dynamic_debug.verbose=1 // enable pr_debugs in 2 builtins, #cmt is stripped dyndbg="module params +p #cmt ; module sys +p" // enable pr_debugs in 2 functions in a module loaded later pc87360.dyndbg="func pc87360_init_device +p; func pc87360_find +p" |