Blame view
Documentation/dev-tools/gdb-kernel-debugging.rst
6.13 KB
5f0962748
|
1 |
.. highlight:: none |
bda1a9216
|
2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
Debugging kernel and modules via gdb ==================================== The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware interfaces allow to debug the Linux kernel and its modules during runtime using gdb. Gdb comes with a powerful scripting interface for python. The kernel provides a collection of helper scripts that can simplify typical kernel debugging steps. This is a short tutorial about how to enable and use them. It focuses on QEMU/KVM virtual machines as target, but the examples can be transferred to the other gdb stubs as well. Requirements ------------ |
5f0962748
|
16 17 |
- gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true for distributions) |
bda1a9216
|
18 19 20 21 |
Setup ----- |
5f0962748
|
22 23 24 25 |
- Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and www.qemu.org for more details). For cross-development, http://landley.net/aboriginal/bin keeps a pool of machine images and toolchains that can be helpful to start from. |
bda1a9216
|
26 |
|
5f0962748
|
27 28 29 |
- Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports CONFIG_FRAME_POINTER, keep it enabled. |
bda1a9216
|
30 |
|
e604f1cb8
|
31 32 |
- Install that kernel on the guest, turn off KASLR if necessary by adding "nokaslr" to the kernel command line. |
5f0962748
|
33 34 35 |
Alternatively, QEMU allows to boot the kernel directly using -kernel, -append, -initrd command line switches. This is generally only useful if you do not depend on modules. See QEMU documentation for more details on |
e604f1cb8
|
36 37 |
this mode. In this case, you should build the kernel with CONFIG_RANDOMIZE_BASE disabled if the architecture supports KASLR. |
bda1a9216
|
38 |
|
5f0962748
|
39 |
- Enable the gdb stub of QEMU/KVM, either |
bda1a9216
|
40 |
|
bda1a9216
|
41 |
- at VM startup time by appending "-s" to the QEMU command line |
5f0962748
|
42 43 |
or |
bda1a9216
|
44 45 |
- during runtime by issuing "gdbserver" from the QEMU monitor console |
5f0962748
|
46 |
- cd /path/to/linux-build |
bda1a9216
|
47 |
|
5f0962748
|
48 |
- Start gdb: gdb vmlinux |
bda1a9216
|
49 |
|
5f0962748
|
50 51 |
Note: Some distros may restrict auto-loading of gdb scripts to known safe directories. In case gdb reports to refuse loading vmlinux-gdb.py, add:: |
bda1a9216
|
52 53 |
add-auto-load-safe-path /path/to/linux-build |
5f0962748
|
54 55 56 |
to ~/.gdbinit. See gdb help for more details. - Attach to the booted guest:: |
bda1a9216
|
57 |
|
bda1a9216
|
58 59 60 61 62 |
(gdb) target remote :1234 Examples of using the Linux-provided gdb helpers ------------------------------------------------ |
5f0962748
|
63 |
- Load module (and main kernel) symbols:: |
bda1a9216
|
64 65 66 67 68 69 70 71 72 73 |
(gdb) lx-symbols loading vmlinux scanning for modules in /home/user/linux/build loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko ... loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko |
5f0962748
|
74 |
- Set a breakpoint on some not yet loaded module function, e.g.:: |
bda1a9216
|
75 76 77 78 |
(gdb) b btrfs_init_sysfs Function "btrfs_init_sysfs" not defined. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 1 (btrfs_init_sysfs) pending. |
5f0962748
|
79 |
- Continue the target:: |
bda1a9216
|
80 |
(gdb) c |
5f0962748
|
81 82 |
- Load the module on the target and watch the symbols being loaded as well as the breakpoint hit:: |
bda1a9216
|
83 84 85 86 87 88 89 |
loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36 36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj); |
5f0962748
|
90 |
- Dump the log buffer of the target kernel:: |
bda1a9216
|
91 92 93 94 95 96 97 98 99 |
(gdb) lx-dmesg [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 3.8.0-rc4-dbg+ (... [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved .... |
5f0962748
|
100 |
- Examine fields of the current task struct:: |
bda1a9216
|
101 102 103 104 |
(gdb) p $lx_current().pid $1 = 4998 (gdb) p $lx_current().comm $2 = "modprobe\000\000\000\000\000\000\000" |
5f0962748
|
105 |
- Make use of the per-cpu function for the current or a specified CPU:: |
bda1a9216
|
106 107 108 109 |
(gdb) p $lx_per_cpu("runqueues").nr_running $3 = 1 (gdb) p $lx_per_cpu("runqueues", 2).nr_running $4 = 0 |
5f0962748
|
110 |
- Dig into hrtimers using the container_of helper:: |
bda1a9216
|
111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 |
(gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next (gdb) p *$container_of($next, "struct hrtimer", "node") $5 = { node = { node = { __rb_parent_color = 18446612133355256072, rb_right = 0x0 <irq_stack_union>, rb_left = 0x0 <irq_stack_union> }, expires = { tv64 = 1835268000000 } }, _softexpires = { tv64 = 1835268000000 }, function = 0xffffffff81078232 <tick_sched_timer>, base = 0xffff88003fd0d6f0, state = 1, start_pid = 0, start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>, start_comm = "swapper/2\000\000\000\000\000\000" } List of commands and functions ------------------------------ The number of commands and convenience functions may evolve over the time, |
5f0962748
|
140 |
this is just a snapshot of the initial version:: |
bda1a9216
|
141 142 143 144 145 146 147 148 149 150 151 152 153 |
(gdb) apropos lx function lx_current -- Return current task function lx_module -- Find module by name and return the module variable function lx_per_cpu -- Return per-cpu variable function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable function lx_thread_info -- Calculate Linux thread_info from task variable lx-dmesg -- Print Linux kernel log buffer lx-lsmod -- List currently loaded modules lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules Detailed help can be obtained via "help <command-name>" for commands and "help function <function-name>" for convenience functions. |