Blame view
Documentation/fault-injection/fault-injection.txt
6.52 KB
de1ba09b2 [PATCH] fault inj... |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
Fault injection capabilities infrastructure =========================================== See also drivers/md/faulty.c and "every_nth" module option for scsi_debug. Available fault injection capabilities -------------------------------------- o failslab injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...) o fail_page_alloc injects page allocation failures. (alloc_pages(), get_free_pages(), ...) o fail_make_request |
5d0ffa2b8 [PATCH] fault-inj... |
19 |
injects disk IO errors on devices permitted by setting |
de1ba09b2 [PATCH] fault inj... |
20 21 |
/sys/block/<device>/make-it-fail or /sys/block/<device>/<partition>/make-it-fail. (generic_make_request()) |
1e4cb22bb fault-inject: add... |
22 23 24 25 |
o fail_mmc_request injects MMC data errors on devices permitted by setting debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request |
de1ba09b2 [PATCH] fault inj... |
26 27 28 29 30 31 32 |
Configure fault-injection capabilities behavior ----------------------------------------------- o debugfs entries fault-inject-debugfs kernel module provides some debugfs entries for runtime configuration of fault-injection capabilities. |
156f5a780 debugfs: Fix term... |
33 |
- /sys/kernel/debug/fail*/probability: |
de1ba09b2 [PATCH] fault inj... |
34 35 36 |
likelihood of failure injection, in percent. Format: <percent> |
5d0ffa2b8 [PATCH] fault-inj... |
37 38 |
Note that one-failure-per-hundred is a very high error rate for some testcases. Consider setting probability=100 and configure |
156f5a780 debugfs: Fix term... |
39 |
/sys/kernel/debug/fail*/interval for such testcases. |
de1ba09b2 [PATCH] fault inj... |
40 |
|
156f5a780 debugfs: Fix term... |
41 |
- /sys/kernel/debug/fail*/interval: |
de1ba09b2 [PATCH] fault inj... |
42 43 44 45 46 47 |
specifies the interval between failures, for calls to should_fail() that pass all the other tests. Note that if you enable this, by setting interval>1, you will probably want to set probability=100. |
156f5a780 debugfs: Fix term... |
48 |
- /sys/kernel/debug/fail*/times: |
de1ba09b2 [PATCH] fault inj... |
49 50 51 |
specifies how many times failures may happen at most. A value of -1 means "no limit". |
156f5a780 debugfs: Fix term... |
52 |
- /sys/kernel/debug/fail*/space: |
de1ba09b2 [PATCH] fault inj... |
53 54 55 56 |
specifies an initial resource "budget", decremented by "size" on each call to should_fail(,size). Failure injection is suppressed until "space" reaches zero. |
156f5a780 debugfs: Fix term... |
57 |
- /sys/kernel/debug/fail*/verbose |
de1ba09b2 [PATCH] fault inj... |
58 59 |
Format: { 0 | 1 | 2 } |
5d0ffa2b8 [PATCH] fault-inj... |
60 61 62 63 |
specifies the verbosity of the messages when failure is injected. '0' means no messages; '1' will print only a single log line per failure; '2' will print a call trace too -- useful to debug the problems revealed by fault injection. |
de1ba09b2 [PATCH] fault inj... |
64 |
|
156f5a780 debugfs: Fix term... |
65 |
- /sys/kernel/debug/fail*/task-filter: |
de1ba09b2 [PATCH] fault inj... |
66 |
|
5d0ffa2b8 [PATCH] fault-inj... |
67 68 |
Format: { 'Y' | 'N' } A value of 'N' disables filtering by process (default). |
de1ba09b2 [PATCH] fault inj... |
69 70 |
Any positive value limits failures to only processes indicated by /proc/<pid>/make-it-fail==1. |
156f5a780 debugfs: Fix term... |
71 72 73 74 |
- /sys/kernel/debug/fail*/require-start: - /sys/kernel/debug/fail*/require-end: - /sys/kernel/debug/fail*/reject-start: - /sys/kernel/debug/fail*/reject-end: |
de1ba09b2 [PATCH] fault inj... |
75 76 77 |
specifies the range of virtual addresses tested during stacktrace walking. Failure is injected only if some caller |
329409aed [PATCH] fault inj... |
78 79 80 81 |
in the walked stacktrace lies within the required range, and none lies within the rejected range. Default required range is [0,ULONG_MAX) (whole of virtual address space). Default rejected range is [0,0). |
de1ba09b2 [PATCH] fault inj... |
82 |
|
156f5a780 debugfs: Fix term... |
83 |
- /sys/kernel/debug/fail*/stacktrace-depth: |
de1ba09b2 [PATCH] fault inj... |
84 85 |
specifies the maximum stacktrace depth walked during search |
5d0ffa2b8 [PATCH] fault-inj... |
86 87 |
for a caller within [require-start,require-end) OR [reject-start,reject-end). |
de1ba09b2 [PATCH] fault inj... |
88 |
|
156f5a780 debugfs: Fix term... |
89 |
- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem: |
de1ba09b2 [PATCH] fault inj... |
90 |
|
5d0ffa2b8 [PATCH] fault-inj... |
91 92 |
Format: { 'Y' | 'N' } default is 'N', setting it to 'Y' won't inject failures into |
de1ba09b2 [PATCH] fault inj... |
93 |
highmem/user allocations. |
156f5a780 debugfs: Fix term... |
94 95 |
- /sys/kernel/debug/failslab/ignore-gfp-wait: - /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait: |
de1ba09b2 [PATCH] fault inj... |
96 |
|
5d0ffa2b8 [PATCH] fault-inj... |
97 98 |
Format: { 'Y' | 'N' } default is 'N', setting it to 'Y' will inject failures |
de1ba09b2 [PATCH] fault inj... |
99 |
only into non-sleep allocations (GFP_ATOMIC allocations). |
156f5a780 debugfs: Fix term... |
100 |
- /sys/kernel/debug/fail_page_alloc/min-order: |
54114994f fault-injection: ... |
101 102 103 |
specifies the minimum page allocation order to be injected failures. |
de1ba09b2 [PATCH] fault inj... |
104 105 106 107 108 109 110 |
o Boot option In order to inject faults while debugfs is not available (early boot time), use the boot option: failslab= fail_page_alloc= |
1e4cb22bb fault-inject: add... |
111 |
fail_make_request= |
199e3f4b4 fault-injection: ... |
112 |
mmc_core.fail_request=<interval>,<probability>,<space>,<times> |
de1ba09b2 [PATCH] fault inj... |
113 114 115 116 117 118 119 120 121 122 123 124 |
How to add new fault injection capability ----------------------------------------- o #include <linux/fault-inject.h> o define the fault attributes DECLARE_FAULT_INJECTION(name); Please see the definition of struct fault_attr in fault-inject.h for details. |
5d0ffa2b8 [PATCH] fault-inj... |
125 |
o provide a way to configure fault attributes |
de1ba09b2 [PATCH] fault inj... |
126 127 128 129 |
- boot option If you need to enable the fault injection capability from boot time, you can |
5d0ffa2b8 [PATCH] fault-inj... |
130 |
provide boot option to configure it. There is a helper function for it: |
de1ba09b2 [PATCH] fault inj... |
131 |
|
5d0ffa2b8 [PATCH] fault-inj... |
132 |
setup_fault_attr(attr, str); |
de1ba09b2 [PATCH] fault inj... |
133 134 135 136 |
- debugfs entries failslab, fail_page_alloc, and fail_make_request use this way. |
5d0ffa2b8 [PATCH] fault-inj... |
137 |
Helper functions: |
de1ba09b2 [PATCH] fault inj... |
138 |
|
dd48c085c fault-injection: ... |
139 |
fault_create_debugfs_attr(name, parent, attr); |
de1ba09b2 [PATCH] fault inj... |
140 141 142 143 144 145 146 147 |
- module parameters If the scope of the fault injection capability is limited to a single kernel module, it is better to provide module parameters to configure the fault attributes. o add a hook to insert failures |
5d0ffa2b8 [PATCH] fault-inj... |
148 |
Upon should_fail() returning true, client code should inject a failure. |
de1ba09b2 [PATCH] fault inj... |
149 |
|
5d0ffa2b8 [PATCH] fault-inj... |
150 |
should_fail(attr, size); |
de1ba09b2 [PATCH] fault inj... |
151 152 153 |
Application Examples -------------------- |
185848707 fault-injection: ... |
154 |
o Inject slab allocation failures into module init/exit code |
de1ba09b2 [PATCH] fault inj... |
155 |
|
de1ba09b2 [PATCH] fault inj... |
156 |
#!/bin/bash |
185848707 fault-injection: ... |
157 |
FAILTYPE=failslab |
156f5a780 debugfs: Fix term... |
158 159 160 161 162 163 164 |
echo Y > /sys/kernel/debug/$FAILTYPE/task-filter echo 10 > /sys/kernel/debug/$FAILTYPE/probability echo 100 > /sys/kernel/debug/$FAILTYPE/interval echo -1 > /sys/kernel/debug/$FAILTYPE/times echo 0 > /sys/kernel/debug/$FAILTYPE/space echo 2 > /sys/kernel/debug/$FAILTYPE/verbose echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait |
de1ba09b2 [PATCH] fault inj... |
165 |
|
185848707 fault-injection: ... |
166 |
faulty_system() |
de1ba09b2 [PATCH] fault inj... |
167 |
{ |
185848707 fault-injection: ... |
168 |
bash -c "echo 1 > /proc/self/make-it-fail && exec $*" |
de1ba09b2 [PATCH] fault inj... |
169 |
} |
185848707 fault-injection: ... |
170 171 172 173 174 175 176 177 178 179 |
if [ $# -eq 0 ] then echo "Usage: $0 modulename [ modulename ... ]" exit 1 fi for m in $* do echo inserting $m... faulty_system modprobe $m |
de1ba09b2 [PATCH] fault inj... |
180 |
|
185848707 fault-injection: ... |
181 182 183 |
echo removing $m... faulty_system modprobe -r $m done |
de1ba09b2 [PATCH] fault inj... |
184 185 |
------------------------------------------------------------------------------ |
185848707 fault-injection: ... |
186 |
o Inject page allocation failures only for a specific module |
de1ba09b2 [PATCH] fault inj... |
187 |
|
de1ba09b2 [PATCH] fault inj... |
188 |
#!/bin/bash |
185848707 fault-injection: ... |
189 190 |
FAILTYPE=fail_page_alloc module=$1 |
de1ba09b2 [PATCH] fault inj... |
191 |
|
185848707 fault-injection: ... |
192 193 194 195 196 |
if [ -z $module ] then echo "Usage: $0 <modulename>" exit 1 fi |
de1ba09b2 [PATCH] fault inj... |
197 |
|
185848707 fault-injection: ... |
198 |
modprobe $module |
de1ba09b2 [PATCH] fault inj... |
199 |
|
185848707 fault-injection: ... |
200 201 202 203 204 |
if [ ! -d /sys/module/$module/sections ] then echo Module $module is not loaded exit 1 fi |
156f5a780 debugfs: Fix term... |
205 206 |
cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end |
185848707 fault-injection: ... |
207 |
|
156f5a780 debugfs: Fix term... |
208 209 210 211 212 213 214 215 216 |
echo N > /sys/kernel/debug/$FAILTYPE/task-filter echo 10 > /sys/kernel/debug/$FAILTYPE/probability echo 100 > /sys/kernel/debug/$FAILTYPE/interval echo -1 > /sys/kernel/debug/$FAILTYPE/times echo 0 > /sys/kernel/debug/$FAILTYPE/space echo 2 > /sys/kernel/debug/$FAILTYPE/verbose echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth |
185848707 fault-injection: ... |
217 |
|
156f5a780 debugfs: Fix term... |
218 |
trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT |
185848707 fault-injection: ... |
219 220 221 |
echo "Injecting errors into the module $module... (interrupt to stop)" sleep 1000000 |
de1ba09b2 [PATCH] fault inj... |
222 |