workqueue: Document debugging tricks

It is not obvious how to debug run-away workers. These are some tips given by Tejun on lkml. Signed-off-by: Florian Mickler <florian@mickler.org> Signed-off-by: Tejun Heo <tj@kernel.org>

workqueue: Document debugging tricks
It is not obvious how to debug run-away workers. These are some tips given by Tejun on lkml. Signed-off-by: Florian Mickler <florian@mickler.org> Signed-off-by: Tejun Heo <tj@kernel.org>
Florian Mickler · Tejun Heo
1 parent 6aba74f279
Showing 1 changed file with 40 additions and 0 deletions Side-by-side Diff
Documentation/workqueue.txt
@@ -12,6 +12,7 @@
 4. Application Programming Interface (API)
 5. Example Execution Scenarios
 6. Guidelines
+7. Debugging
  
  
 1. Introduction
@@ -379,4 +380,43 @@
 * Unless work items are expected to consume a huge amount of CPU
   cycles, using a bound wq is usually beneficial due to the increased
   level of locality in wq operations and work item execution.
+
+
+7. Debugging
+
+Because the work functions are executed by generic worker threads
+there are a few tricks needed to shed some light on misbehaving
+workqueue users.
+
+Worker threads show up in the process list as:
+
+root      5671  0.0  0.0      0     0 ?        S    12:07   0:00 [kworker/0:1]
+root      5672  0.0  0.0      0     0 ?        S    12:07   0:00 [kworker/1:2]
+root      5673  0.0  0.0      0     0 ?        S    12:12   0:00 [kworker/0:0]
+root      5674  0.0  0.0      0     0 ?        S    12:13   0:00 [kworker/1:0]
+
+If kworkers are going crazy (using too much cpu), there are two types
+of possible problems:
+
+	1. Something beeing scheduled in rapid succession
+	2. A single work item that consumes lots of cpu cycles
+
+The first one can be tracked using tracing:
+
+	$ echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
+	$ cat /sys/kernel/debug/tracing/trace_pipe > out.txt
+	(wait a few secs)
+	^C
+
+If something is busy looping on work queueing, it would be dominating
+the output and the offender can be determined with the work item
+function.
+
+For the second type of problems it should be possible to just check
+the stack trace of the offending worker thread.
+
+	$ cat /proc/THE_OFFENDING_KWORKER/stack
+
+The work item's function should be trivially visible in the stack
+trace.
...	...	@@ -12,6 +12,7 @@
12	12	4. Application Programming Interface (API)
13	13	5. Example Execution Scenarios
14	14	6. Guidelines
	15	+7. Debugging
15	16
16	17
17	18	1. Introduction
...	...	@@ -379,4 +380,43 @@
379	380	* Unless work items are expected to consume a huge amount of CPU
380	381	cycles, using a bound wq is usually beneficial due to the increased
381	382	level of locality in wq operations and work item execution.
	383	+
	384	+
	385	+7. Debugging
	386	+
	387	+Because the work functions are executed by generic worker threads
	388	+there are a few tricks needed to shed some light on misbehaving
	389	+workqueue users.
	390	+
	391	+Worker threads show up in the process list as:
	392	+
	393	+root 5671 0.0 0.0 0 0 ? S 12:07 0:00 [kworker/0:1]
	394	+root 5672 0.0 0.0 0 0 ? S 12:07 0:00 [kworker/1:2]
	395	+root 5673 0.0 0.0 0 0 ? S 12:12 0:00 [kworker/0:0]
	396	+root 5674 0.0 0.0 0 0 ? S 12:13 0:00 [kworker/1:0]
	397	+
	398	+If kworkers are going crazy (using too much cpu), there are two types
	399	+of possible problems:
	400	+
	401	+ 1. Something beeing scheduled in rapid succession
	402	+ 2. A single work item that consumes lots of cpu cycles
	403	+
	404	+The first one can be tracked using tracing:
	405	+
	406	+ $ echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
	407	+ $ cat /sys/kernel/debug/tracing/trace_pipe > out.txt
	408	+ (wait a few secs)
	409	+ ^C
	410	+
	411	+If something is busy looping on work queueing, it would be dominating
	412	+the output and the offender can be determined with the work item
	413	+function.
	414	+
	415	+For the second type of problems it should be possible to just check
	416	+the stack trace of the offending worker thread.
	417	+
	418	+ $ cat /proc/THE_OFFENDING_KWORKER/stack
	419	+
	420	+The work item's function should be trivially visible in the stack
	421	+trace.