Document the slow work thread pool

Document the slow work thread pool. Signed-off-by: David Howells <dhowells@redhat.com> Acked-by: Steve Dickson <steved@redhat.com> Acked-by: Trond Myklebust <Trond.Myklebust@netapp.com> Acked-by: Al Viro <viro@zeniv.linux.org.uk> Tested-by: Daire Byrne <Daire.Byrne@framestore.com>

Document the slow work thread pool
Document the slow work thread pool. Signed-off-by: David Howells <dhowells@redhat.com> Acked-by: Steve Dickson <steved@redhat.com> Acked-by: Trond Myklebust <Trond.Myklebust@netapp.com> Acked-by: Al Viro <viro@zeniv.linux.org.uk> Tested-by: Daire Byrne <Daire.Byrne@framestore.com>
David Howells
1 parent 12e22c5e4b
Showing 3 changed files with 178 additions and 0 deletions Side-by-side Diff
Documentation/slow-work.txt
include/linux/slow-work.h
kernel/slow-work.c
+		     ====================================
+		     SLOW WORK ITEM EXECUTION THREAD POOL
+		     ====================================
+
+By: David Howells <dhowells@redhat.com>
+
+The slow work item execution thread pool is a pool of threads for performing
+things that take a relatively long time, such as making mkdir calls.
+Typically, when processing something, these items will spend a lot of time
+blocking a thread on I/O, thus making that thread unavailable for doing other
+work.
+
+The standard workqueue model is unsuitable for this class of work item as that
+limits the owner to a single thread or a single thread per CPU.  For some
+tasks, however, more threads - or fewer - are required.
+
+There is just one pool per system.  It contains no threads unless something
+wants to use it - and that something must register its interest first.  When
+the pool is active, the number of threads it contains is dynamic, varying
+between a maximum and minimum setting, depending on the load.
+
+
+====================
+CLASSES OF WORK ITEM
+====================
+
+This pool support two classes of work items:
+
+ (*) Slow work items.
+
+ (*) Very slow work items.
+
+The former are expected to finish much quicker than the latter.
+
+An operation of the very slow class may do a batch combination of several
+lookups, mkdirs, and a create for instance.
+
+An operation of the ordinarily slow class may, for example, write stuff or
+expand files, provided the time taken to do so isn't too long.
+
+Operations of both types may sleep during execution, thus tying up the thread
+loaned to it.
+
+
+THREAD-TO-CLASS ALLOCATION
+--------------------------
+
+Not all the threads in the pool are available to work on very slow work items.
+The number will be between one and one fewer than the number of active threads.
+This is configurable (see the "Pool Configuration" section).
+
+All the threads are available to work on ordinarily slow work items, but a
+percentage of the threads will prefer to work on very slow work items.
+
+The configuration ensures that at least one thread will be available to work on
+very slow work items, and at least one thread will be available that won't work
+on very slow work items at all.
+
+
+=====================
+USING SLOW WORK ITEMS
+=====================
+
+Firstly, a module or subsystem wanting to make use of slow work items must
+register its interest:
+
+	 int ret = slow_work_register_user();
+
+This will return 0 if successful, or a -ve error upon failure.
+
+
+Slow work items may then be set up by:
+
+ (1) Declaring a slow_work struct type variable:
+
+	#include <linux/slow-work.h>
+
+	struct slow_work myitem;
+
+ (2) Declaring the operations to be used for this item:
+
+	struct slow_work_ops myitem_ops = {
+		.get_ref = myitem_get_ref,
+		.put_ref = myitem_put_ref,
+		.execute = myitem_execute,
+	};
+
+     [*] For a description of the ops, see section "Item Operations".
+
+ (3) Initialising the item:
+
+	slow_work_init(&myitem, &myitem_ops);
+
+     or:
+
+	vslow_work_init(&myitem, &myitem_ops);
+
+     depending on its class.
+
+A suitably set up work item can then be enqueued for processing:
+
+	int ret = slow_work_enqueue(&myitem);
+
+This will return a -ve error if the thread pool is unable to gain a reference
+on the item, 0 otherwise.
+
+
+The items are reference counted, so there ought to be no need for a flush
+operation.  When all a module's slow work items have been processed, and the
+module has no further interest in the facility, it should unregister its
+interest:
+
+	slow_work_unregister_user();
+
+
+===============
+ITEM OPERATIONS
+===============
+
+Each work item requires a table of operations of type struct slow_work_ops.
+All members are required:
+
+ (*) Get a reference on an item:
+
+	int (*get_ref)(struct slow_work *work);
+
+     This allows the thread pool to attempt to pin an item by getting a
+     reference on it.  This function should return 0 if the reference was
+     granted, or a -ve error otherwise.  If an error is returned,
+     slow_work_enqueue() will fail.
+
+     The reference is held whilst the item is queued and whilst it is being
+     executed.  The item may then be requeued with the same reference held, or
+     the reference will be released.
+
+ (*) Release a reference on an item:
+
+	void (*put_ref)(struct slow_work *work);
+
+     This allows the thread pool to unpin an item by releasing the reference on
+     it.  The thread pool will not touch the item again once this has been
+     called.
+
+ (*) Execute an item:
+
+	void (*execute)(struct slow_work *work);
+
+     This should perform the work required of the item.  It may sleep, it may
+     perform disk I/O and it may wait for locks.
+
+
+==================
+POOL CONFIGURATION
+==================
+
+The slow-work thread pool has a number of configurables:
+
+ (*) /proc/sys/kernel/slow-work/min-threads
+
+     The minimum number of threads that should be in the pool whilst it is in
+     use.  This may be anywhere between 2 and max-threads.
+
+ (*) /proc/sys/kernel/slow-work/max-threads
+
+     The maximum number of threads that should in the pool.  This may be
+     anywhere between min-threads and 255 or NR_CPUS * 2, whichever is greater.
+
+ (*) /proc/sys/kernel/slow-work/vslow-percentage
+
+     The percentage of active threads in the pool that may be used to execute
+     very slow work items.  This may be between 1 and 99.  The resultant number
+     is bounded to between 1 and one fewer than the number of active threads.
+     This ensures there is always at least one thread that can process very
+     slow work items, and always at least one thread that won't.
@@ -7,6 +7,8 @@
  * modify it under the terms of the GNU General Public Licence
  * as published by the Free Software Foundation; either version
  * 2 of the Licence, or (at your option) any later version.
+ *
+ * See Documentation/slow-work.txt
  */
  
 #ifndef _LINUX_SLOW_WORK_H
@@ -7,6 +7,8 @@
  * modify it under the terms of the GNU General Public Licence
  * as published by the Free Software Foundation; either version
  * 2 of the Licence, or (at your option) any later version.
+ *
+ * See Documentation/slow-work.txt
  */
  
 #include <linux/module.h>
	1	+ ====================================
	2	+ SLOW WORK ITEM EXECUTION THREAD POOL
	3	+ ====================================
	4	+
	5	+By: David Howells <dhowells@redhat.com>
	6	+
	7	+The slow work item execution thread pool is a pool of threads for performing
	8	+things that take a relatively long time, such as making mkdir calls.
	9	+Typically, when processing something, these items will spend a lot of time
	10	+blocking a thread on I/O, thus making that thread unavailable for doing other
	11	+work.
	12	+
	13	+The standard workqueue model is unsuitable for this class of work item as that
	14	+limits the owner to a single thread or a single thread per CPU. For some
	15	+tasks, however, more threads - or fewer - are required.
	16	+
	17	+There is just one pool per system. It contains no threads unless something
	18	+wants to use it - and that something must register its interest first. When
	19	+the pool is active, the number of threads it contains is dynamic, varying
	20	+between a maximum and minimum setting, depending on the load.
	21	+
	22	+
	23	+====================
	24	+CLASSES OF WORK ITEM
	25	+====================
	26	+
	27	+This pool support two classes of work items:
	28	+
	29	+ (*) Slow work items.
	30	+
	31	+ (*) Very slow work items.
	32	+
	33	+The former are expected to finish much quicker than the latter.
	34	+
	35	+An operation of the very slow class may do a batch combination of several
	36	+lookups, mkdirs, and a create for instance.
	37	+
	38	+An operation of the ordinarily slow class may, for example, write stuff or
	39	+expand files, provided the time taken to do so isn't too long.
	40	+
	41	+Operations of both types may sleep during execution, thus tying up the thread
	42	+loaned to it.
	43	+
	44	+
	45	+THREAD-TO-CLASS ALLOCATION
	46	+--------------------------
	47	+
	48	+Not all the threads in the pool are available to work on very slow work items.
	49	+The number will be between one and one fewer than the number of active threads.
	50	+This is configurable (see the "Pool Configuration" section).
	51	+
	52	+All the threads are available to work on ordinarily slow work items, but a
	53	+percentage of the threads will prefer to work on very slow work items.
	54	+
	55	+The configuration ensures that at least one thread will be available to work on
	56	+very slow work items, and at least one thread will be available that won't work
	57	+on very slow work items at all.
	58	+
	59	+
	60	+=====================
	61	+USING SLOW WORK ITEMS
	62	+=====================
	63	+
	64	+Firstly, a module or subsystem wanting to make use of slow work items must
	65	+register its interest:
	66	+
	67	+ int ret = slow_work_register_user();
	68	+
	69	+This will return 0 if successful, or a -ve error upon failure.
	70	+
	71	+
	72	+Slow work items may then be set up by:
	73	+
	74	+ (1) Declaring a slow_work struct type variable:
	75	+
	76	+ #include <linux/slow-work.h>
	77	+
	78	+ struct slow_work myitem;
	79	+
	80	+ (2) Declaring the operations to be used for this item:
	81	+
	82	+ struct slow_work_ops myitem_ops = {
	83	+ .get_ref = myitem_get_ref,
	84	+ .put_ref = myitem_put_ref,
	85	+ .execute = myitem_execute,
	86	+ };
	87	+
	88	+ [*] For a description of the ops, see section "Item Operations".
	89	+
	90	+ (3) Initialising the item:
	91	+
	92	+ slow_work_init(&myitem, &myitem_ops);
	93	+
	94	+ or:
	95	+
	96	+ vslow_work_init(&myitem, &myitem_ops);
	97	+
	98	+ depending on its class.
	99	+
	100	+A suitably set up work item can then be enqueued for processing:
	101	+
	102	+ int ret = slow_work_enqueue(&myitem);
	103	+
	104	+This will return a -ve error if the thread pool is unable to gain a reference
	105	+on the item, 0 otherwise.
	106	+
	107	+
	108	+The items are reference counted, so there ought to be no need for a flush
	109	+operation. When all a module's slow work items have been processed, and the
	110	+module has no further interest in the facility, it should unregister its
	111	+interest:
	112	+
	113	+ slow_work_unregister_user();
	114	+
	115	+
	116	+===============
	117	+ITEM OPERATIONS
	118	+===============
	119	+
	120	+Each work item requires a table of operations of type struct slow_work_ops.
	121	+All members are required:
	122	+
	123	+ (*) Get a reference on an item:
	124	+
	125	+ int (get_ref)(struct slow_work work);
	126	+
	127	+ This allows the thread pool to attempt to pin an item by getting a
	128	+ reference on it. This function should return 0 if the reference was
	129	+ granted, or a -ve error otherwise. If an error is returned,
	130	+ slow_work_enqueue() will fail.
	131	+
	132	+ The reference is held whilst the item is queued and whilst it is being
	133	+ executed. The item may then be requeued with the same reference held, or
	134	+ the reference will be released.
	135	+
	136	+ (*) Release a reference on an item:
	137	+
	138	+ void (put_ref)(struct slow_work work);
	139	+
	140	+ This allows the thread pool to unpin an item by releasing the reference on
	141	+ it. The thread pool will not touch the item again once this has been
	142	+ called.
	143	+
	144	+ (*) Execute an item:
	145	+
	146	+ void (execute)(struct slow_work work);
	147	+
	148	+ This should perform the work required of the item. It may sleep, it may
	149	+ perform disk I/O and it may wait for locks.
	150	+
	151	+
	152	+==================
	153	+POOL CONFIGURATION
	154	+==================
	155	+
	156	+The slow-work thread pool has a number of configurables:
	157	+
	158	+ (*) /proc/sys/kernel/slow-work/min-threads
	159	+
	160	+ The minimum number of threads that should be in the pool whilst it is in
	161	+ use. This may be anywhere between 2 and max-threads.
	162	+
	163	+ (*) /proc/sys/kernel/slow-work/max-threads
	164	+
	165	+ The maximum number of threads that should in the pool. This may be
	166	+ anywhere between min-threads and 255 or NR_CPUS * 2, whichever is greater.
	167	+
	168	+ (*) /proc/sys/kernel/slow-work/vslow-percentage
	169	+
	170	+ The percentage of active threads in the pool that may be used to execute
	171	+ very slow work items. This may be between 1 and 99. The resultant number
	172	+ is bounded to between 1 and one fewer than the number of active threads.
	173	+ This ensures there is always at least one thread that can process very
	174	+ slow work items, and always at least one thread that won't.
...	...	@@ -7,6 +7,8 @@
7	7	* modify it under the terms of the GNU General Public Licence
8	8	* as published by the Free Software Foundation; either version
9	9	* 2 of the Licence, or (at your option) any later version.
	10	+ *
	11	+ * See Documentation/slow-work.txt
10	12	*/
11	13
12	14	#ifndef _LINUX_SLOW_WORK_H