[PATCH] round_jiffies infrastructure

Introduce a round_jiffies() function as well as a round_jiffies_relative() function. These functions round a jiffies value to the next whole second. The primary purpose of this rounding is to cause all "we don't care exactly when" timers to happen at the same jiffy. This avoids multiple timers firing within the second for no real reason; with dynamic ticks these extra timers cause wakeups from deep sleep CPU sleep states and thus waste power. The exact wakeup moment is skewed by the cpu number, to avoid all cpus from waking up at the exact same time (and hitting the same lock/cachelines there) [akpm@osdl.org: fix variable type] Signed-off-by: Arjan van de Ven <arjan@linux.intel.com> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Ingo Molnar <mingo@elte.hu> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>

[PATCH] round_jiffies infrastructure
Introduce a round_jiffies() function as well as a round_jiffies_relative() function. These functions round a jiffies value to the next whole second. The primary purpose of this rounding is to cause all "we don't care exactly when" timers to happen at the same jiffy. This avoids multiple timers firing within the second for no real reason; with dynamic ticks these extra timers cause wakeups from deep sleep CPU sleep states and thus waste power. The exact wakeup moment is skewed by the cpu number, to avoid all cpus from waking up at the exact same time (and hitting the same lock/cachelines there) [akpm@osdl.org: fix variable type] Signed-off-by: Arjan van de Ven <arjan@linux.intel.com> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Ingo Molnar <mingo@elte.hu> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Arjan van de Ven · Linus Torvalds
1 parent 5466b456ed
Showing 2 changed files with 138 additions and 0 deletions Side-by-side Diff
include/linux/timer.h
kernel/timer.c
@@ -98,5 +98,11 @@
 struct hrtimer;
 extern int it_real_fn(struct hrtimer *);
  
+unsigned long __round_jiffies(unsigned long j, int cpu);
+unsigned long __round_jiffies_relative(unsigned long j, int cpu);
+unsigned long round_jiffies(unsigned long j);
+unsigned long round_jiffies_relative(unsigned long j);
+
+
 #endif
@@ -80,6 +80,138 @@
 EXPORT_SYMBOL(boot_tvec_bases);
 static DEFINE_PER_CPU(tvec_base_t *, tvec_bases) = &boot_tvec_bases;
  
+/**
+ * __round_jiffies - function to round jiffies to a full second
+ * @j: the time in (absolute) jiffies that should be rounded
+ * @cpu: the processor number on which the timeout will happen
+ *
+ * __round_jiffies rounds an absolute time in the future (in jiffies)
+ * up or down to (approximately) full seconds. This is useful for timers
+ * for which the exact time they fire does not matter too much, as long as
+ * they fire approximately every X seconds.
+ *
+ * By rounding these timers to whole seconds, all such timers will fire
+ * at the same time, rather than at various times spread out. The goal
+ * of this is to have the CPU wake up less, which saves power.
+ *
+ * The exact rounding is skewed for each processor to avoid all
+ * processors firing at the exact same time, which could lead
+ * to lock contention or spurious cache line bouncing.
+ *
+ * The return value is the rounded version of the "j" parameter.
+ */
+unsigned long __round_jiffies(unsigned long j, int cpu)
+{
+	int rem;
+	unsigned long original = j;
+
+	/*
+	 * We don't want all cpus firing their timers at once hitting the
+	 * same lock or cachelines, so we skew each extra cpu with an extra
+	 * 3 jiffies. This 3 jiffies came originally from the mm/ code which
+	 * already did this.
+	 * The skew is done by adding 3*cpunr, then round, then subtract this
+	 * extra offset again.
+	 */
+	j += cpu * 3;
+
+	rem = j % HZ;
+
+	/*
+	 * If the target jiffie is just after a whole second (which can happen
+	 * due to delays of the timer irq, long irq off times etc etc) then
+	 * we should round down to the whole second, not up. Use 1/4th second
+	 * as cutoff for this rounding as an extreme upper bound for this.
+	 */
+	if (rem < HZ/4) /* round down */
+		j = j - rem;
+	else /* round up */
+		j = j - rem + HZ;
+
+	/* now that we have rounded, subtract the extra skew again */
+	j -= cpu * 3;
+
+	if (j <= jiffies) /* rounding ate our timeout entirely; */
+		return original;
+	return j;
+}
+EXPORT_SYMBOL_GPL(__round_jiffies);
+
+/**
+ * __round_jiffies_relative - function to round jiffies to a full second
+ * @j: the time in (relative) jiffies that should be rounded
+ * @cpu: the processor number on which the timeout will happen
+ *
+ * __round_jiffies_relative rounds a time delta  in the future (in jiffies)
+ * up or down to (approximately) full seconds. This is useful for timers
+ * for which the exact time they fire does not matter too much, as long as
+ * they fire approximately every X seconds.
+ *
+ * By rounding these timers to whole seconds, all such timers will fire
+ * at the same time, rather than at various times spread out. The goal
+ * of this is to have the CPU wake up less, which saves power.
+ *
+ * The exact rounding is skewed for each processor to avoid all
+ * processors firing at the exact same time, which could lead
+ * to lock contention or spurious cache line bouncing.
+ *
+ * The return value is the rounded version of the "j" parameter.
+ */
+unsigned long __round_jiffies_relative(unsigned long j, int cpu)
+{
+	/*
+	 * In theory the following code can skip a jiffy in case jiffies
+	 * increments right between the addition and the later subtraction.
+	 * However since the entire point of this function is to use approximate
+	 * timeouts, it's entirely ok to not handle that.
+	 */
+	return  __round_jiffies(j + jiffies, cpu) - jiffies;
+}
+EXPORT_SYMBOL_GPL(__round_jiffies_relative);
+
+/**
+ * round_jiffies - function to round jiffies to a full second
+ * @j: the time in (absolute) jiffies that should be rounded
+ *
+ * round_jiffies rounds an absolute time in the future (in jiffies)
+ * up or down to (approximately) full seconds. This is useful for timers
+ * for which the exact time they fire does not matter too much, as long as
+ * they fire approximately every X seconds.
+ *
+ * By rounding these timers to whole seconds, all such timers will fire
+ * at the same time, rather than at various times spread out. The goal
+ * of this is to have the CPU wake up less, which saves power.
+ *
+ * The return value is the rounded version of the "j" parameter.
+ */
+unsigned long round_jiffies(unsigned long j)
+{
+	return __round_jiffies(j, raw_smp_processor_id());
+}
+EXPORT_SYMBOL_GPL(round_jiffies);
+
+/**
+ * round_jiffies_relative - function to round jiffies to a full second
+ * @j: the time in (relative) jiffies that should be rounded
+ *
+ * round_jiffies_relative rounds a time delta  in the future (in jiffies)
+ * up or down to (approximately) full seconds. This is useful for timers
+ * for which the exact time they fire does not matter too much, as long as
+ * they fire approximately every X seconds.
+ *
+ * By rounding these timers to whole seconds, all such timers will fire
+ * at the same time, rather than at various times spread out. The goal
+ * of this is to have the CPU wake up less, which saves power.
+ *
+ * The return value is the rounded version of the "j" parameter.
+ */
+unsigned long round_jiffies_relative(unsigned long j)
+{
+	return __round_jiffies_relative(j, raw_smp_processor_id());
+}
+EXPORT_SYMBOL_GPL(round_jiffies_relative);
+
+
 static inline void set_running_timer(tvec_base_t *base,
 					struct timer_list *timer)
 {
...	...	@@ -98,5 +98,11 @@
98	98	struct hrtimer;
99	99	extern int it_real_fn(struct hrtimer *);
100	100
	101	+unsigned long __round_jiffies(unsigned long j, int cpu);
	102	+unsigned long __round_jiffies_relative(unsigned long j, int cpu);
	103	+unsigned long round_jiffies(unsigned long j);
	104	+unsigned long round_jiffies_relative(unsigned long j);
	105	+
	106	+
101	107	#endif
...	...	@@ -80,6 +80,138 @@
80	80	EXPORT_SYMBOL(boot_tvec_bases);
81	81	static DEFINE_PER_CPU(tvec_base_t *, tvec_bases) = &boot_tvec_bases;
82	82
	83	+/**
	84	+ * __round_jiffies - function to round jiffies to a full second
	85	+ * @j: the time in (absolute) jiffies that should be rounded
	86	+ * @cpu: the processor number on which the timeout will happen
	87	+ *
	88	+ * __round_jiffies rounds an absolute time in the future (in jiffies)
	89	+ * up or down to (approximately) full seconds. This is useful for timers
	90	+ * for which the exact time they fire does not matter too much, as long as
	91	+ * they fire approximately every X seconds.
	92	+ *
	93	+ * By rounding these timers to whole seconds, all such timers will fire
	94	+ * at the same time, rather than at various times spread out. The goal
	95	+ * of this is to have the CPU wake up less, which saves power.
	96	+ *
	97	+ * The exact rounding is skewed for each processor to avoid all
	98	+ * processors firing at the exact same time, which could lead
	99	+ * to lock contention or spurious cache line bouncing.
	100	+ *
	101	+ * The return value is the rounded version of the "j" parameter.
	102	+ */
	103	+unsigned long __round_jiffies(unsigned long j, int cpu)
	104	+{
	105	+ int rem;
	106	+ unsigned long original = j;
	107	+
	108	+ /*
	109	+ * We don't want all cpus firing their timers at once hitting the
	110	+ * same lock or cachelines, so we skew each extra cpu with an extra
	111	+ * 3 jiffies. This 3 jiffies came originally from the mm/ code which
	112	+ * already did this.
	113	+ * The skew is done by adding 3*cpunr, then round, then subtract this
	114	+ * extra offset again.
	115	+ */
	116	+ j += cpu * 3;
	117	+
	118	+ rem = j % HZ;
	119	+
	120	+ /*
	121	+ * If the target jiffie is just after a whole second (which can happen
	122	+ * due to delays of the timer irq, long irq off times etc etc) then
	123	+ * we should round down to the whole second, not up. Use 1/4th second
	124	+ * as cutoff for this rounding as an extreme upper bound for this.
	125	+ */
	126	+ if (rem < HZ/4) /* round down */
	127	+ j = j - rem;
	128	+ else /* round up */
	129	+ j = j - rem + HZ;
	130	+
	131	+ /* now that we have rounded, subtract the extra skew again */
	132	+ j -= cpu * 3;
	133	+
	134	+ if (j <= jiffies) /* rounding ate our timeout entirely; */
	135	+ return original;
	136	+ return j;
	137	+}
	138	+EXPORT_SYMBOL_GPL(__round_jiffies);
	139	+
	140	+/**
	141	+ * __round_jiffies_relative - function to round jiffies to a full second
	142	+ * @j: the time in (relative) jiffies that should be rounded
	143	+ * @cpu: the processor number on which the timeout will happen
	144	+ *
	145	+ * __round_jiffies_relative rounds a time delta in the future (in jiffies)
	146	+ * up or down to (approximately) full seconds. This is useful for timers
	147	+ * for which the exact time they fire does not matter too much, as long as
	148	+ * they fire approximately every X seconds.
	149	+ *
	150	+ * By rounding these timers to whole seconds, all such timers will fire
	151	+ * at the same time, rather than at various times spread out. The goal
	152	+ * of this is to have the CPU wake up less, which saves power.
	153	+ *
	154	+ * The exact rounding is skewed for each processor to avoid all
	155	+ * processors firing at the exact same time, which could lead
	156	+ * to lock contention or spurious cache line bouncing.
	157	+ *
	158	+ * The return value is the rounded version of the "j" parameter.
	159	+ */
	160	+unsigned long __round_jiffies_relative(unsigned long j, int cpu)
	161	+{
	162	+ /*
	163	+ * In theory the following code can skip a jiffy in case jiffies
	164	+ * increments right between the addition and the later subtraction.
	165	+ * However since the entire point of this function is to use approximate
	166	+ * timeouts, it's entirely ok to not handle that.
	167	+ */
	168	+ return __round_jiffies(j + jiffies, cpu) - jiffies;
	169	+}
	170	+EXPORT_SYMBOL_GPL(__round_jiffies_relative);
	171	+
	172	+/**
	173	+ * round_jiffies - function to round jiffies to a full second
	174	+ * @j: the time in (absolute) jiffies that should be rounded
	175	+ *
	176	+ * round_jiffies rounds an absolute time in the future (in jiffies)
	177	+ * up or down to (approximately) full seconds. This is useful for timers
	178	+ * for which the exact time they fire does not matter too much, as long as
	179	+ * they fire approximately every X seconds.
	180	+ *
	181	+ * By rounding these timers to whole seconds, all such timers will fire
	182	+ * at the same time, rather than at various times spread out. The goal
	183	+ * of this is to have the CPU wake up less, which saves power.
	184	+ *
	185	+ * The return value is the rounded version of the "j" parameter.
	186	+ */
	187	+unsigned long round_jiffies(unsigned long j)
	188	+{
	189	+ return __round_jiffies(j, raw_smp_processor_id());
	190	+}
	191	+EXPORT_SYMBOL_GPL(round_jiffies);
	192	+
	193	+/**
	194	+ * round_jiffies_relative - function to round jiffies to a full second
	195	+ * @j: the time in (relative) jiffies that should be rounded
	196	+ *
	197	+ * round_jiffies_relative rounds a time delta in the future (in jiffies)
	198	+ * up or down to (approximately) full seconds. This is useful for timers
	199	+ * for which the exact time they fire does not matter too much, as long as
	200	+ * they fire approximately every X seconds.
	201	+ *
	202	+ * By rounding these timers to whole seconds, all such timers will fire
	203	+ * at the same time, rather than at various times spread out. The goal
	204	+ * of this is to have the CPU wake up less, which saves power.
	205	+ *
	206	+ * The return value is the rounded version of the "j" parameter.
	207	+ */
	208	+unsigned long round_jiffies_relative(unsigned long j)
	209	+{
	210	+ return __round_jiffies_relative(j, raw_smp_processor_id());
	211	+}
	212	+EXPORT_SYMBOL_GPL(round_jiffies_relative);
	213	+
	214	+
83	215	static inline void set_running_timer(tvec_base_t *base,
84	216	struct timer_list *timer)
85	217	{