Doug / smarc-fsl-linux-kernel

Download as
Browse Code ยป

Commit f69b17d7e745d8edd7c0d90390cbaa77e63c5ea3

Authored by Lai Jiangshan
Committed by Ingo Molnar
1 parent dfbbe89e19

rcu: rcu_barrier VS cpu_hotplug: Ensure callbacks in dead cpu are migrated to online cpu 

cpu hotplug may happen asynchronously, some rcu callbacks are maybe
still on dead cpu, rcu_barrier() also needs to wait for these rcu
callbacks to complete, so we must ensure callbacks in dead cpu are
migrated to online cpu.

Paul E. McKenney's review:

  Good stuff, Lai!!!  Simpler than any of the approaches that I was
  considering, and, better yet, independent of the underlying RCU
  implementation!!!

  I was initially worried that wake_up() might wake only one of two
  possible wait_event()s, namely rcu_barrier() and the CPU_POST_DEAD code,
  but the fact that wait_event() clears WQ_FLAG_EXCLUSIVE avoids that issue.
  I was also worried about the fact that different RCU implementations have
  different mappings of call_rcu(), call_rcu_bh(), and call_rcu_sched(), but
  this is OK as well because we just get an extra (harmless) callback in the
  case that they map together (for example, Classic RCU has call_rcu_sched()
  mapping to call_rcu()).

  Overlap of CPU-hotplug operations is prevented by cpu_add_remove_lock,
  and any stray callbacks that arrive (for example, from irq handlers
  running on the dying CPU) either are ahead of the CPU_DYING callbacks on
  the one hand (and thus accounted for), or happened after the rcu_barrier()
  started on the other (and thus don't need to be accounted for).

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
LKML-Reference: <49C36476.1010400@cn.fujitsu.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>

Showing 1 changed file with 44 additions and 0 deletions Side-by-side Diff

... ... @@ -122,6 +122,8 @@
122 122 }
123 123 }
124 124  
  125 +static inline void wait_migrated_callbacks(void);
  126 +
125 127 /*
126 128 * Orchestrate the specified type of RCU barrier, waiting for all
127 129 * RCU callbacks of the specified type to complete.
... ... @@ -147,6 +149,7 @@
147 149 complete(&rcu_barrier_completion);
148 150 wait_for_completion(&rcu_barrier_completion);
149 151 mutex_unlock(&rcu_barrier_mutex);
  152 + wait_migrated_callbacks();
150 153 }
151 154  
152 155 /**
153 156  
... ... @@ -176,9 +179,50 @@
176 179 }
177 180 EXPORT_SYMBOL_GPL(rcu_barrier_sched);
178 181  
  182 +static atomic_t rcu_migrate_type_count = ATOMIC_INIT(0);
  183 +static struct rcu_head rcu_migrate_head[3];
  184 +static DECLARE_WAIT_QUEUE_HEAD(rcu_migrate_wq);
  185 +
  186 +static void rcu_migrate_callback(struct rcu_head *notused)
  187 +{
  188 + if (atomic_dec_and_test(&rcu_migrate_type_count))
  189 + wake_up(&rcu_migrate_wq);
  190 +}
  191 +
  192 +static inline void wait_migrated_callbacks(void)
  193 +{
  194 + wait_event(rcu_migrate_wq, !atomic_read(&rcu_migrate_type_count));
  195 +}
  196 +
  197 +static int __cpuinit rcu_barrier_cpu_hotplug(struct notifier_block *self,
  198 + unsigned long action, void *hcpu)
  199 +{
  200 + if (action == CPU_DYING) {
  201 + /*
  202 + * preempt_disable() in on_each_cpu() prevents stop_machine(),
  203 + * so when "on_each_cpu(rcu_barrier_func, (void *)type, 1);"
  204 + * returns, all online cpus have queued rcu_barrier_func(),
  205 + * and the dead cpu(if it exist) queues rcu_migrate_callback()s.
  206 + *
  207 + * These callbacks ensure _rcu_barrier() waits for all
  208 + * RCU callbacks of the specified type to complete.
  209 + */
  210 + atomic_set(&rcu_migrate_type_count, 3);
  211 + call_rcu_bh(rcu_migrate_head, rcu_migrate_callback);
  212 + call_rcu_sched(rcu_migrate_head + 1, rcu_migrate_callback);
  213 + call_rcu(rcu_migrate_head + 2, rcu_migrate_callback);
  214 + } else if (action == CPU_POST_DEAD) {
  215 + /* rcu_migrate_head is protected by cpu_add_remove_lock */
  216 + wait_migrated_callbacks();
  217 + }
  218 +
  219 + return NOTIFY_OK;
  220 +}
  221 +
179 222 void __init rcu_init(void)
180 223 {
181 224 __rcu_init();
  225 + hotcpu_notifier(rcu_barrier_cpu_hotplug, 0);
182 226 }
183 227  
184 228 void rcu_scheduler_starting(void)