Commit 4aab3b5b3ccf94fc907e66233e6ca4d8675759a6

Authored by Tejun Heo
1 parent cac7f24298

percpu-ref: fix DEAD flag contamination of percpu pointer

While decoupling ATOMIC and DEAD flags, f47ad4578461 ("percpu_ref:
decouple switching to percpu mode and reinit") updated
__ref_is_percpu() so that it only tests ATOMIC flag to determine
whether the ref is in percpu mode or not; however, while DEAD implies
ATOMIC, the two flags are set separately during percpu_ref_kill() and
if __ref_is_percpu() races percpu_ref_kill(), it may see DEAD w/o
ATOMIC.  Because __ref_is_percpu() returns @ref->percpu_count_ptr
value verbatim as the percpu pointer after testing ATOMIC, the pointer
may now be contaminated with the DEAD flag.

This can be fixed by clearing the flag bits before returning the
pointer which was the fix proposed by Shaohua; however, as DEAD
implies ATOMIC, we can just test for both flags at once and avoid the
explicit masking.

Update __ref_is_percpu() so that it tests that both ATOMIC and DEAD
are clear before returning @ref->percpu_count_ptr as the percpu
pointer.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-and-Reviewed-by: Shaohua Li <shli@kernel.org>
Link: http://lkml.kernel.org/r/995deb699f5b873c45d667df4add3b06f73c2c25.1416638887.git.shli@kernel.org
Fixes: f47ad4578461 ("percpu_ref: decouple switching to percpu mode and reinit")

Showing 1 changed file with 7 additions and 1 deletions Side-by-side Diff

include/linux/percpu-refcount.h
... ... @@ -133,7 +133,13 @@
133 133 /* paired with smp_store_release() in percpu_ref_reinit() */
134 134 smp_read_barrier_depends();
135 135  
136   - if (unlikely(percpu_ptr & __PERCPU_REF_ATOMIC))
  136 + /*
  137 + * Theoretically, the following could test just ATOMIC; however,
  138 + * then we'd have to mask off DEAD separately as DEAD may be
  139 + * visible without ATOMIC if we race with percpu_ref_kill(). DEAD
  140 + * implies ATOMIC anyway. Test them together.
  141 + */
  142 + if (unlikely(percpu_ptr & __PERCPU_REF_ATOMIC_DEAD))
137 143 return false;
138 144  
139 145 *percpu_countp = (unsigned long __percpu *)percpu_ptr;