Commit 1b96b1053c63fb751a58444f32bef28fb5ddf3e7
Committed by
Greg Kroah-Hartman
1 parent
6bf64643e2
Exists in
ti-linux-3.14.y
and in
2 other branches
percpu-refcount: fix usage of this_cpu_ops
commit 0c36b390a546055b6815d4b93a2c9fed4d980ffb upstream. The percpu-refcount infrastructure uses the underscore variants of this_cpu_ops in order to modify percpu reference counters. (e.g. __this_cpu_inc()). However the underscore variants do not atomically update the percpu variable, instead they may be implemented using read-modify-write semantics (more than one instruction). Therefore it is only safe to use the underscore variant if the context is always the same (process, softirq, or hardirq). Otherwise it is possible to lose updates. This problem is something that Sebastian has seen within the aio subsystem which uses percpu refcounters both in process and softirq context leading to reference counts that never dropped to zeroes; even though the number of "get" and "put" calls matched. Fix this by using the non-underscore this_cpu_ops variant which provides correct per cpu atomic semantics and fixes the corrupted reference counts. Cc: Kent Overstreet <kmo@daterainc.com> Reported-by: Sebastian Ott <sebott@linux.vnet.ibm.com> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Tejun Heo <tj@kernel.org> References: http://lkml.kernel.org/g/alpine.LFD.2.11.1406041540520.21183@denkbrett Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Showing 1 changed file with 3 additions and 3 deletions Side-by-side Diff
include/linux/percpu-refcount.h
... | ... | @@ -110,7 +110,7 @@ |
110 | 110 | pcpu_count = ACCESS_ONCE(ref->pcpu_count); |
111 | 111 | |
112 | 112 | if (likely(REF_STATUS(pcpu_count) == PCPU_REF_PTR)) |
113 | - __this_cpu_inc(*pcpu_count); | |
113 | + this_cpu_inc(*pcpu_count); | |
114 | 114 | else |
115 | 115 | atomic_inc(&ref->count); |
116 | 116 | |
... | ... | @@ -139,7 +139,7 @@ |
139 | 139 | pcpu_count = ACCESS_ONCE(ref->pcpu_count); |
140 | 140 | |
141 | 141 | if (likely(REF_STATUS(pcpu_count) == PCPU_REF_PTR)) { |
142 | - __this_cpu_inc(*pcpu_count); | |
142 | + this_cpu_inc(*pcpu_count); | |
143 | 143 | ret = true; |
144 | 144 | } |
145 | 145 | |
... | ... | @@ -164,7 +164,7 @@ |
164 | 164 | pcpu_count = ACCESS_ONCE(ref->pcpu_count); |
165 | 165 | |
166 | 166 | if (likely(REF_STATUS(pcpu_count) == PCPU_REF_PTR)) |
167 | - __this_cpu_dec(*pcpu_count); | |
167 | + this_cpu_dec(*pcpu_count); | |
168 | 168 | else if (unlikely(atomic_dec_and_test(&ref->count))) |
169 | 169 | ref->release(ref); |
170 | 170 |