22 Nov, 2014
1 commit
-
To avoid include hell, the per_cpu variable printk_func was declared
in percpu.h. But it is only defined if printk is defined.As users of printk may also use the printk_func variable, it needs to
be defined even if CONFIG_PRINTK is not.Also add a printk.h include in percpu.h just to be safe.
Link: http://lkml.kernel.org/r/20141121183215.01ba539c@canb.auug.org.au
Reported-by: Stephen Rothwell
Signed-off-by: Steven Rostedt
20 Nov, 2014
1 commit
-
Being able to divert printk to call another function besides the normal
logging is useful for such things like NMI handling. If some functions
are to be called from NMI that does printk() it is possible to lock up
the box if the nmi handler triggers when another printk is happening.One example of this use is to perform a stack trace on all CPUs via NMI.
But if the NMI is to do the printk() it can cause the system to lock up.
By allowing the printk to be diverted to another function that can safely
record the printk output and then print it when it in a safe context
then NMIs will be safe to call these functions like show_regs().Link: http://lkml.kernel.org/p/20140619213952.209176403@goodmis.org
Tested-by: Jiri Kosina
Acked-by: Jiri Kosina
Acked-by: Paul E. McKenney
Reviewed-by: Petr Mladek
Signed-off-by: Steven Rostedt
03 Sep, 2014
2 commits
-
The percpu allocator now supports atomic allocations by only
allocating from already populated areas but the mechanism to ensure
that there's adequate amount of populated areas was missing.This patch expands pcpu_balance_work so that in addition to freeing
excess free chunks it also populates chunks to maintain an adequate
level of populated areas. pcpu_alloc() schedules pcpu_balance_work if
the amount of free populated areas is too low or after an atomic
allocation failure.* PERPCU_DYNAMIC_RESERVE is increased by two pages to account for
PCPU_EMPTY_POP_PAGES_LOW.* pcpu_async_enabled is added to gate both async jobs -
chunk->map_extend_work and pcpu_balance_work - so that we don't end
up scheduling them while the needed subsystems aren't up yet.Signed-off-by: Tejun Heo
-
Now that pcpu_alloc_area() can allocate only from populated areas,
it's easy to add atomic allocation support to [__]alloc_percpu().
Update pcpu_alloc() so that it accepts @gfp and skips all the blocking
operations and allocates only from the populated areas if @gfp doesn't
contain GFP_KERNEL. New interface functions [__]alloc_percpu_gfp()
are added.While this means that atomic allocations are possible, this isn't
complete yet as there's no mechanism to ensure that certain amount of
populated areas is kept available and atomic allocations may keep
failing under certain conditions.Signed-off-by: Tejun Heo
18 Jun, 2014
4 commits
-
We're in the process of moving all percpu accessors and operations to
include/linux/percpu-defs.h so that they're available to arch headers
without having to include full include/linux/percpu.h which may cause
cyclic inclusion dependency.This patch moves {raw|this}_cpu_*() definitions from
include/linux/percpu.h to include/linux/percpu-defs.h. The code is
moved mostly verbatim; however, raw_cpu_*() are placed above
this_cpu_*() which is more conventional as the raw operations may be
used to defined other variants.This is pure reorganization.
Signed-off-by: Tejun Heo
Acked-by: Christoph Lameter -
{raw|this}_cpu_*_N() operations are expected to be provided by archs
and the generic definitions are provided as fallbacks. As such, these
firmly belong to include/asm-generic/percpu.h.Move the generic definitions to include/asm-generic/percpu.h. The
code is moved mostly verbatim; however, raw_cpu_*_N() are placed above
this_cpu_*_N() which is more conventional as the raw operations may be
used to defined other variants.This is pure reorganization.
Signed-off-by: Tejun Heo
Acked-by: Christoph Lameter -
Currently, percpu allows two separate methods for overriding
{raw|this}_cpu_*() ops - for a given operation, an arch can provide
whole replacement or sized sub operations to override specific parts
of it. e.g. arch either can provide this_cpu_add() or
this_cpu_add_4() to override only the 4 byte operation.While quite flexible on a glance, the dual-overriding scheme
complicates the code path for no actual gain. It compilcates the
already complex operation definitions and if an arch wants to override
all sizes, it can easily provide all variants anyway. In fact, no
arch is actually making use of whole operation override.Another oddity is that __this_cpu_*() operations are defined in the
same way as raw_cpu_*() but ignores full overrides of the raw_cpu_*()
and doesn't allow full operation override, so if an arch provides
whole overrides for raw_cpu_*() operations __this_cpu_*() ends up
using the generic implementations.More importantly, it takes away the layering between arch-specific and
generic parts making it impossible for the generic part to implement
arch-independent features on top of arch-specific overrides.This patch removes the support for whole operation overrides. As no
arch is using it, this doesn't cause any actual difference.Signed-off-by: Tejun Heo
Acked-by: Christoph Lameter -
include/linux/percpu-defs.h is gonna host all accessors and operations
so that arch headers can make use of them too without worrying about
circular dependency through include/linux/percpu.h.This patch moves the following accessors from include/linux/percpu.h
to include/linux/percpu-defs.h.* get/put_cpu_var()
* get/put_cpu_ptr()
* per_cpu_ptr()This is pure reorgniazation.
Signed-off-by: Tejun Heo
Acked-by: Christoph Lameter
10 Jun, 2014
1 commit
-
Pull percpu updates from Tejun Heo:
"Nothing too exciting. percpu_ref is going through some interface
changes and getting new features with more changes in the pipeline but
given its young age and few users, it's very low impact"* 'for-3.16' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu:
percpu-refcount: implement percpu_ref_tryget()
percpu-refcount: rename percpu_ref_tryget() to percpu_ref_tryget_live()
percpu: Replace __get_cpu_var with this_cpu_ptr
15 May, 2014
1 commit
-
The definition for raw_cpu_add_return() uses the operation prefix
"raw_add_return_", but the definitions in the various percpu.h files
expect "raw_cpu_add_return_". This commit therefore appropriately
adjusts the definition of raw_cpu_add_return().Signed-off-by: Paul E. McKenney
Acked-by: Christoph Lameter
Reviewed-by: Josh Triplett
16 Apr, 2014
1 commit
-
__this_cpu_ptr is being phased out. Use raw_cpu_ptr instead which was
introduced in 3.15-rc1. One case of using __get_cpu_var in the
get_cpu_var macro for address calculation was remaining in
include/linux/percpu.h.tj: Updated patch description.
Signed-off-by: Christoph Lameter
Signed-off-by: Tejun Heo
08 Apr, 2014
2 commits
-
We define a check function in order to avoid trouble with the include
files. Then the higher level __this_cpu macros are modified to invoke
the preemption check.[akpm@linux-foundation.org: coding-style fixes]
Signed-off-by: Christoph Lameter
Acked-by: Ingo Molnar
Cc: Tejun Heo
Tested-by: Grygorii Strashko
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
The kernel has never been audited to ensure that this_cpu operations are
consistently used throughout the kernel. The code generated in many
places can be improved through the use of this_cpu operations (which
uses a segment register for relocation of per cpu offsets instead of
performing address calculations).The patch set also addresses various consistency issues in general with
the per cpu macros.A. The semantics of __this_cpu_ptr() differs from this_cpu_ptr only
because checks are skipped. This is typically shown through a raw_
prefix. So this patch set changes the places where __this_cpu_ptr()
is used to raw_cpu_ptr().B. There has been the long term wish by some that __this_cpu operations
would check for preemption. However, there are cases where preemption
checks need to be skipped. This patch set adds raw_cpu operations that
do not check for preemption and then adds preemption checks to the
__this_cpu operations.C. The use of __get_cpu_var is always a reference to a percpu variable
that can also be handled via a this_cpu operation. This patch set
replaces all uses of __get_cpu_var with this_cpu operations.D. We can then use this_cpu RMW operations in various places replacing
sequences of instructions by a single one.E. The use of this_cpu operations throughout will allow other arches than
x86 to implement optimized references and RMV operations to work with
per cpu local data.F. The use of this_cpu operations opens up the possibility to
further optimize code that relies on synchronization through
per cpu data.The patch set works in a couple of stages:
I. Patch 1 adds the additional raw_cpu operations and raw_cpu_ptr().
Also converts the existing __this_cpu_xx_# primitive in the x86
code to raw_cpu_xx_#.II. Patch 2-4 use the raw_cpu operations in places that would give
us false positives once they are enabled.III. Patch 5 adds preemption checks to __this_cpu operations to allow
checking if preemption is properly disabled when these functions
are used.IV. Patches 6-20 are patches that simply replace uses of __get_cpu_var
with this_cpu_ptr. They do not depend on any changes to the percpu
code. No preemption tests are skipped if they are applied.V. Patches 21-46 are conversion patches that use this_cpu operations
in various kernel subsystems/drivers or arch code.VI. Patches 47/48 (not included in this series) remove no longer used
functions (__this_cpu_ptr and __get_cpu_var). These should only be
applied after all the conversion patches have made it and after we
have done additional passes through the kernel to ensure that none of
the uses of these functions remain.This patch (of 46):
The patches following this one will add preemption checks to __this_cpu
ops so we need to have an alternative way to use this_cpu operations
without preemption checks.raw_cpu_ops will be the basis for all other ops since these will be the
operations that do not implement any checks.Primitive operations are renamed by this patch from __this_cpu_xxx to
raw_cpu_xxxx.Also change the uses of the x86 percpu primitives in preempt.h.
These depend directly on asm/percpu.h (header #include nesting issue).Signed-off-by: Peter Zijlstra
Signed-off-by: Christoph Lameter
Acked-by: Ingo Molnar
Cc: Tejun Heo
Cc: "James E.J. Bottomley"
Cc: "Paul E. McKenney"
Cc: Alex Shi
Cc: Arnd Bergmann
Cc: Benjamin Herrenschmidt
Cc: Bryan Wu
Cc: Catalin Marinas
Cc: Chris Metcalf
Cc: Daniel Lezcano
Cc: David Daney
Cc: David Miller
Cc: David S. Miller
Cc: Dimitri Sivanich
Cc: Dipankar Sarma
Cc: Eric Dumazet
Cc: Fenghua Yu
Cc: Frederic Weisbecker
Cc: Greg Kroah-Hartman
Cc: H. Peter Anvin
Cc: Haavard Skinnemoen
Cc: Hans-Christian Egtvedt
Cc: Hedi Berriche
Cc: Heiko Carstens
Cc: Helge Deller
Cc: Ivan Kokshaysky
Cc: James Hogan
Cc: Jens Axboe
Cc: John Stultz
Cc: Martin Schwidefsky
Cc: Masami Hiramatsu
Cc: Matt Turner
Cc: Mike Frysinger
Cc: Mike Travis
Cc: Neil Brown
Cc: Nicolas Pitre
Cc: Paul Mackerras
Cc: Paul Mundt
Cc: Rafael J. Wysocki
Cc: Ralf Baechle
Cc: Richard Henderson
Cc: Robert Richter
Cc: Russell King
Cc: Russell King
Cc: Rusty Russell
Cc: Steven Rostedt
Cc: Thomas Gleixner
Cc: Tony Luck
Cc: Will Deacon
Cc: Wim Van Sebroeck
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
24 Jan, 2014
1 commit
-
Most of the VM_BUG_ON assertions are performed on a page. Usually, when
one of these assertions fails we'll get a BUG_ON with a call stack and
the registers.I've recently noticed based on the requests to add a small piece of code
that dumps the page to various VM_BUG_ON sites that the page dump is
quite useful to people debugging issues in mm.This patch adds a VM_BUG_ON_PAGE(cond, page) which beyond doing what
VM_BUG_ON() does, also dumps the page before executing the actual
BUG_ON.[akpm@linux-foundation.org: fix up includes]
Signed-off-by: Sasha Levin
Cc: "Kirill A. Shutemov"
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
13 Nov, 2013
1 commit
-
Pull percpu changes from Tejun Heo:
"Two smallish changes for percpu. Two patches to remove unused
this_cpu_xor() and one to fix a bug in percpu init failure path so
that it can reach the proper BUG() instead of oopsing earlier"* 'for-3.13' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu:
x86: remove this_cpu_xor() implementation
percpu: remove this_cpu_xor() implementation
percpu: fix bootmem error handling in pcpu_page_first_chunk()
31 Oct, 2013
1 commit
-
this_cpu_sub() is implemented as negation and addition.
This patch casts the adjustment to the counter type before negation to
sign extend the adjustment. This helps in cases where the counter type
is wider than an unsigned adjustment. An alternative to this patch is
to declare such operations unsupported, but it seemed useful to avoid
surprises.This patch specifically helps the following example:
unsigned int delta = 1
preempt_disable()
this_cpu_write(long_counter, 0)
this_cpu_sub(long_counter, delta)
preempt_enable()Before this change long_counter on a 64 bit machine ends with value
0xffffffff, rather than 0xffffffffffffffff. This is because
this_cpu_sub(pcp, delta) boils down to this_cpu_add(pcp, -delta),
which is basically:
long_counter = 0 + 0xffffffffAlso apply the same cast to:
__this_cpu_sub()
__this_cpu_sub_return()
this_cpu_sub_return()All percpu_test.ko passes, especially the following cases which
previously failed:l -= ui_one;
__this_cpu_sub(long_counter, ui_one);
CHECK(l, long_counter, -1);l -= ui_one;
this_cpu_sub(long_counter, ui_one);
CHECK(l, long_counter, -1);
CHECK(l, long_counter, 0xffffffffffffffff);ul -= ui_one;
__this_cpu_sub(ulong_counter, ui_one);
CHECK(ul, ulong_counter, -1);
CHECK(ul, ulong_counter, 0xffffffffffffffff);ul = this_cpu_sub_return(ulong_counter, ui_one);
CHECK(ul, ulong_counter, 2);ul = __this_cpu_sub_return(ulong_counter, ui_one);
CHECK(ul, ulong_counter, 1);Signed-off-by: Greg Thelen
Acked-by: Tejun Heo
Acked-by: Johannes Weiner
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
27 Oct, 2013
1 commit
-
There is not a single user in the whole kernel.
Besides that this_cpu_xor() is broken anyway since it gets
translated to this_cpu_or() (see __pcpu_size_call() line).So instead of fixing an unused definition just remove it.
Signed-off-by: Heiko Carstens
Acked-by: Ingo Molnar
Signed-off-by: Tejun Heo
06 Oct, 2012
1 commit
-
Signed-off-by: Andi Kleen
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
15 May, 2012
1 commit
-
Remove percpu_xxx serial functions, all of them were replaced by
this_cpu_xxx or __this_cpu_xxx serial functionsSigned-off-by: Alex Shi
Acked-by: Christoph Lameter
Acked-by: Tejun Heo
Acked-by: "H. Peter Anvin"
Cc: Ingo Molnar
Cc: Thomas Gleixner
Signed-off-by: Andrew Morton
Signed-off-by: Tejun Heo
05 Mar, 2012
1 commit
-
This patch adds missed "__" prefixes, otherwise these functions
works as irq/preemption safe.Reported-by: Torsten Kaiser
Signed-off-by: Konstantin Khlebnikov
Signed-off-by: Tejun Heo
22 Feb, 2012
2 commits
-
It doesn't make sense to trace irq off or do irq flags
lock proving inside 'this_cpu' operations, so replace local_irq_*
with raw_local_irq_* in 'this_cpu' op.Also the patch fixes onelockdep warning[1] by the replacement, see
below:In commit: 933393f58fef9963eac61db8093689544e29a600(percpu:
Remove irqsafe_cpu_xxx variants), local_irq_save/restore(flags) are
added inside this_cpu_inc operation, so that trace_hardirqs_off_caller
will be called by trace_hardirqs_on_caller directly because
__debug_atomic_inc is implemented as this_cpu_inc, which may trigger
the lockdep warning[1], for example in the below ARM scenary:kernel_thread_helper /*irq disabled*/
->trace_hardirqs_on_caller /*hardirqs_enabled was set*/
->trace_hardirqs_off_caller /*hardirqs_enabled cleared*/
__this_cpu_add(redundant_hardirqs_on)
->trace_hardirqs_off_caller /*irq disabled, so call here*/The 'unannotated irqs-on' warning will be triggered somewhere because
irq is just enabled after the irq trace in kernel_thread_helper.[1],
[ 0.162841] ------------[ cut here ]------------
[ 0.167694] WARNING: at kernel/lockdep.c:3493 check_flags+0xc0/0x1d0()
[ 0.174468] Modules linked in:
[ 0.177703] Backtrace:
[ 0.180328] [] (dump_backtrace+0x0/0x110) from [] (dump_stack+0x18/0x1c)
[ 0.189086] r6:c051f778 r5:00000da5 r4:00000000 r3:60000093
[ 0.195007] [] (dump_stack+0x0/0x1c) from [] (warn_slowpath_common+0x54/0x6c)
[ 0.204223] [] (warn_slowpath_common+0x0/0x6c) from [] (warn_slowpath_null+0x24/0x2c)
[ 0.214111] r8:00000000 r7:00000000 r6:ee069598 r5:60000013 r4:ee082000
[ 0.220825] r3:00000009
[ 0.223693] [] (warn_slowpath_null+0x0/0x2c) from [] (check_flags+0xc0/0x1d0)
[ 0.232910] [] (check_flags+0x0/0x1d0) from [] (lock_acquire+0x4c/0x11c)
[ 0.241668] [] (lock_acquire+0x0/0x11c) from [] (_raw_spin_lock+0x3c/0x74)
[ 0.250610] [] (_raw_spin_lock+0x0/0x74) from [] (set_task_comm+0x20/0xc0)
[ 0.259521] r6:ee069588 r5:ee0691c0 r4:ee082000
[ 0.264404] [] (set_task_comm+0x0/0xc0) from [] (kthreadd+0x28/0x108)
[ 0.272857] r8:00000000 r7:00000013 r6:c0044a08 r5:ee0691c0 r4:ee082000
[ 0.279571] r3:ee083fe0
[ 0.282470] [] (kthreadd+0x0/0x108) from [] (do_exit+0x0/0x6dc)
[ 0.290405] r5:c0060758 r4:00000000
[ 0.294189] ---[ end trace 1b75b31a2719ed1c ]---
[ 0.299041] possible reason: unannotated irqs-on.
[ 0.303955] irq event stamp: 5
[ 0.307159] hardirqs last enabled at (4): [] no_work_pending+0x8/0x2c
[ 0.314880] hardirqs last disabled at (5): [] trace_hardirqs_on_caller+0x60/0x26c
[ 0.323547] softirqs last enabled at (0): [] copy_process+0x33c/0xef4
[ 0.331207] softirqs last disabled at (0): [< (null)>] (null)
[ 0.337585] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000Acked-by: Christoph Lameter
Signed-off-by: Ming Lei
Signed-off-by: Tejun Heo -
This patch adds missed "__" into function prefix.
Otherwise on all archectures (except x86) it expands to irq/preemtion-safe
variant: _this_cpu_generic_add_return(), which do extra irq-save/irq-restore.
Optimal generic implementation is __this_cpu_generic_add_return().Signed-off-by: Konstantin Khlebnikov
Acked-by: Christoph Lameter
Signed-off-by: Tejun Heo
23 Dec, 2011
1 commit
-
We simply say that regular this_cpu use must be safe regardless of
preemption and interrupt state. That has no material change for x86
and s390 implementations of this_cpu operations. However, arches that
do not provide their own implementation for this_cpu operations will
now get code generated that disables interrupts instead of preemption.-tj: This is part of on-going percpu API cleanup. For detailed
discussion of the subject, please refer to the following thread.http://thread.gmane.org/gmane.linux.kernel/1222078
Signed-off-by: Christoph Lameter
Signed-off-by: Tejun Heo
LKML-Reference:
04 Jun, 2011
1 commit
-
On an architecture without CMPXCHG_LOCAL but with DEBUG_VM enabled,
the VM_BUG_ON() in __pcpu_double_call_return_bool() will cause an early
panic during boot unless we always align cpu_slab properly.In principle we could remove the alignment-testing VM_BUG_ON() for
architectures that don't have CMPXCHG_LOCAL, but leaving it in means
that new code will tend not to break x86 even if it is introduced
on another platform, and it's low cost to require alignment.Acked-by: David Rientjes
Acked-by: Christoph Lameter
Signed-off-by: Chris Metcalf
Signed-off-by: Pekka Enberg
05 May, 2011
1 commit
-
The SLUB allocator use of the cmpxchg_double logic was wrong: it
actually needs the irq-safe one.That happens automatically when we use the native unlocked 'cmpxchg8b'
instruction, but when compiling the kernel for older x86 CPUs that do
not support that instruction, we fall back to the generic emulation
code.And if you don't specify that you want the irq-safe version, the generic
code ends up just open-coding the cmpxchg8b equivalent without any
protection against interrupts or preemption. Which definitely doesn't
work for SLUB.This was reported by Werner Landgraf , who saw
instability with his distro-kernel that was compiled to support pretty
much everything under the sun. Most big Linux distributions tend to
compile for PPro and later, and would never have noticed this problem.This also fixes the prototypes for the irqsafe cmpxchg_double functions
to use 'bool' like they should.[ Btw, that whole "generic code defaults to no protection" design just
sounds stupid - if the code needs no protection, there is no reason to
use "cmpxchg_double" to begin with. So we should probably just remove
the unprotected version entirely as pointless. - Linus ]Signed-off-by: Thomas Gleixner
Reported-and-tested-by: werner
Acked-and-tested-by: Ingo Molnar
Acked-by: Christoph Lameter
Cc: Pekka Enberg
Cc: Jens Axboe
Cc: Tejun Heo
Link: http://lkml.kernel.org/r/alpine.LFD.2.02.1105041539050.3005@ionos
Signed-off-by: Ingo Molnar
Signed-off-by: Linus Torvalds
28 Feb, 2011
1 commit
-
Introduce this_cpu_cmpxchg_double(). this_cpu_cmpxchg_double() allows
the comparison between two consecutive words and replaces them if
there is a match.bool this_cpu_cmpxchg_double(pcp1, pcp2,
old_word1, old_word2, new_word1, new_word2)this_cpu_cmpxchg_double does not return the old value (difficult since
there are two words) but a boolean indicating if the operation was
successful.The first percpu variable must be double word aligned!
-tj: Updated to return bool instead of int, converted size check to
BUILD_BUG_ON() instead of VM_BUG_ON() and other cosmetic changes.Signed-off-by: Christoph Lameter
Signed-off-by: Tejun Heo
18 Dec, 2010
1 commit
-
Generic code to provide new per cpu atomic features
this_cpu_cmpxchg
this_cpu_xchgFallback occurs to functions using interrupts disable/enable
to ensure correct per cpu atomicity.Fallback to regular cmpxchg and xchg is not possible since per cpu atomic
semantics include the guarantee that the current cpus per cpu data is
accessed atomically. Use of regular cmpxchg and xchg requires the
determination of the address of the per cpu data before regular cmpxchg
or xchg which therefore cannot be atomically included in an xchg or
cmpxchg without segment override.tj: - Relocated new ops to conform better to the general organization.
- This patch contains a trivial comment fix.Signed-off-by: Christoph Lameter
Signed-off-by: Tejun Heo
17 Dec, 2010
2 commits
-
- include/linux/percpu.h: this_cpu_add_return() and friends were
located next to __this_cpu_add_return(). However, the overall
organization is to first group by preemption safeness. Relocate
this_cpu_add_return() and friends to preemption-safe area.- arch/x86/include/asm/percpu.h: Relocate percpu_add_return_op() after
other more basic operations. Relocate [__]this_cpu_add_return_8()
so that they're first grouped by preemption safeness.Signed-off-by: Tejun Heo
Cc: Christoph Lameter -
Introduce generic support for this_cpu_add_return etc.
The fallback is to realize these operations with simpler __this_cpu_ops.
tj: - Reformatted __cpu_size_call_return2() to make it more consistent
with its neighbors.
- Dropped unnecessary temp variable ret__ from
__this_cpu_generic_add_return().Reviewed-by: Tejun Heo
Reviewed-by: Mathieu Desnoyers
Acked-by: H. Peter Anvin
Signed-off-by: Christoph Lameter
Signed-off-by: Tejun Heo
23 Oct, 2010
1 commit
-
* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu:
percpu: update comments to reflect that percpu allocations are always zero-filled
percpu: Optimize __get_cpu_var()
x86, percpu: Optimize this_cpu_ptr
percpu: clear memory allocated with the km allocator
percpu: fix build breakage on s390 and cleanup build configuration tests
percpu: use percpu allocator on UP too
percpu: reduce PCPU_MIN_UNIT_SIZE to 32k
vmalloc: pcpu_get/free_vm_areas() aren't needed on UPFixed up trivial conflicts in include/linux/percpu.h
21 Sep, 2010
1 commit
-
These are similar to {get,put}_cpu_var() except for dynamically
allocated per-cpu memory.Signed-off-by: Peter Zijlstra
Acked-by: Tejun Heo
LKML-Reference:
Signed-off-by: Ingo Molnar
08 Sep, 2010
2 commits
-
On UP, percpu allocations were redirected to kmalloc. This has the
following problems.* For certain amount of allocations (determined by
PERCPU_DYNAMIC_EARLY_SLOTS and PERCPU_DYNAMIC_EARLY_SIZE), percpu
allocator can be used before the usual kernel memory allocator is
brought online. On SMP, this is used to initialize the kernel
memory allocator.* percpu allocator honors alignment upto PAGE_SIZE but kmalloc()
doesn't. For example, workqueue makes use of larger alignments for
cpu_workqueues.Currently, users of percpu allocators need to handle UP differently,
which is somewhat fragile and ugly. Other than small amount of
memory, there isn't much to lose by enabling percpu allocator on UP.
It can simply use kernel memory based chunk allocation which was added
for SMP archs w/o MMUs.This patch removes mm/percpu_up.c, builds mm/percpu.c on UP too and
makes UP build use percpu-km. As percpu addresses and kernel
addresses are always identity mapped and static percpu variables don't
need any special treatment, nothing is arch dependent and mm/percpu.c
implements generic setup_per_cpu_areas() for UP.Signed-off-by: Tejun Heo
Reviewed-by: Christoph Lameter
Acked-by: Pekka Enberg -
In preparation of enabling percpu allocator for UP, reduce
PCPU_MIN_UNIT_SIZE to 32k. On UP, the first chunk doesn't have to
include static percpu variables and chunk size can be smaller which is
important as UP percpu allocator will use contiguous kernel memory to
populate chunks.PCPU_MIN_UNIT_SIZE also determines the maximum supported allocation
size but 32k should still be enough.Signed-off-by: Tejun Heo
Reviewed-by: Christoph Lameter
07 Aug, 2010
1 commit
-
UP accessors didn't take care of __percpu notations leading to a lot
of spurious sparse warnings on UP configurations. Fix it.Signed-off-by: Namhyung Kim
Signed-off-by: Tejun Heo
28 Jun, 2010
2 commits
-
This patch updates percpu allocator such that it can serve limited
amount of allocation before slab comes online. This is primarily to
allow slab to depend on working percpu allocator.Two parameters, PERCPU_DYNAMIC_EARLY_SIZE and SLOTS, determine how
much memory space and allocation map slots are reserved. If this
reserved area is exhausted, WARN_ON_ONCE() will trigger and allocation
will fail till slab comes online.The following changes are made to implement early alloc.
* pcpu_mem_alloc() now checks slab_is_available()
* Chunks are allocated using pcpu_mem_alloc()
* Init paths make sure ai->dyn_size is at least as large as
PERCPU_DYNAMIC_EARLY_SIZE.* Initial alloc maps are allocated in __initdata and copied to
kmalloc'd areas once slab is online.Signed-off-by: Tejun Heo
Cc: Christoph Lameter -
In pcpu_build_alloc_info() and pcpu_embed_first_chunk(), @dyn_size was
ssize_t, -1 meant auto-size, 0 forced 0 and positive meant minimum
size. There's no use case for forcing 0 and the upcoming early alloc
support always requires non-zero dynamic size. Make @dyn_size always
mean minimum dyn_size.While at it, make pcpu_build_alloc_info() static which doesn't have
any external caller as suggested by David Rientjes.Signed-off-by: Tejun Heo
Cc: David Rientjes
06 Apr, 2010
1 commit
-
* 'slabh' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc:
eeepc-wmi: include slab.h
staging/otus: include slab.h from usbdrv.h
percpu: don't implicitly include slab.h from percpu.h
kmemcheck: Fix build errors due to missing slab.h
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
iwlwifi: don't include iwl-dev.h from iwl-devtrace.h
x86: don't include slab.h from arch/x86/include/asm/pgtable_32.hFix up trivial conflicts in include/linux/percpu.h due to
is_kernel_percpu_address() having been introduced since the slab.h
cleanup with the percpu_up.c splitup.
30 Mar, 2010
1 commit
-
percpu.h has always been including slab.h to get k[mz]alloc/free() for
UP inline implementation. percpu.h being used by very low level
headers including module.h and sched.h, this meant that a lot files
unintentionally got slab.h inclusion.Lee Schermerhorn was trying to make topology.h use percpu.h and got
bitten by this implicit inclusion. The right thing to do is break
this ultimately unnecessary dependency. The previous patch added
explicit inclusion of either gfp.h or slab.h to the source files using
them. This patch updates percpu.h such that slab.h is no longer
included from percpu.h.Signed-off-by: Tejun Heo
Reviewed-by: Christoph Lameter
Cc: Ingo Molnar
Cc: Lee Schermerhorn
29 Mar, 2010
1 commit
-
lockdep has custom code to check whether a pointer belongs to static
percpu area which is somewhat broken. Implement proper
is_kernel/module_percpu_address() and replace the custom code.On UP, percpu variables are regular static variables and can't be
distinguished from them. Always return %false on UP.Signed-off-by: Tejun Heo
Acked-by: Peter Zijlstra
Cc: Rusty Russell
Cc: Ingo Molnar
05 Jan, 2010
1 commit
-
Conflicts:
arch/powerpc/platforms/pseries/hvCall.S
include/linux/percpu.h