07 Aug, 2010
1 commit
-
* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/penberg/slab-2.6:
slub: Allow removal of slab caches during boot
Revert "slub: Allow removal of slab caches during boot"
slub numa: Fix rare allocation from unexpected node
slab: use deferable timers for its periodic housekeeping
slub: Use kmem_cache flags to detect if slab is in debugging mode.
slub: Allow removal of slab caches during boot
slub: Check kasprintf results in kmem_cache_init()
SLUB: Constants need UL
slub: Use a constant for a unspecified node.
SLOB: Free objects to their own list
slab: fix caller tracking on !CONFIG_DEBUG_SLAB && CONFIG_TRACING
05 Aug, 2010
1 commit
03 Aug, 2010
2 commits
-
Serialize kmem_cache_create and kmem_cache_destroy using the slub_lock. Only
possible after the use of the slub_lock during dynamic dma creation has been
removed.Then make sure that the setup of the slab sysfs entries does not race
with kmem_cache_create and kmem_cache destroy.If a slab cache is removed before we have setup sysfs then simply skip over
the sysfs handling.Cc: Benjamin Herrenschmidt
Cc: Roland Dreier
Signed-off-by: Christoph Lameter
Signed-off-by: Pekka Enberg -
This reverts commit f5b801ac38a9612b380ee9a75ab1861f0594e79f.
29 Jul, 2010
1 commit
-
The network developers have seen sporadic allocations resulting in objects
coming from unexpected NUMA nodes despite asking for objects from a
specific node.This is due to get_partial() calling get_any_partial() if partial
slabs are exhausted for a node even if a node was specified and therefore
one would expect allocations only from the specified node.get_any_partial() sporadically may return a slab from a foreign
node to gradually reduce the size of partial lists on remote nodes
and thereby reduce total memory use for a slab cache.The behavior is controlled by the remote_defrag_ratio of each cache.
Strictly speaking this is permitted behavior since __GFP_THISNODE was
not specified for the allocation but it is certain surprising.This patch makes sure that the remote defrag behavior only occurs
if no node was specified.Signed-off-by: Christoph Lameter
Signed-off-by: Pekka Enberg
16 Jul, 2010
5 commits
-
The cacheline with the flags is reachable from the hot paths after the
percpu allocator changes went in. So there is no need anymore to put a
flag into each slab page. Get rid of the SlubDebug flag and use
the flags in kmem_cache instead.Acked-by: David Rientjes
Signed-off-by: Christoph Lameter
Signed-off-by: Pekka Enberg -
If a slab cache is removed before we have setup sysfs then simply skip over
the sysfs handling.Cc: Benjamin Herrenschmidt
Cc: Roland Dreier
Acked-by: David Rientjes
Signed-off-by: Christoph Lameter
Signed-off-by: Pekka Enberg -
Small allocations may fail during slab bringup which is fatal. Add a BUG_ON()
so that we fail immediately rather than failing later during sysfs
processing.Acked-by: David Rientjes
Signed-off-by: Christoph Lameter
Signed-off-by: Pekka Enberg -
UL suffix is missing in some constants. Conform to how slab.h uses constants.
Acked-by: David Rientjes
Signed-off-by: Christoph Lameter
Signed-off-by: Pekka Enberg -
kmalloc_node() and friends can be passed a constant -1 to indicate
that no choice was made for the node from which the object needs to
come.Use NUMA_NO_NODE instead of -1.
CC: KAMEZAWA Hiroyuki
Signed-off-by: David Rientjes
Signed-off-by: Christoph Lameter
Signed-off-by: Pekka Enberg
10 Jun, 2010
1 commit
-
…ic/random-tracing into perf/core
09 Jun, 2010
1 commit
-
We have been resisting new ftrace plugins and removing existing
ones, and kmemtrace has been superseded by kmem trace events
and perf-kmem, so we remove it.Signed-off-by: Li Zefan
Acked-by: Pekka Enberg
Acked-by: Eduard - Gabriel Munteanu
Cc: Ingo Molnar
Cc: Steven Rostedt
[ remove kmemtrace from the makefile, handle slob too ]
Signed-off-by: Frederic Weisbecker
31 May, 2010
1 commit
-
* 'slub/urgent' of git://git.kernel.org/pub/scm/linux/kernel/git/penberg/slab-2.6:
SLUB: Allow full duplication of kmalloc array for 390
slub: move kmem_cache_node into it's own cacheline
25 May, 2010
2 commits
-
Before applying this patch, cpuset updates task->mems_allowed and
mempolicy by setting all new bits in the nodemask first, and clearing all
old unallowed bits later. But in the way, the allocator may find that
there is no node to alloc memory.The reason is that cpuset rebinds the task's mempolicy, it cleans the
nodes which the allocater can alloc pages on, for example:(mpol: mempolicy)
task1 task1's mpol task2
alloc page 1
alloc on node0? NO 1
1 change mems from 1 to 0
1 rebind task1's mpol
0-1 set new bits
0 clear disallowed bits
alloc on node1? NO 0
...
can't alloc page
goto oomThis patch fixes this problem by expanding the nodes range first(set newly
allowed bits) and shrink it lazily(clear newly disallowed bits). So we
use a variable to tell the write-side task that read-side task is reading
nodemask, and the write-side task clears newly disallowed nodes after
read-side task ends the current memory allocation.[akpm@linux-foundation.org: fix spello]
Signed-off-by: Miao Xie
Cc: David Rientjes
Cc: Nick Piggin
Cc: Paul Menage
Cc: Lee Schermerhorn
Cc: Hugh Dickins
Cc: Ravikiran Thirumalai
Cc: KOSAKI Motohiro
Cc: Christoph Lameter
Cc: Andi Kleen
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
This patch is meant to improve the performance of SLUB by moving the local
kmem_cache_node lock into it's own cacheline separate from kmem_cache.
This is accomplished by simply removing the local_node when NUMA is enabled.On my system with 2 nodes I saw around a 5% performance increase w/
hackbench times dropping from 6.2 seconds to 5.9 seconds on average. I
suspect the performance gain would increase as the number of nodes
increases, but I do not have the data to currently back that up.Bugzilla-Reference: http://bugzilla.kernel.org/show_bug.cgi?id=15713
Cc:
Reported-by: Alex Shi
Tested-by: Alex Shi
Acked-by: Yanmin Zhang
Acked-by: Christoph Lameter
Signed-off-by: Alexander Duyck
Signed-off-by: Pekka Enberg
22 May, 2010
4 commits
-
… 'slub/fixes' into slab-for-linus
-
The alloc_slab_page() in SLUB uses alloc_pages() if node is '-1'. This means
that node validity check in alloc_pages_node is unnecessary and we can use
alloc_pages_exact_node() to avoid comparison and branch as commit
6484eb3e2a81807722 ("page allocator: do not check NUMA node ID when the caller
knows the node is valid") did for the page allocator.Cc: Christoph Lameter
Reviewed-by: KAMEZAWA Hiroyuki
Reviewed-by: Mel Gorman
Signed-off-by: Minchan Kim
Signed-off-by: Pekka Enberg -
commit 94b528d (kmemtrace: SLUB hooks for caller-tracking functions)
missed tracing kmalloc_large_node in __kmalloc_node_track_caller. We
should trace it same as __kmalloc_node.Acked-by: David Rientjes
Cc: Matt Mackall
Cc: Ingo Molnar
Cc: Vegard Nossum
Signed-off-by: Xiaotian Feng
Signed-off-by: Pekka Enberg -
I discovered that we can overflow stack if CONFIG_SLUB_DEBUG=y and use slabs
with many objects, since list_slab_objects() and process_slab() use
DECLARE_BITMAP(map, page->objects).With 65535 bits, we use 8192 bytes of stack ...
Switch these allocations to dynamic allocations.
Signed-off-by: Eric Dumazet
Signed-off-by: Christoph Lameter
Signed-off-by: Pekka Enberg
20 May, 2010
1 commit
-
Acked-by: Herbert Xu
Signed-off-by: David Woodhouse
Signed-off-by: Pekka Enberg
06 May, 2010
1 commit
-
Function init_kmem_cache_nodes is incorrect when checking upper limitation of
kmalloc_caches. The breakage was introduced by commit
91efd773c74bb26b5409c85ad755d536448e229c ("dma kmalloc handling fixes").Acked-by: Christoph Lameter
Signed-off-by: Pekka Enberg
10 Apr, 2010
1 commit
-
As suggested by Linus, fix up kmem_ptr_validate() to handle non-kernel pointers
more graciously. The patch changes kmem_ptr_validate() to use the newly
introduced kern_ptr_validate() helper to check that a pointer is a valid kernel
pointer before we attempt to convert it into a 'struct page'.Cc: Andrew Morton
Cc: Ingo Molnar
Cc: Matt Mackall
Cc: Nick Piggin
Signed-off-by: Pekka Enberg
Acked-by: Christoph Lameter
Acked-by: David Rientjes
Signed-off-by: Linus Torvalds
13 Mar, 2010
1 commit
-
* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial: (56 commits)
doc: fix typo in comment explaining rb_tree usage
Remove fs/ntfs/ChangeLog
doc: fix console doc typo
doc: cpuset: Update the cpuset flag file
Fix of spelling in arch/sparc/kernel/leon_kernel.c no longer needed
Remove drivers/parport/ChangeLog
Remove drivers/char/ChangeLog
doc: typo - Table 1-2 should refer to "status", not "statm"
tree-wide: fix typos "ass?o[sc]iac?te" -> "associate" in comments
No need to patch AMD-provided drivers/gpu/drm/radeon/atombios.h
devres/irq: Fix devm_irq_match comment
Remove reference to kthread_create_on_cpu
tree-wide: Assorted spelling fixes
tree-wide: fix 'lenght' typo in comments and code
drm/kms: fix spelling in error message
doc: capitalization and other minor fixes in pnp doc
devres: typo fix s/dev/devm/
Remove redundant trailing semicolons from macros
fix typo "definetly" -> "definitely" in comment
tree-wide: s/widht/width/g typo in comments
...Fix trivial conflict in Documentation/laptops/00-INDEX
08 Mar, 2010
3 commits
-
Conflicts:
Documentation/filesystems/proc.txt
arch/arm/mach-u300/include/mach/debug-macro.S
drivers/net/qlge/qlge_ethtool.c
drivers/net/qlge/qlge_main.c
drivers/net/typhoon.c -
Constify struct sysfs_ops.
This is part of the ops structure constification
effort started by Arjan van de Ven et al.Benefits of this constification:
* prevents modification of data that is shared
(referenced) by many other structure instances
at runtime* detects/prevents accidental (but not intentional)
modification attempts on archs that enforce
read-only kernel data at runtime* potentially better optimized code as the compiler
can assume that the const data cannot be changed* the compiler/linker move const data into .rodata
and therefore exclude them from false sharingSigned-off-by: Emese Revfy
Acked-by: David Teigland
Acked-by: Matt Domsch
Acked-by: Maciej Sosnowski
Acked-by: Hans J. Koch
Acked-by: Pekka Enberg
Acked-by: Jens Axboe
Acked-by: Stephen Hemminger
Signed-off-by: Greg Kroah-Hartman -
Constify struct kset_uevent_ops.
This is part of the ops structure constification
effort started by Arjan van de Ven et al.Benefits of this constification:
* prevents modification of data that is shared
(referenced) by many other structure instances
at runtime* detects/prevents accidental (but not intentional)
modification attempts on archs that enforce
read-only kernel data at runtime* potentially better optimized code as the compiler
can assume that the const data cannot be changed* the compiler/linker move const data into .rodata
and therefore exclude them from false sharingSigned-off-by: Emese Revfy
Signed-off-by: Greg Kroah-Hartman
04 Mar, 2010
2 commits
-
The slab tree adds a percpu variable usage case (commit
9dfc6e68bfe6ee452efb1a4e9ca26a9007f2b864 "SLUB: Use this_cpu operations in
slub"), but the percpu tree removes the prefixing of percpu variables (commit
dd17c8f72993f9461e9c19250e3f155d6d99df22 "percpu: remove per_cpu__ prefix"),
thus causing the following compilation error:CC mm/slub.o
mm/slub.c: In function ‘alloc_kmem_cache_cpus’:
mm/slub.c:2078: error: implicit declaration of function ‘per_cpu_var’
mm/slub.c:2078: warning: assignment makes pointer from integer without a cast
make[1]: *** [mm/slub.o] Error 1Signed-off-by: Pekka Enberg
27 Feb, 2010
1 commit
-
This patch allow to inject faults only for specific slabs.
In order to preserve default behavior cache filter is off by
default (all caches are faulty).One may define specific set of slabs like this:
# mark skbuff_head_cache as faulty
echo 1 > /sys/kernel/slab/skbuff_head_cache/failslab
# Turn on cache filter (off by default)
echo 1 > /sys/kernel/debug/failslab/cache-filter
# Turn on fault injection
echo 1 > /sys/kernel/debug/failslab/times
echo 1 > /sys/kernel/debug/failslab/probabilityAcked-by: David Rientjes
Acked-by: Akinobu Mita
Acked-by: Christoph Lameter
Signed-off-by: Dmitry Monakhov
Signed-off-by: Pekka Enberg
05 Feb, 2010
1 commit
-
Some comments misspell "should" or "shouldn't"; this fixes them. No code changes.
Signed-off-by: Adam Buchbinder
Signed-off-by: Jiri Kosina
23 Jan, 2010
2 commits
-
1. We need kmalloc_percpu for all of the now extended kmalloc caches
array not just for each shift value.2. init_kmem_cache_nodes() must assume node 0 locality for statically
allocated dma kmem_cache structures even after boot is complete.Reported-and-tested-by: Alex Chiang
Signed-off-by: Christoph Lameter
Signed-off-by: Pekka Enberg -
`s' cannot be NULL if kmalloc_caches is not NULL.
This conditional would trigger a NULL pointer on `s', anyway, since it is
immediately derefernced if true.Acked-by: Christoph Lameter
Signed-off-by: David Rientjes
Signed-off-by: Pekka Enberg
20 Dec, 2009
4 commits
-
this_cpu_inc() translates into a single instruction on x86 and does not
need any register. So use it in stat(). We also want to avoid the
calculation of the per cpu kmem_cache_cpu structure pointer. So pass
a kmem_cache pointer instead of a kmem_cache_cpu pointer.Signed-off-by: Christoph Lameter
Signed-off-by: Pekka Enberg -
Remove the fields in struct kmem_cache_cpu that were used to cache data from
struct kmem_cache when they were in different cachelines. The cacheline that
holds the per cpu array pointer now also holds these values. We can cut down
the struct kmem_cache_cpu size to almost half.The get_freepointer() and set_freepointer() functions that used to be only
intended for the slow path now are also useful for the hot path since access
to the size field does not require accessing an additional cacheline anymore.
This results in consistent use of functions for setting the freepointer of
objects throughout SLUB.Also we initialize all possible kmem_cache_cpu structures when a slab is
created. No need to initialize them when a processor or node comes online.Signed-off-by: Christoph Lameter
Signed-off-by: Pekka Enberg -
Dynamic DMA kmalloc cache allocation is troublesome since the
new percpu allocator does not support allocations in atomic contexts.
Reserve some statically allocated kmalloc_cpu structures instead.Signed-off-by: Christoph Lameter
Signed-off-by: Pekka Enberg -
Using per cpu allocations removes the needs for the per cpu arrays in the
kmem_cache struct. These could get quite big if we have to support systems
with thousands of cpus. The use of this_cpu_xx operations results in:1. The size of kmem_cache for SMP configuration shrinks since we will only
need 1 pointer instead of NR_CPUS. The same pointer can be used by all
processors. Reduces cache footprint of the allocator.2. We can dynamically size kmem_cache according to the actual nodes in the
system meaning less memory overhead for configurations that may potentially
support up to 1k NUMA nodes / 4k cpus.3. We can remove the diddle widdle with allocating and releasing of
kmem_cache_cpu structures when bringing up and shutting down cpus. The cpu
alloc logic will do it all for us. Removes some portions of the cpu hotplug
functionality.4. Fastpath performance increases since per cpu pointer lookups and
address calculations are avoided.V7-V8
- Convert missed get_cpu_slab() under CONFIG_SLUB_STATSSigned-off-by: Christoph Lameter
Signed-off-by: Pekka Enberg
15 Dec, 2009
1 commit
-
…/git/tip/linux-2.6-tip
* 'perf-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip:
perf sched: Fix build failure on sparc
perf bench: Add "all" pseudo subsystem and "all" pseudo suite
perf tools: Introduce perf_session class
perf symbols: Ditch dso->find_symbol
perf symbols: Allow lookups by symbol name too
perf symbols: Add missing "Variables" entry to map_type__name
perf symbols: Add support for 'variable' symtabs
perf symbols: Introduce ELF counterparts to symbol_type__is_a
perf symbols: Introduce symbol_type__is_a
perf symbols: Rename kthreads to kmaps, using another abstraction for it
perf tools: Allow building for ARM
hw-breakpoints: Handle bad modify_user_hw_breakpoint off-case return value
perf tools: Allow cross compiling
tracing, slab: Fix no callsite ifndef CONFIG_KMEMTRACE
tracing, slab: Define kmem_cache_alloc_notrace ifdef CONFIG_TRACINGTrivial conflict due to different fixes to modify_user_hw_breakpoint()
in include/linux/hw_breakpoint.h
12 Dec, 2009
1 commit
11 Dec, 2009
1 commit
-
Define kmem_trace_alloc_{,node}_notrace() if CONFIG_TRACING is
enabled, otherwise perf-kmem will show wrong stats ifndef
CONFIG_KMEM_TRACE, because a kmalloc() memory allocation may
be traced by both trace_kmalloc() and trace_kmem_cache_alloc().Signed-off-by: Li Zefan
Reviewed-by: Pekka Enberg
Cc: Christoph Lameter
Cc: Steven Rostedt
Cc: Frederic Weisbecker
Cc: linux-mm@kvack.org
Cc: Eduard - Gabriel Munteanu
LKML-Reference:
Signed-off-by: Ingo Molnar
29 Nov, 2009
1 commit
-
The unlikely() annotation in slab_alloc() covers too much of the expression.
It's actually very likely that the object is not NULL so use unlikely() only
for the __GFP_ZERO expression like SLAB does.The patch reduces kernel text by 29 bytes on x86-64:
text data bss dec hex filename
24185 8560 176 32921 8099 mm/slub.o.orig
24156 8560 176 32892 807c mm/slub.oAcked-by: Christoph Lameter
Signed-off-by: Pekka Enberg