08 Dec, 2006
4 commits
-
Rearrange the struct members in the 'struct zonelist_cache' structure, so
as to put the readonly (once initialized) z_to_n[] array first, where it
will come right after the zones[] array in struct zonelist.This pretty much eliminates the chance that the two frequently written
elements of 'struct zonelist_cache', the fullzones bitmap and last_full_zap
times, will end up on the same cache line as the performance sensitive,
frequently read, never (after init) written zones[] array.Keeping frequently written data off frequently read cache lines is good for
performance.Thanks to Rohit Seth for the suggestion.
Signed-off-by: Paul Jackson
Cc: Rohit Seth
Cc: Paul Menage
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
Optimize the critical zonelist scanning for free pages in the kernel memory
allocator by caching the zones that were found to be full recently, and
skipping them.Remembers the zones in a zonelist that were short of free memory in the
last second. And it stashes a zone-to-node table in the zonelist struct,
to optimize that conversion (minimize its cache footprint.)Recent changes:
This differs in a significant way from a similar patch that I
posted a week ago. Now, instead of having a nodemask_t of
recently full nodes, I have a bitmask of recently full zones.
This solves a problem that last weeks patch had, which on
systems with multiple zones per node (such as DMA zone) would
take seeing any of these zones full as meaning that all zones
on that node were full.Also I changed names - from "zonelist faster" to "zonelist cache",
as that seemed to better convey what we're doing here - caching
some of the key zonelist state (for faster access.)See below for some performance benchmark results. After all that
discussion with David on why I didn't need them, I went and got
some ;). I wanted to verify that I had not hurt the normal case
of memory allocation noticeably. At least for my one little
microbenchmark, I found (1) the normal case wasn't affected, and
(2) workloads that forced scanning across multiple nodes for
memory improved up to 10% fewer System CPU cycles and lower
elapsed clock time ('sys' and 'real'). Good. See details, below.I didn't have the logic in get_page_from_freelist() for various
full nodes and zone reclaim failures correct. That should be
fixed up now - notice the new goto labels zonelist_scan,
this_zone_full, and try_next_zone, in get_page_from_freelist().There are two reasons I persued this alternative, over some earlier
proposals that would have focused on optimizing the fake numa
emulation case by caching the last useful zone:1) Contrary to what I said before, we (SGI, on large ia64 sn2 systems)
have seen real customer loads where the cost to scan the zonelist
was a problem, due to many nodes being full of memory before
we got to a node we could use. Or at least, I think we have.
This was related to me by another engineer, based on experiences
from some time past. So this is not guaranteed. Most likely, though.The following approach should help such real numa systems just as
much as it helps fake numa systems, or any combination thereof.2) The effort to distinguish fake from real numa, using node_distance,
so that we could cache a fake numa node and optimize choosing
it over equivalent distance fake nodes, while continuing to
properly scan all real nodes in distance order, was going to
require a nasty blob of zonelist and node distance munging.The following approach has no new dependency on node distances or
zone sorting.See comment in the patch below for a description of what it actually does.
Technical details of note (or controversy):
- See the use of "zlc_active" and "did_zlc_setup" below, to delay
adding any work for this new mechanism until we've looked at the
first zone in zonelist. I figured the odds of the first zone
having the memory we needed were high enough that we should just
look there, first, then get fancy only if we need to keep looking.- Some odd hackery was needed to add items to struct zonelist, while
not tripping up the custom zonelists built by the mm/mempolicy.c
code for MPOL_BIND. My usual wordy comments below explain this.
Search for "MPOL_BIND".- Some per-node data in the struct zonelist is now modified frequently,
with no locking. Multiple CPU cores on a node could hit and mangle
this data. The theory is that this is just performance hint data,
and the memory allocator will work just fine despite any such mangling.
The fields at risk are the struct 'zonelist_cache' fields 'fullzones'
(a bitmask) and 'last_full_zap' (unsigned long jiffies). It should
all be self correcting after at most a one second delay.- This still does a linear scan of the same lengths as before. All
I've optimized is making the scan faster, not algorithmically
shorter. It is now able to scan a compact array of 'unsigned
short' in the case of many full nodes, so one cache line should
cover quite a few nodes, rather than each node hitting another
one or two new and distinct cache lines.- If both Andi and Nick don't find this too complicated, I will be
(pleasantly) flabbergasted.- I removed the comment claiming we only use one cachline's worth of
zonelist. We seem, at least in the fake numa case, to have put the
lie to that claim.- I pay no attention to the various watermarks and such in this performance
hint. A node could be marked full for one watermark, and then skipped
over when searching for a page using a different watermark. I think
that's actually quite ok, as it will tend to slightly increase the
spreading of memory over other nodes, away from a memory stressed node.===============
Performance - some benchmark results and analysis:
This benchmark runs a memory hog program that uses multiple
threads to touch alot of memory as quickly as it can.Multiple runs were made, touching 12, 38, 64 or 90 GBytes out of
the total 96 GBytes on the system, and using 1, 19, 37, or 55
threads (on a 56 CPU system.) System, user and real (elapsed)
timings were recorded for each run, shown in units of seconds,
in the table below.Two kernels were tested - 2.6.18-mm3 and the same kernel with
this zonelist caching patch added. The table also shows the
percentage improvement the zonelist caching sys time is over
(lower than) the stock *-mm kernel.number 2.6.18-mm3 zonelist-cache delta (< 0 good) percent
GBs N ------------ -------------- ---------------- systime
mem threads sys user real sys user real sys user real better
12 1 153 24 177 151 24 176 -2 0 -1 1%
12 19 99 22 8 99 22 8 0 0 0 0%
12 37 111 25 6 112 25 6 1 0 0 -0%
12 55 115 25 5 110 23 5 -5 -2 0 4%
38 1 502 74 576 497 73 570 -5 -1 -6 0%
38 19 426 78 48 373 76 39 -53 -2 -9 12%
38 37 544 83 36 547 82 36 3 -1 0 -0%
38 55 501 77 23 511 80 24 10 3 1 -1%
64 1 917 125 1042 890 124 1014 -27 -1 -28 2%
64 19 1118 138 119 965 141 103 -153 3 -16 13%
64 37 1202 151 94 1136 150 81 -66 -1 -13 5%
64 55 1118 141 61 1072 140 58 -46 -1 -3 4%
90 1 1342 177 1519 1275 174 1450 -67 -3 -69 4%
90 19 2392 199 192 2116 189 176 -276 -10 -16 11%
90 37 3313 238 175 2972 225 145 -341 -13 -30 10%
90 55 1948 210 104 1843 213 100 -105 3 -4 5%Notes:
1) This test ran a memory hog program that started a specified number N of
threads, and had each thread allocate and touch 1/N'th of
the total memory to be used in the test run in a single loop,
writing a constant word to memory, one store every 4096 bytes.
Watching this test during some earlier trial runs, I would see
each of these threads sit down on one CPU and stay there, for
the remainder of the pass, a different CPU for each thread.2) The 'real' column is not comparable to the 'sys' or 'user' columns.
The 'real' column is seconds wall clock time elapsed, from beginning
to end of that test pass. The 'sys' and 'user' columns are total
CPU seconds spent on that test pass. For a 19 thread test run,
for example, the sum of 'sys' and 'user' could be up to 19 times the
number of 'real' elapsed wall clock seconds.3) Tests were run on a fresh, single-user boot, to minimize the amount
of memory already in use at the start of the test, and to minimize
the amount of background activity that might interfere.4) Tests were done on a 56 CPU, 28 Node system with 96 GBytes of RAM.
5) Notice that the 'real' time gets large for the single thread runs, even
though the measured 'sys' and 'user' times are modest. I'm not sure what
that means - probably something to do with it being slow for one thread to
be accessing memory along ways away. Perhaps the fake numa system, running
ostensibly the same workload, would not show this substantial degradation
of 'real' time for one thread on many nodes -- lets hope not.6) The high thread count passes (one thread per CPU - on 55 of 56 CPUs)
ran quite efficiently, as one might expect. Each pair of threads needed
to allocate and touch the memory on the node the two threads shared, a
pleasantly parallizable workload.7) The intermediate thread count passes, when asking for alot of memory forcing
them to go to a few neighboring nodes, improved the most with this zonelist
caching patch.Conclusions:
* This zonelist cache patch probably makes little difference one way or the
other for most workloads on real numa hardware, if those workloads avoid
heavy off node allocations.
* For memory intensive workloads requiring substantial off-node allocations
on real numa hardware, this patch improves both kernel and elapsed timings
up to ten per-cent.
* For fake numa systems, I'm optimistic, but will have to leave that up to
Rohit Seth to actually test (once I get him a 2.6.18 backport.)Signed-off-by: Paul Jackson
Cc: Rohit Seth
Cc: Christoph Lameter
Cc: David Rientjes
Cc: Paul Menage
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
The zone table is mostly not needed. If we have a node in the page flags
then we can get to the zone via NODE_DATA() which is much more likely to be
already in the cpu cache.In case of SMP and UP NODE_DATA() is a constant pointer which allows us to
access an exact replica of zonetable in the node_zones field. In all of
the above cases there will be no need at all for the zone table.The only remaining case is if in a NUMA system the node numbers do not fit
into the page flags. In that case we make sparse generate a table that
maps sections to nodes and use that table to to figure out the node number.
This table is sized to fit in a single cache line for the known 32 bit
NUMA platform which makes it very likely that the information can be
obtained without a cache miss.For sparsemem the zone table seems to be have been fairly large based on
the maximum possible number of sections and the number of zones per node.
There is some memory saving by removing zone_table. The main benefit is to
reduce the cache foootprint of the VM from the frequent lookups of zones.
Plus it simplifies the page allocator.[akpm@osdl.org: build fix]
Signed-off-by: Christoph Lameter
Cc: Dave Hansen
Cc: Andy Whitcroft
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
With CONFIG_SMP=n:
drivers/input/ff-memless.c:384: warning: implicit declaration of function 'local_bh_disable'
drivers/input/ff-memless.c:393: warning: implicit declaration of function 'local_bh_enable'Really linux/spinlock.h should include linux/interrupt.h. But interrupt.h
includes sched.h which will need spinlock.h.So the patch breaks the _bh declarations out into a separate header and
includes it in both interrupt.h and spinlock.h.Cc: "Randy.Dunlap"
Cc: Andi Kleen
Cc:
Cc: Ingo Molnar
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
07 Dec, 2006
7 commits
-
* 'upstream' of git://ftp.linux-mips.org/pub/scm/upstream-linus:
[MIPS] Import updates from i386's i8259.c
[MIPS] *-berr: Header inclusions for DEC bus error handlers
[MIPS] Compile __do_IRQ() when really needed
[MIPS] genirq: use name instead of typename
[MIPS] Do not use handle_level_irq for ioasic_dma_irq_type.
[MIPS] pte_offset(dir,addr): parenthesis fix -
Any code that relies on the volatile would be a bug waiting to happen
anyway.Don't encourage people to think that putting 'volatile' on data
structures somehow fixes problems. We should always use proper locking
(and other serialization) techniques.Signed-off-by: Linus Torvalds
-
This is a resubmission of patches originally created by Ingo Molnar.
The link below is the initial (?) posting of the patch.http://marc.theaimsgroup.com/?l=linux-kernel&m=115217423929806&w=2
Remove 'volatile' from spinlock_types as it causes GCC to generate bad
code (see link) and locking should be used on kernel data.Signed-off-by: Art Haas
Signed-off-by: Linus Torvalds -
Import many updates from i386's i8259.c, especially genirq transitions.
Signed-off-by: Atsushi Nemoto
Signed-off-by: Ralf Baechle -
This patch adds missing parenthesis around 'dir' argument in pte_offset()
macro definition.It also removes an extra space in the definition of pte_offset_kernel()
macro.Signed-off-by: Franck Bui-Huu
Signed-off-by: Ralf Baechle -
* master.kernel.org:/pub/scm/linux/kernel/git/lethal/sh-2.6: (43 commits)
sh: sh775x/titan fixes for irq header changes.
sh: update r7780rp defconfig.
sh: compile fixes for header cleanup.
sh: Fixup pte_mkhuge() build failure.
sh: set KBUILD_IMAGE to something sensible.
sh: show held locks in stack trace with lockdep.
sh: platform_pata support for R7780RP
sh: stacktrace/lockdep/irqflags tracing support.
sh: Fixup movli.l/movco.l atomic ops for gcc4.
sh: dyntick infrastructure.
sh: Clock framework tidying.
sh: Turn off IRQs around get_timer_offset() calls.
sh: Get the PGD right in oops case with 64-bit PTEs.
sh: Fix store queue bitmap end.
sh: More flexible + SH7780 earlyprintk SCIF support.
sh: Fixup various PAGE_SIZE == 4096 assumptions.
sh: Fixup 4K irq stacks.
sh: dma-api channel capability extensions.
sh: Drop name overload in dma-sh.
sh: Make dma-isa depend on ISA_DMA_API.
... -
* git://git.infradead.org/users/dhowells/workq-2.6:
Actually update the fixed up compile failures.
WorkQueue: Fix up arch-specific work items where possible
WorkStruct: make allyesconfig
WorkStruct: Pass the work_struct pointer instead of context data
WorkStruct: Merge the pending bit into the wq_data pointer
WorkStruct: Typedef the work function prototype
WorkStruct: Separate delayable and non-delayable events.
06 Dec, 2006
28 commits
-
The first patch is to the 2.6 kernel include file (for m68knommu), to get
rid of the conditional definitions, otherwise the structures have different
sizes depending on whether there's an FPU or not.Signed-off-by: Greg Ungerer
Signed-off-by: Linus Torvalds -
Add a null definition for irq_canonicalize(). It is used in the gerneric
serial subsystem code, can't compile without it.Signed-off-by: Greg Ungerer
Signed-off-by: Linus Torvalds -
This adds support for RTCs (through genrtc) for M68KNOMMU.
Board-specific code will have to link the appropriate RTC driver to the
mach_hwclk callback, at minimum.Signed-off-by: Gavin Lambert
Signed-off-by: Greg Ungerer
Signed-off-by: Linus Torvalds -
Signed-Off-By: David Howells
-
Conflicts:
drivers/pcmcia/ds.c
Fix up merge failures with Linus's head and fix new compile failures.
Signed-Off-By: David Howells
-
The following moves the creation of IPR interupts into setup-7750.c
and updates a few other things to make it all work after the "Drop
CPU subtype IRQ headers" commit. It boots and runs fine on my titan
board.- adds an ipr_idx to the ipr_data and uses a function in the subtype
code to calculate the address of the IPR registers- adds a function to enable individual interrupt mode for externals
in the subtype code and calls that from the titan board code
instead of doing it directly.- I changed the shift in the ipr_data to be the actual # of bits to
shift, instead of the numnber / 4 - made it easier to match with
the manual.Signed-off-by: Jamie Lenehan
Signed-off-by: Paul Mundt -
When hugetlbpage support isn't enabled, this can be bogus.
Wrap it back in _PAGE_FLAGS_HARD to avoid changes to the
base PTE when not aiming for larger sizes.Signed-off-by: Paul Mundt
-
Wire up all of the essentials for lockdep..
Signed-off-by: Paul Mundt
-
gcc4 gets a bit pissy about the outputs:
include/asm/atomic.h: In function 'atomic_add':
include/asm/atomic.h:37: error: invalid lvalue in asm statement
include/asm/atomic.h:30: error: invalid lvalue in asm output 1
...this ended up being a thinko anyways, so just fix it up.
Verified for proper behaviour with the older toolchains, too.
Signed-off-by: Paul Mundt
-
This adds basic NO_IDLE_HZ support to the SH timer API so timers
are able to wire it up. Taken from the ARM version, as it fit in
to our API with very few changes needed.Signed-off-by: Paul Mundt
-
This syncs up the SH clock framework with the linux/clk.h API,
for which there were only some minor changes required, namely
the clk_get() dev_id and subsequent callsites.Signed-off-by: Paul Mundt
-
There were a number of places that made evil PAGE_SIZE == 4k
assumptions that ended up breaking when trying to play with
8k and 64k page sizes, this fixes those up.The most significant change is the way we load THREAD_SIZE,
previously this was done via:mov #(THREAD_SIZE >> 8), reg
shll8 regto avoid a memory access and allow the immediate load. With
a 64k PAGE_SIZE, we're out of range for the immediate load
size without resorting to special instructions available in
later ISAs (movi20s and so on). The "workaround" for this is
to bump up the shift to 10 and insert a shll2, which gives a
bit more flexibility while still being much cheaper than a
memory access.Signed-off-by: Paul Mundt
-
This extends the SH DMA API for allowing handling of DMA
channels based off of their respective capabilities.A couple of functions are added to the existing API,
the core bits are register_chan_caps() for registering
channel capabilities, and request_dma_bycap() for fetching
a channel dynamically based off of a capability set.Signed-off-by: Mark Glaisher
Signed-off-by: Paul Mundt -
Two of the fields in /proc/[number]/stat are documented in
proc(5) as:kstkesp %lu
The current value of esp (stack pointer), as
found in the kernel stack page for the process.kstkeip %lu
The current EIP (instruction pointer).The SH currently prints the the last SP and PC of the process
inside the kernel, while most other archs use the last user
space values.This patch modifes the SH to display the user space values.
Signed-off-by: Stuart Menefy
Signed-off-by: Paul Mundt -
Handle simple TLB miss faults which can be resolved completely
from the page table in assembler.Signed-off-by: Stuart Menefy
Signed-off-by: Paul Mundt -
This adds support for a generic push switch framework. Adaptable for
various switches, including GPIO switches and the push switches commonly
found on Renesas debug boards.This allows switch states to be trivially reported through sysfs.
Signed-off-by: Paul Mundt
-
Remove extra bits from the pmd structure and store a kernel logical
address rather than a physical address. This allows it to be directly
dereferenced. Another piece of wierdness inherited from x86.Signed-off-by: Stuart Menefy
Signed-off-by: Paul Mundt -
Add TTB accessor functions and give it a sensible default
value. We will use this later for optimizing the fault
path.Signed-off-by: Stuart Menefy
Signed-off-by: Paul Mundt -
Remove the previous saving of fault codes into the thread_struct
as they are never used, and appeared to be inherited from x86.Signed-off-by: Stuart Menefy
Signed-off-by: Paul Mundt -
This adds some preliminary support for the SH-X2 MMU, used by
newer SH-4A parts (particularly SH7785).This MMU implements a 'compat' mode with SH-X MMUs and an
'extended' mode for SH-X2 extended features. Extended features
include additional page sizes (8kB, 4MB, 64MB), as well as the
addition of page execute permissions.The extended mode attributes are placed in a second data array,
which requires us to switch to 64-bit PTEs when in X2 mode.With the addition of the exec perms, we also overhaul the mmap
prots somewhat, now that it's possible to handle them more
intelligently.Signed-off-by: Paul Mundt
-
Simple 7785 placeholders to start hooking up other bits of code.
Signed-off-by: Paul Mundt
-
This drops the various IRQ headers that were floating around
and primarily providing hardcoded IRQ definitions for the
various CPU subtypes. This quickly got to be an unmaintainable
mess, made even more evident by the subtle breakage introduced
by the SH-2 and SH-2A changes.Now that subtypes are able to register IRQ maps directly, just
rip all of the headers out.Signed-off-by: Paul Mundt
-
A number of API changes happened underneath the 7206 patches, update
for everything that broke.Signed-off-by: Paul Mundt
-
Mostly SH-2 wrappers..
Signed-off-by: Yoshinori Sato
Signed-off-by: Paul Mundt -
* master.kernel.org:/pub/scm/linux/kernel/git/jejb/scsi-misc-2.6: (73 commits)
[SCSI] aic79xx: Add ASC-29320LPE ids to driver
[SCSI] stex: version update
[SCSI] stex: change wait loop code
[SCSI] stex: add new device type support
[SCSI] stex: update device id info
[SCSI] stex: adjust default queue length
[SCSI] stex: add value check in hard reset routine
[SCSI] stex: fix controller_info command handling
[SCSI] stex: fix biosparam calculation
[SCSI] megaraid: fix MMIO casts
[SCSI] tgt: fix undefined flush_dcache_page() problem
[SCSI] libsas: better error handling in sas_expander.c
[SCSI] lpfc 8.1.11 : Change version number to 8.1.11
[SCSI] lpfc 8.1.11 : Misc Fixes
[SCSI] lpfc 8.1.11 : Add soft_wwnn sysfs attribute, rename soft_wwn_enable
[SCSI] lpfc 8.1.11 : Removed decoding of PCI Subsystem Id
[SCSI] lpfc 8.1.11 : Add MSI (Message Signalled Interrupts) support
[SCSI] lpfc 8.1.11 : Adjust LOG_FCP logging
[SCSI] lpfc 8.1.11 : Fix Memory leaks
[SCSI] lpfc 8.1.11 : Fix lpfc_multi_ring_support
... -
* master.kernel.org:/pub/scm/linux/kernel/git/brodo/pcmcia-2.6:
[PATCH] pcmcia: at91_cf update
[PATCH] pcmcia: fix m32r_cfc.c compilation
[PATCH] pcmcia: ds.c debug enhancements
[PATCH] pcmcia: at91_cf update
[PATCH] pcmcia: conf.ConfigBase and conf.Present consolidation
[PATCH] pcmcia: remove prod_id indirection
[PATCH] pcmcia: remove manf_id and card_id indirection
[PATCH] pcmcia: IDs for Elan serial PCMCIA devcies
[PATCH] pcmcia: allow for four multifunction subdevices
[PATCH] pcmcia: handle __copy_from_user() return value in ioctl
[PATCH] pcmcia: multifunction card handling fixes
[PATCH] pcmcia: allow shared IRQs on pd6729 sockets
[PATCH] pcmcia: start over after CIS override
[PATCH] cm4000_cs: fix return value check
[PATCH] pcmcia: yet another IDE ID
[PATCH] pcmcia: Add an id to ide-cs.c -
Fix up arch-specific work items where possible to use the new work_struct and
delayed_work structs.Three places that enqueue bits of their stack and then return have been marked
with #error as this is not permitted.Signed-Off-By: David Howells
-
Conflicts:
drivers/ata/libata-scsi.c
include/linux/libata.hFuther merge of Linus's head and compilation fixups.
Signed-Off-By: David Howells
05 Dec, 2006
1 commit
-
Conflicts:
drivers/infiniband/core/iwcm.c
drivers/net/chelsio/cxgb2.c
drivers/net/wireless/bcm43xx/bcm43xx_main.c
drivers/net/wireless/prism54/islpci_eth.c
drivers/usb/core/hub.h
drivers/usb/input/hid-core.c
net/core/netpoll.cFix up merge failures with Linus's head and fix new compilation failures.
Signed-Off-By: David Howells