13 Aug, 2014
1 commit
-
Specifically:
Documentation/locking/lockdep-design.txt
Documentation/locking/lockstat.txt
Documentation/locking/mutex-design.txt
Documentation/locking/rt-mutex-design.txt
Documentation/locking/rt-mutex.txt
Documentation/locking/spinlocks.txt
Documentation/locking/ww-mutex-design.txtSigned-off-by: Davidlohr Bueso
Acked-by: Randy Dunlap
Signed-off-by: Peter Zijlstra
Cc: jason.low2@hp.com
Cc: aswin@hp.com
Cc: Alexei Starovoitov
Cc: Al Viro
Cc: Andrew Morton
Cc: Chris Mason
Cc: Dan Streetman
Cc: David Airlie
Cc: Davidlohr Bueso
Cc: David S. Miller
Cc: Greg Kroah-Hartman
Cc: Heiko Carstens
Cc: Jason Low
Cc: Josef Bacik
Cc: Kees Cook
Cc: Linus Torvalds
Cc: Lubomir Rintel
Cc: Masanari Iida
Cc: Paul E. McKenney
Cc: Randy Dunlap
Cc: Tim Chen
Cc: Vineet Gupta
Cc: fengguang.wu@intel.com
Link: http://lkml.kernel.org/r/1406752916-3341-6-git-send-email-davidlohr@hp.com
Signed-off-by: Ingo Molnar
16 Jul, 2014
5 commits
-
Just like with mutexes (CONFIG_MUTEX_SPIN_ON_OWNER),
encapsulate the dependencies for rwsem optimistic spinning.
No logical changes here as it continues to depend on both
SMP and the XADD algorithm variant.Signed-off-by: Davidlohr Bueso
Acked-by: Jason Low
[ Also make it depend on ARCH_SUPPORTS_ATOMIC_RMW. ]
Signed-off-by: Peter Zijlstra
Link: http://lkml.kernel.org/r/1405112406-13052-2-git-send-email-davidlohr@hp.com
Cc: aswin@hp.com
Cc: Chris Mason
Cc: Davidlohr Bueso
Cc: Josef Bacik
Cc: Linus Torvalds
Cc: Waiman Long
Signed-off-by: Ingo MolnarSigned-off-by: Ingo Molnar
-
Recent optimistic spinning additions to rwsem provide significant performance
benefits on many workloads on large machines. The cost of it was increasing
the size of the rwsem structure by up to 128 bits.However, now that the previous patches in this series bring the overhead of
struct optimistic_spin_queue to 32 bits, this patch reorders some fields in
struct rw_semaphore such that we can reduce the overhead of the rwsem structure
by 64 bits (on 64 bit systems).The extra overhead required for rwsem optimistic spinning would now be up
to 8 additional bytes instead of up to 16 bytes. Additionally, the size of
rwsem would now be more in line with mutexes.Signed-off-by: Jason Low
Signed-off-by: Peter Zijlstra
Cc: Scott Norton
Cc: "Paul E. McKenney"
Cc: Dave Chinner
Cc: Waiman Long
Cc: Davidlohr Bueso
Cc: Rik van Riel
Cc: Andrew Morton
Cc: "H. Peter Anvin"
Cc: Steven Rostedt
Cc: Tim Chen
Cc: Konrad Rzeszutek Wilk
Cc: Aswin Chandramouleeswaran
Cc: Linus Torvalds
Cc: Chris Mason
Cc: Josef Bacik
Link: http://lkml.kernel.org/r/1405358872-3732-6-git-send-email-jason.low2@hp.com
Signed-off-by: Ingo Molnar -
Currently, we initialize the osq lock by directly setting the lock's values. It
would be preferable if we use an init macro to do the initialization like we do
with other locks.This patch introduces and uses a macro and function for initializing the osq lock.
Signed-off-by: Jason Low
Signed-off-by: Peter Zijlstra
Cc: Scott Norton
Cc: "Paul E. McKenney"
Cc: Dave Chinner
Cc: Waiman Long
Cc: Davidlohr Bueso
Cc: Rik van Riel
Cc: Andrew Morton
Cc: "H. Peter Anvin"
Cc: Steven Rostedt
Cc: Tim Chen
Cc: Konrad Rzeszutek Wilk
Cc: Aswin Chandramouleeswaran
Cc: Linus Torvalds
Cc: Chris Mason
Cc: Josef Bacik
Link: http://lkml.kernel.org/r/1405358872-3732-4-git-send-email-jason.low2@hp.com
Signed-off-by: Ingo Molnar -
The cancellable MCS spinlock is currently used to queue threads that are
doing optimistic spinning. It uses per-cpu nodes, where a thread obtaining
the lock would access and queue the local node corresponding to the CPU that
it's running on. Currently, the cancellable MCS lock is implemented by using
pointers to these nodes.In this patch, instead of operating on pointers to the per-cpu nodes, we
store the CPU numbers in which the per-cpu nodes correspond to in atomic_t.
A similar concept is used with the qspinlock.By operating on the CPU # of the nodes using atomic_t instead of pointers
to those nodes, this can reduce the overhead of the cancellable MCS spinlock
by 32 bits (on 64 bit systems).Signed-off-by: Jason Low
Signed-off-by: Peter Zijlstra
Cc: Scott Norton
Cc: "Paul E. McKenney"
Cc: Dave Chinner
Cc: Waiman Long
Cc: Davidlohr Bueso
Cc: Rik van Riel
Cc: Andrew Morton
Cc: "H. Peter Anvin"
Cc: Steven Rostedt
Cc: Tim Chen
Cc: Konrad Rzeszutek Wilk
Cc: Aswin Chandramouleeswaran
Cc: Linus Torvalds
Cc: Chris Mason
Cc: Heiko Carstens
Cc: Josef Bacik
Link: http://lkml.kernel.org/r/1405358872-3732-3-git-send-email-jason.low2@hp.com
Signed-off-by: Ingo Molnar -
Currently, the per-cpu nodes structure for the cancellable MCS spinlock is
named "optimistic_spin_queue". However, in a follow up patch in the series
we will be introducing a new structure that serves as the new "handle" for
the lock. It would make more sense if that structure is named
"optimistic_spin_queue". Additionally, since the current use of the
"optimistic_spin_queue" structure are "nodes", it might be better if we
rename them to "node" anyway.This preparatory patch renames all current "optimistic_spin_queue"
to "optimistic_spin_node".Signed-off-by: Jason Low
Signed-off-by: Peter Zijlstra
Cc: Scott Norton
Cc: "Paul E. McKenney"
Cc: Dave Chinner
Cc: Waiman Long
Cc: Davidlohr Bueso
Cc: Rik van Riel
Cc: Andrew Morton
Cc: "H. Peter Anvin"
Cc: Steven Rostedt
Cc: Tim Chen
Cc: Konrad Rzeszutek Wilk
Cc: Aswin Chandramouleeswaran
Cc: Linus Torvalds
Cc: Chris Mason
Cc: Heiko Carstens
Cc: Josef Bacik
Link: http://lkml.kernel.org/r/1405358872-3732-2-git-send-email-jason.low2@hp.com
Signed-off-by: Ingo Molnar
05 Jun, 2014
2 commits
-
Optimistic spinning is only used by the xadd variant
of rw-semaphores. Make sure that we use the old version
of the __RWSEM_INITIALIZER macro for systems that rely
on the spinlock one, otherwise warnings can be triggered,
such as the following reported on an arm box:ipc/ipcns_notifier.c:22:8: warning: excess elements in struct initializer [enabled by default]
ipc/ipcns_notifier.c:22:8: warning: (near initialization for 'ipcns_chain.rwsem') [enabled by default]
ipc/ipcns_notifier.c:22:8: warning: excess elements in struct initializer [enabled by default]
ipc/ipcns_notifier.c:22:8: warning: (near initialization for 'ipcns_chain.rwsem') [enabled by default]Signed-off-by: Davidlohr Bueso
Signed-off-by: Peter Zijlstra
Cc: Tim Chen
Cc: Linus Torvalds
Cc: Paul McKenney
Cc: Michel Lespinasse
Cc: Peter Hurley
Cc: Alex Shi
Cc: Rik van Riel
Cc: Andrew Morton
Cc: Andrea Arcangeli
Cc: "H. Peter Anvin"
Cc: Jason Low
Cc: Andi Kleen
Cc: Chris Mason
Cc: Josef Bacik
Link: http://lkml.kernel.org/r/1400545677.6399.10.camel@buesod1.americas.hpqcorp.net
Signed-off-by: Ingo Molnar -
We have reached the point where our mutexes are quite fine tuned
for a number of situations. This includes the use of heuristics
and optimistic spinning, based on MCS locking techniques.Exclusive ownership of read-write semaphores are, conceptually,
just about the same as mutexes, making them close cousins. To
this end we need to make them both perform similarly, and
right now, rwsems are simply not up to it. This was discovered
by both reverting commit 4fc3f1d6 (mm/rmap, migration: Make
rmap_walk_anon() and try_to_unmap_anon() more scalable) and
similarly, converting some other mutexes (ie: i_mmap_mutex) to
rwsems. This creates a situation where users have to choose
between a rwsem and mutex taking into account this important
performance difference. Specifically, biggest difference between
both locks is when we fail to acquire a mutex in the fastpath,
optimistic spinning comes in to play and we can avoid a large
amount of unnecessary sleeping and overhead of moving tasks in
and out of wait queue. Rwsems do not have such logic.This patch, based on the work from Tim Chen and I, adds support
for write-side optimistic spinning when the lock is contended.
It also includes support for the recently added cancelable MCS
locking for adaptive spinning. Note that is is only applicable
to the xadd method, and the spinlock rwsem variant remains intact.Allowing optimistic spinning before putting the writer on the wait
queue reduces wait queue contention and provided greater chance
for the rwsem to get acquired. With these changes, rwsem is on par
with mutex. The performance benefits can be seen on a number of
workloads. For instance, on a 8 socket, 80 core 64bit Westmere box,
aim7 shows the following improvements in throughput:+--------------+---------------------+-----------------+
| Workload | throughput-increase | number of users |
+--------------+---------------------+-----------------+
| alltests | 20% | >1000 |
| custom | 27%, 60% | 10-100, >1000 |
| high_systime | 36%, 30% | >100, >1000 |
| shared | 58%, 29% | 10-100, >1000 |
+--------------+---------------------+-----------------+There was also improvement on smaller systems, such as a quad-core
x86-64 laptop running a 30Gb PostgreSQL (pgbench) workload for up
to +60% in throughput for over 50 clients. Additionally, benefits
were also noticed in exim (mail server) workloads. Furthermore, no
performance regression have been seen at all.Based-on-work-from: Tim Chen
Signed-off-by: Davidlohr Bueso
[peterz: rej fixup due to comment patches, sched/rt.h header]
Signed-off-by: Peter Zijlstra
Cc: Alex Shi
Cc: Andi Kleen
Cc: Michel Lespinasse
Cc: Rik van Riel
Cc: Peter Hurley
Cc: "Paul E.McKenney"
Cc: Jason Low
Cc: Aswin Chandramouleeswaran
Cc: Andrew Morton
Cc: Linus Torvalds
Cc: "Scott J Norton"
Cc: Andrea Arcangeli
Cc: Chris Mason
Cc: Josef Bacik
Link: http://lkml.kernel.org/r/1399055055.6275.15.camel@buesod1.americas.hpqcorp.net
Signed-off-by: Ingo Molnar
29 Jan, 2014
1 commit
-
Btrfs needs a simple way to know if it needs to let go of it's read lock on a
rwsem. Introduce rwsem_is_contended to check to see if there are any waiters on
this rwsem currently. This is just a hueristic, it is meant to be light and not
100% accurate and called by somebody already holding on to the rwsem in either
read or write. Thanks,Signed-off-by: Josef Bacik
Signed-off-by: Chris Mason
Acked-by: Ingo Molnar
24 Mar, 2013
1 commit
-
This reverts commit 11b80f459adaf91a712f95e7734a17655a36bf30.
Bcache needs rw semaphores for cache coherency in writeback mode -
writes have to take a read lock on a per cache device rw sem, and
release it when the bio completes.But since this is for bios it's naturally not in the context of the
process that originally took the lock.Signed-off-by: Kent Overstreet
CC: Christoph Hellwig
CC: David Howells
17 Jan, 2013
1 commit
-
Commit 1b963c81b145 ("lockdep, rwsem: provide down_write_nest_lock()")
contains a bug in a codepath when CONFIG_DEBUG_LOCK_ALLOC is disabled,
which causes down_read() to be called instead of down_write() by mistake
on such configurations. Fix that.Reported-and-tested-by: Andrew Clayton
Reported-and-tested-by: Zlatko Calusic
Signed-off-by: Jiri Kosina
Reviewed-by: Rik van Riel
Signed-off-by: Linus Torvalds
12 Jan, 2013
1 commit
-
down_write_nest_lock() provides a means to annotate locking scenario
where an outer lock is guaranteed to serialize the order nested locks
are being acquired.This is analogoue to already existing mutex_lock_nest_lock() and
spin_lock_nest_lock().Signed-off-by: Jiri Kosina
Cc: Rik van Riel
Cc: Ingo Molnar
Cc: Peter Zijlstra
Cc: Mel Gorman
Tested-by: Sedat Dilek
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
29 Mar, 2012
1 commit
-
Remove all #inclusions of asm/system.h preparatory to splitting and killing
it. Performed with the following command:perl -p -i -e 's!^#\s*include\s*.*\n!!' `grep -Irl '^#\s*include\s*' *`
Signed-off-by: David Howells
13 Sep, 2011
1 commit
-
There is no reason to allow the lock protecting rwsems (the
ownerless variant) to be preemptible on -rt. Convert it to raw.In mainline this change documents the low level nature of
the lock - otherwise there's no functional difference. Lockdep
and Sparse checking will work as usual.Signed-off-by: Thomas Gleixner
Signed-off-by: Ingo Molnar
27 Jul, 2011
1 commit
-
This allows us to move duplicated code in
(atomic_inc_not_zero() for now) toSigned-off-by: Arun Sharma
Reviewed-by: Eric Dumazet
Cc: Ingo Molnar
Cc: David Miller
Cc: Eric Dumazet
Acked-by: Mike Frysinger
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
21 Jul, 2011
1 commit
-
Now that the last users is gone these can be removed.
Signed-off-by: Christoph Hellwig
Signed-off-by: Al Viro
27 Jan, 2011
6 commits
-
Peter Zijlstra pointed out, that the only user of asmregparm (x86) is
compiling the kernel already with -mregparm=3. So the annotation of
the rwsem functions is redundant. Remove it.Signed-off-by: Thomas Gleixner
Cc: Peter Zijlstra
Cc: David Howells
Cc: Benjamin Herrenschmidt
Cc: Matt Turner
Cc: Tony Luck
Cc: Heiko Carstens
Cc: Paul Mundt
Cc: David Miller
Cc: Chris Zankel
LKML-Reference:
Signed-off-by: Thomas Gleixner -
All architecture specific rwsem headers carry the same function
prototypes. Just x86 adds asmregparm, which is an empty define on all
other architectures. S390 has a stale rwsem_downgrade_write()
prototype.Remove the duplicates and add the prototypes to linux/rwsem.h
Signed-off-by: Thomas Gleixner
Cc: Peter Zijlstra
Cc: David Howells
Cc: Benjamin Herrenschmidt
Cc: Richard Henderson
Acked-by: Tony Luck
Acked-by: Heiko Carstens
Cc: Paul Mundt
Acked-by: David Miller
Cc: Chris Zankel
LKML-Reference:
Signed-off-by: Thomas Gleixner -
Instead of having the same implementation in each architecture, move
it to linux/rwsem.h and remove the duplicates. It's unlikely that an
arch will ever implement something different, but we can deal with
that when it happens.Signed-off-by: Thomas Gleixner
Cc: Peter Zijlstra
Cc: David Howells
Cc: Benjamin Herrenschmidt
Cc: Matt Turner
Acked-by: Tony Luck
Acked-by: Heiko Carstens
Cc: Paul Mundt
Acked-by: David Miller
Cc: Chris Zankel
LKML-Reference:
Signed-off-by: Thomas Gleixner -
The rwsem initializers and related macros and functions are mostly the
same. Some of them lack the lockdep initializer, but having it in
place does not matter for architectures which do not support lockdep.powerpc, sparc, x86: No functional change
sh, s390: Removes the duplicate init_rwsem (inline and #define)
alpha, ia64, xtensa: Use the lockdep capable init function in
lib/rwsem.c which is just uninlining the init
function for the LOCKDEP=n caseSigned-off-by: Thomas Gleixner
Cc: Peter Zijlstra
Cc: David Howells
Cc: Benjamin Herrenschmidt
Cc: Matt Turner
Acked-by: Tony Luck
Acked-by: Heiko Carstens
Cc: Paul Mundt
Acked-by: David Miller
Cc: Chris Zankel
LKML-Reference: -
The difference between these declarations is the data type of the
count member and the lack of lockdep in some architectures/long is equivivalent to signed long and the #ifdef guarded dep_map
member does not hurt anyone.Signed-off-by: Thomas Gleixner
Cc: Peter Zijlstra
Cc: David Howells
Cc: Benjamin Herrenschmidt
Cc: Matt Turner
Acked-by: Tony Luck
Acked-by: Heiko Carstens
Cc: Paul Mundt
Acked-by: David Miller
Cc: Chris Zankel
LKML-Reference:
Signed-off-by: Thomas Gleixner -
All rwsem implementations include the same headers. Include them from
include/linux/rwsem.hSigned-off-by: Thomas Gleixner
Cc: Peter Zijlstra
Cc: David Howells
Cc: Benjamin Herrenschmidt
Cc: Matt Turner
Acked-by: Tony Luck
Acked-by: Heiko Carstens
Cc: Paul Mundt
Acked-by: David Miller
Cc: Chris Zankel
LKML-Reference:
30 Apr, 2008
1 commit
-
Remove the "#ifdef __KERNEL__" tests from unexported header files in
linux/include whose entire contents are wrapped in that preprocessor
test.Signed-off-by: Robert P. J. Day
Cc: David Woodhouse
Cc: Sam Ravnborg
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
11 Jul, 2006
1 commit
-
Add more documentation to rwsem.h.
Signed-off-by: Ingo Molnar
Cc: David Howells
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
04 Jul, 2006
2 commits
-
Use the lock validator framework to prove rwsem locking correctness.
Signed-off-by: Ingo Molnar
Signed-off-by: Arjan van de Ven
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
Clean up rwsems.
Signed-off-by: Ingo Molnar
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
26 Apr, 2006
1 commit
-
Signed-off-by: David Woodhouse
17 Apr, 2005
1 commit
-
Initial git repository build. I'm not bothering with the full history,
even though we have it. We can create a separate "historical" git
archive of that later if we want to, and in the meantime it's about
3.2GB when imported into git - space that would just make the early
git days unnecessarily complicated, when we don't have a lot of good
infrastructure for it.Let it rip!
