06 Jan, 2017
1 commit
-
commit 4d1f0fb096aedea7bb5489af93498a82e467c480 upstream.
NMI handler doesn't call set_irq_regs(), it's set only by normal IRQ.
Thus get_irq_regs() returns NULL or stale registers snapshot with IP/SP
pointing to the code interrupted by IRQ which was interrupted by NMI.
NULL isn't a problem: in this case watchdog calls dump_stack() and
prints full stack trace including NMI. But if we're stuck in IRQ
handler then NMI watchlog will print stack trace without IRQ part at
all.This patch uses registers snapshot passed into NMI handler as arguments:
these registers point exactly to the instruction interrupted by NMI.Fixes: 55537871ef66 ("kernel/watchdog.c: perform all-CPU backtrace in case of hard lockup")
Link: http://lkml.kernel.org/r/146771764784.86724.6006627197118544150.stgit@buzz
Signed-off-by: Konstantin Khlebnikov
Cc: Jiri Kosina
Cc: Ulrich Obergfell
Cc: Aaron Tomlin
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
Signed-off-by: Greg Kroah-Hartman
18 Mar, 2016
1 commit
-
While working on a script to restore all sysctl params before a series of
tests I found that writing any value into the
/proc/sys/kernel/{nmi_watchdog,soft_watchdog,watchdog,watchdog_thresh}
causes them to call proc_watchdog_update().NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.There doesn't appear to be a reason for doing this work every time a write
occurs, so only do it when the values change.Signed-off-by: Josh Hunt
Acked-by: Don Zickus
Reviewed-by: Aaron Tomlin
Cc: Ulrich Obergfell
Cc: [4.1.x+]
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
12 Jan, 2016
1 commit
-
Pull workqueue update from Tejun Heo:
"Workqueue changes for v4.5. One cleanup patch and three to improve
the debuggability.Workqueue now has a stall detector which dumps workqueue state if any
worker pool hasn't made forward progress over a certain amount of time
(30s by default) and also triggers a warning if a workqueue which can
be used in memory reclaim path tries to wait on something which can't
be.These should make workqueue hangs a lot easier to debug."
* 'for-4.5' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq:
workqueue: simplify the apply_workqueue_attrs_locked()
workqueue: implement lockup detector
watchdog: introduce touch_softlockup_watchdog_sched()
workqueue: warn if memory reclaim tries to flush !WQ_MEM_RECLAIM workqueue
19 Dec, 2015
2 commits
-
Currently, kdump_nmi_shootdown_cpus(), a subroutine of crash_kexec(),
sends an NMI IPI to CPUs which haven't called panic() to stop them,
save their register information and do some cleanups for crash dumping.
However, if such a CPU is infinitely looping in NMI context, we fail to
save its register information into the crash dump.For example, this can happen when unknown NMIs are broadcast to all
CPUs as follows:CPU 0 CPU 1
=========================== ==========================
receive an unknown NMI
unknown_nmi_error()
panic() receive an unknown NMI
spin_trylock(&panic_lock) unknown_nmi_error()
crash_kexec() panic()
spin_trylock(&panic_lock)
panic_smp_self_stop()
infinite loop
kdump_nmi_shootdown_cpus()
issue NMI IPI -----------> blocked until IRET
infinite loop...Here, since CPU 1 is in NMI context, the second NMI from CPU 0 is
blocked until CPU 1 executes IRET. However, CPU 1 never executes IRET,
so the NMI is not handled and the callback function to save registers is
never called.In practice, this can happen on some servers which broadcast NMIs to all
CPUs when the NMI button is pushed.To save registers in this case, we need to:
a) Return from NMI handler instead of looping infinitely
or
b) Call the callback function directly from the infinite loopInherently, a) is risky because NMI is also used to prevent corrupted
data from being propagated to devices. So, we chose b).This patch does the following:
1. Move the infinite looping of CPUs which haven't called panic() in NMI
context (actually done by panic_smp_self_stop()) outside of panic() to
enable us to refer pt_regs. Please note that panic_smp_self_stop() is
still used for normal context.2. Call a callback of kdump_nmi_shootdown_cpus() directly to save
registers and do some cleanups after setting waiting_for_crash_ipi which
is used for counting down the number of CPUs which handled the callbackSigned-off-by: Hidehiro Kawai
Acked-by: Michal Hocko
Cc: Aaron Tomlin
Cc: Andrew Morton
Cc: Andy Lutomirski
Cc: Baoquan He
Cc: Chris Metcalf
Cc: Dave Young
Cc: David Hildenbrand
Cc: Don Zickus
Cc: Eric Biederman
Cc: Frederic Weisbecker
Cc: Gobinda Charan Maji
Cc: HATAYAMA Daisuke
Cc: Hidehiro Kawai
Cc: "H. Peter Anvin"
Cc: Ingo Molnar
Cc: Javi Merino
Cc: Jiang Liu
Cc: Jonathan Corbet
Cc: kexec@lists.infradead.org
Cc: linux-doc@vger.kernel.org
Cc: lkml
Cc: Masami Hiramatsu
Cc: Michal Nazarewicz
Cc: Nicolas Iooss
Cc: Oleg Nesterov
Cc: Peter Zijlstra
Cc: Prarit Bhargava
Cc: Rasmus Villemoes
Cc: Seth Jennings
Cc: Stefan Lippers-Hollmann
Cc: Steven Rostedt
Cc: Thomas Gleixner
Cc: Ulrich Obergfell
Cc: Vitaly Kuznetsov
Cc: Vivek Goyal
Cc: Yasuaki Ishimatsu
Link: http://lkml.kernel.org/r/20151210014628.25437.75256.stgit@softrs
[ Cleanup comments, fixup formatting. ]
Signed-off-by: Borislav Petkov
Signed-off-by: Thomas Gleixner -
If panic on NMI happens just after panic() on the same CPU, panic() is
recursively called. Kernel stalls, as a result, after failing to acquire
panic_lock.To avoid this problem, don't call panic() in NMI context if we've
already entered panic().For that, introduce nmi_panic() macro to reduce code duplication. In
the case of panic on NMI, don't return from NMI handlers if another CPU
already panicked.Signed-off-by: Hidehiro Kawai
Acked-by: Michal Hocko
Cc: Aaron Tomlin
Cc: Andrew Morton
Cc: Andy Lutomirski
Cc: Baoquan He
Cc: Chris Metcalf
Cc: David Hildenbrand
Cc: Don Zickus
Cc: "Eric W. Biederman"
Cc: Frederic Weisbecker
Cc: Gobinda Charan Maji
Cc: HATAYAMA Daisuke
Cc: "H. Peter Anvin"
Cc: Ingo Molnar
Cc: Javi Merino
Cc: Jonathan Corbet
Cc: kexec@lists.infradead.org
Cc: linux-doc@vger.kernel.org
Cc: lkml
Cc: Masami Hiramatsu
Cc: Michal Nazarewicz
Cc: Nicolas Iooss
Cc: Peter Zijlstra
Cc: Prarit Bhargava
Cc: Rasmus Villemoes
Cc: Rusty Russell
Cc: Seth Jennings
Cc: Steven Rostedt
Cc: Thomas Gleixner
Cc: Ulrich Obergfell
Cc: Vitaly Kuznetsov
Cc: Vivek Goyal
Link: http://lkml.kernel.org/r/20151210014626.25437.13302.stgit@softrs
[ Cleanup comments, fixup formatting. ]
Signed-off-by: Borislav Petkov
Signed-off-by: Thomas Gleixner
09 Dec, 2015
2 commits
-
Workqueue stalls can happen from a variety of usage bugs such as
missing WQ_MEM_RECLAIM flag or concurrency managed work item
indefinitely staying RUNNING. These stalls can be extremely difficult
to hunt down because the usual warning mechanisms can't detect
workqueue stalls and the internal state is pretty opaque.To alleviate the situation, this patch implements workqueue lockup
detector. It periodically monitors all worker_pools periodically and,
if any pool failed to make forward progress longer than the threshold
duration, triggers warning and dumps workqueue state as follows.BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 31s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x0
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=17/256
pending: monkey_wrench_fn, e1000_watchdog, cache_reap, vmstat_shepherd, release_one_tty, release_one_tty, release_one_tty, release_one_tty, release_one_tty, release_one_tty, release_one_tty, release_one_tty, release_one_tty, release_one_tty, release_one_tty, release_one_tty, cgroup_release_agent
workqueue events_power_efficient: flags=0x80
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256
pending: check_lifetime, neigh_periodic_work
workqueue cgroup_pidlist_destroy: flags=0x0
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1
pending: cgroup_pidlist_destroy_work_fn
...The detection mechanism is controller through kernel parameter
workqueue.watchdog_thresh and can be updated at runtime through the
sysfs module parameter file.v2: Decoupled from softlockup control knobs.
Signed-off-by: Tejun Heo
Acked-by: Don Zickus
Cc: Ulrich Obergfell
Cc: Michal Hocko
Cc: Chris Mason
Cc: Andrew Morton -
touch_softlockup_watchdog() is used to tell watchdog that scheduler
stall is expected. One group of usage is from paths where the task
may not be able to yield for a long time such as performing slow PIO
to finicky device and coming out of suspend. The other is to account
for scheduler and timer going idle.For scheduler softlockup detection, there's no reason to distinguish
the two cases; however, workqueue lockup detector is planned and it
can use the same signals from the former group while the latter would
spuriously prevent detection. This patch introduces a new function
touch_softlockup_watchdog_sched() and convert the latter group to call
it instead. For now, it just calls touch_softlockup_watchdog() and
there's no functional difference.Signed-off-by: Tejun Heo
Cc: Ulrich Obergfell
Cc: Ingo Molnar
Cc: Peter Zijlstra
Cc: Thomas Gleixner
Cc: Andrew Morton
06 Nov, 2015
12 commits
-
Theoretically it is possible that the watchdog timer expires right at the
time when a user sets 'watchdog_thresh' to zero (note: this disables the
lockup detectors). In this scenario, the is_softlockup() function - which
is called by the timer - could produce a false positive.Fix this by checking the current value of 'watchdog_thresh'.
Signed-off-by: Ulrich Obergfell
Acked-by: Don Zickus
Reviewed-by: Aaron Tomlin
Cc: Ulrich Obergfell
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
watchdog_{park|unpark}_threads() are now called in code paths that protect
themselves against CPU hotplug, so {get|put}_online_cpus() calls are
redundant and can be removed.Signed-off-by: Ulrich Obergfell
Acked-by: Don Zickus
Reviewed-by: Aaron Tomlin
Cc: Ulrich Obergfell
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
The handler functions for watchdog parameters in /proc/sys/kernel do not
protect themselves against races with CPU hotplug. Hence, theoretically
it is possible that a new watchdog thread is started on a hotplugged CPU
while a parameter is being modified, and the thread could thus use a
parameter value that is 'in transition'.For example, if 'watchdog_thresh' is being set to zero (note: this
disables the lockup detectors) the thread would erroneously use the value
zero as the sample period.To avoid such races and to keep the /proc handler code consistent,
call
{get|put}_online_cpus() in proc_watchdog_common()
{get|put}_online_cpus() in proc_watchdog_thresh()
{get|put}_online_cpus() in proc_watchdog_cpumask()Signed-off-by: Ulrich Obergfell
Acked-by: Don Zickus
Reviewed-by: Aaron Tomlin
Cc: Ulrich Obergfell
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
The lockup detector suspend/resume interface that was introduced by
commit 8c073d27d7ad ("watchdog: introduce watchdog_suspend() and
watchdog_resume()") does not protect itself against races with CPU
hotplug. Hence, theoretically it is possible that a new watchdog thread
is started on a hotplugged CPU while the lockup detector is suspended,
and the thread could thus interfere unexpectedly with the code that
requested to suspend the lockup detector.Avoid the race by calling
get_online_cpus() in lockup_detector_suspend()
put_online_cpus() in lockup_detector_resume()Signed-off-by: Ulrich Obergfell
Acked-by: Don Zickus
Reviewed-by: Aaron Tomlin
Cc: Ulrich Obergfell
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
The only way to enable a hardlockup to panic the machine is to set
'nmi_watchdog=panic' on the kernel command line.This makes it awkward for end users and folks who want to run automate
tests (like myself).Mimic the softlockup_panic knob and create a /proc/sys/kernel/hardlockup_panic
knob.Signed-off-by: Don Zickus
Cc: Ulrich Obergfell
Acked-by: Jiri Kosina
Reviewed-by: Aaron Tomlin
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
In many cases of hardlockup reports, it's actually not possible to know
why it triggered, because the CPU that got stuck is usually waiting on a
resource (with IRQs disabled) in posession of some other CPU is holding.IOW, we are often looking at the stacktrace of the victim and not the
actual offender.Introduce sysctl / cmdline parameter that makes it possible to have
hardlockup detector perform all-CPU backtrace.Signed-off-by: Jiri Kosina
Reviewed-by: Aaron Tomlin
Cc: Ulrich Obergfell
Acked-by: Don Zickus
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
If kthread_park() returns an error, watchdog_park_threads() should not
blindly 'roll back' the already parked threads to the unparked state.
Instead leave it up to the callers to handle such errors appropriately in
their context. For example, it is redundant to unpark the threads if the
lockup detectors will soon be disabled by the callers anyway.Signed-off-by: Ulrich Obergfell
Reviewed-by: Aaron Tomlin
Acked-by: Don Zickus
Cc: Ulrich Obergfell
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
lockup_detector_suspend() now handles errors from watchdog_park_threads().
Signed-off-by: Ulrich Obergfell
Reviewed-by: Aaron Tomlin
Acked-by: Don Zickus
Cc: Ulrich Obergfell
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
update_watchdog_all_cpus() now passes errors from watchdog_park_threads()
up to functions in the call chain. This allows watchdog_enable_all_cpus()
and proc_watchdog_update() to handle such errors too.Signed-off-by: Ulrich Obergfell
Reviewed-by: Aaron Tomlin
Acked-by: Don Zickus
Cc: Ulrich Obergfell
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
Move watchdog_disable_all_cpus() outside of the ifdef so that it is
available if CONFIG_SYSCTL is not defined. This is preparation for
"watchdog: implement error handling in update_watchdog_all_cpus() and
callers".Signed-off-by: Ulrich Obergfell
Reviewed-by: Aaron Tomlin
Acked-by: Don Zickus
Cc: Ulrich Obergfell
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
The original watchdog_park_threads() function that was introduced by
commit 81a4beef91ba ("watchdog: introduce watchdog_park_threads() and
watchdog_unpark_threads()") takes a very simple approach to handle
errors returned by kthread_park(): It attempts to roll back all watchdog
threads to the unparked state. However, this may be undesired behaviour
from the perspective of the caller which may want to handle errors as
appropriate in its specific context. Currently, there are two possible
call chains:- watchdog suspend/resume interface
lockup_detector_suspend
watchdog_park_threads- write to parameters in /proc/sys/kernel
proc_watchdog_update
watchdog_enable_all_cpus
update_watchdog_all_cpus
watchdog_park_threadsInstead of 'blindly' attempting to unpark the watchdog threads if a
kthread_park() call fails, the new approach is to disable the lockup
detectors in the above call chains. Failure becomes visible to the user
as follows:- error messages from lockup_detector_suspend()
or watchdog_enable_all_cpus()- the state that can be read from /proc/sys/kernel/watchdog_enabled
- the 'write' system call in the latter call chain returns an error
I did not experience kthread_park() failures in practice, I used some
instrumentation to fake error returns from kthread_park() in order to test
the patches.This patch (of 5):
Restore the previous value of watchdog_thresh _and_ sample_period if
proc_watchdog_update() returns an error. The variables must be consistent
to avoid false positives of the lockup detectors.Signed-off-by: Ulrich Obergfell
Reviewed-by: Aaron Tomlin
Acked-by: Don Zickus
Cc: Ulrich Obergfell
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
Make is_hardlockup return bool to improve readability due to this
particular function only using either one or zero as its return value.No functional change.
Signed-off-by: Yaowei Bai
Reviewed-by: Aaron Tomlin
Acked-by: Don Zickus
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
05 Sep, 2015
8 commits
-
Rename watchdog_suspend() to lockup_detector_suspend() and
watchdog_resume() to lockup_detector_resume() to avoid confusion with the
watchdog subsystem and to be consistent with the existing name
lockup_detector_init().Also provide comment blocks to explain the watchdog_running and
watchdog_suspended variables and their relationship.Signed-off-by: Ulrich Obergfell
Reviewed-by: Aaron Tomlin
Cc: Guenter Roeck
Cc: Don Zickus
Cc: Ulrich Obergfell
Cc: Jiri Olsa
Cc: Michal Hocko
Cc: Stephane Eranian
Cc: Chris Metcalf
Cc: Frederic Weisbecker
Cc: Peter Zijlstra
Cc: Ingo Molnar
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
Remove watchdog_nmi_disable_all() and watchdog_nmi_enable_all() since
these functions are no longer needed. If a subsystem has a need to
deactivate the watchdog temporarily, it should utilize the
watchdog_suspend() and watchdog_resume() functions.[akpm@linux-foundation.org: fix build with CONFIG_LOCKUP_DETECTOR=m]
Signed-off-by: Ulrich Obergfell
Reviewed-by: Aaron Tomlin
Cc: Guenter Roeck
Cc: Don Zickus
Cc: Ulrich Obergfell
Cc: Jiri Olsa
Cc: Michal Hocko
Cc: Stephane Eranian
Cc: Chris Metcalf
Cc: Frederic Weisbecker
Cc: Peter Zijlstra
Cc: Ingo Molnar
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
Remove update_watchdog() and restart_watchdog_hrtimer() since these
functions are no longer needed. Changes of parameters such as the sample
period are honored at the time when the watchdog threads are being
unparked.Signed-off-by: Ulrich Obergfell
Reviewed-by: Aaron Tomlin
Cc: Guenter Roeck
Cc: Don Zickus
Cc: Ulrich Obergfell
Cc: Jiri Olsa
Cc: Michal Hocko
Cc: Stephane Eranian
Cc: Chris Metcalf
Cc: Frederic Weisbecker
Cc: Peter Zijlstra
Cc: Ingo Molnar
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
This interface can be utilized to deactivate the hard and soft lockup
detector temporarily. Callers are expected to minimize the duration of
deactivation. Multiple deactivations are allowed to occur in parallel but
should be rare in practice.[akpm@linux-foundation.org: remove unneeded static initialization]
Signed-off-by: Ulrich Obergfell
Reviewed-by: Aaron Tomlin
Cc: Guenter Roeck
Cc: Don Zickus
Cc: Ulrich Obergfell
Cc: Jiri Olsa
Cc: Michal Hocko
Cc: Stephane Eranian
Cc: Chris Metcalf
Cc: Frederic Weisbecker
Cc: Peter Zijlstra
Cc: Ingo Molnar
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
Originally watchdog_nmi_enable(cpu) and watchdog_nmi_disable(cpu) were
only called in watchdog thread context. However, the following commits
utilize these functions outside of watchdog thread context too.commit 9809b18fcf6b8d8ec4d3643677345907e6b50eca
Author: Michal Hocko
Date: Tue Sep 24 15:27:30 2013 -0700watchdog: update watchdog_thresh properly
commit b3738d29323344da3017a91010530cf3a58590fc
Author: Stephane Eranian
Date: Mon Nov 17 20:07:03 2014 +0100watchdog: Add watchdog enable/disable all functions
Hence, it is now possible that these functions execute concurrently with
the same 'cpu' argument. This concurrency is problematic because per-cpu
'watchdog_ev' can be accessed/modified without adequate synchronization.The patch series aims to address the above problem. However, instead of
introducing locks to protect per-cpu 'watchdog_ev' a different approach is
taken: Invoke these functions by parking and unparking the watchdog
threads (to ensure they are always called in watchdog thread context).static struct smp_hotplug_thread watchdog_threads = {
...
.park = watchdog_disable, // calls watchdog_nmi_disable()
.unpark = watchdog_enable, // calls watchdog_nmi_enable()
};Both previously mentioned commits call these functions in a similar way
and thus in principle contain some duplicate code. The patch series also
avoids this duplication by providing a commonly usable mechanism.- Patch 1/4 introduces the watchdog_{park|unpark}_threads functions that
park/unpark all watchdog threads specified in 'watchdog_cpumask'. They
are intended to be called inside of kernel/watchdog.c only.- Patch 2/4 introduces the watchdog_{suspend|resume} functions which can
be utilized by external callers to deactivate the hard and soft lockup
detector temporarily.- Patch 3/4 utilizes watchdog_{park|unpark}_threads to replace some code
that was introduced by commit 9809b18fcf6b8d8ec4d3643677345907e6b50eca.- Patch 4/4 utilizes watchdog_{suspend|resume} to replace some code that
was introduced by commit b3738d29323344da3017a91010530cf3a58590fc.A few corner cases should be mentioned here for completeness.
- kthread_park() of watchdog/N could hang if cpu N is already locked up.
However, if watchdog is enabled the lockup will be detected anyway.- kthread_unpark() of watchdog/N could hang if cpu N got locked up after
kthread_park(). The occurrence of this scenario should be _very_ rare
in practice, in particular because it is not expected that temporary
deactivation will happen frequently, and if it happens at all it is
expected that the duration of deactivation will be short.This patch (of 4): introduce watchdog_park_threads() and watchdog_unpark_threads()
These functions are intended to be used only from inside kernel/watchdog.c
to park/unpark all watchdog threads that are specified in
watchdog_cpumask.Signed-off-by: Ulrich Obergfell
Reviewed-by: Aaron Tomlin
Cc: Guenter Roeck
Cc: Don Zickus
Cc: Ulrich Obergfell
Cc: Jiri Olsa
Cc: Michal Hocko
Cc: Stephane Eranian
Cc: Chris Metcalf
Cc: Frederic Weisbecker
Cc: Peter Zijlstra
Cc: Ingo Molnar
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
The kernel's NMI watchdog has nothing to do with the watchdog subsystem.
Its header declarations should be in linux/nmi.h, not linux/watchdog.h.The code provided two sets of dummy functions if HARDLOCKUP_DETECTOR is
not configured, one in the include file and one in kernel/watchdog.c.
Remove the dummy functions from kernel/watchdog.c and use those from the
include file.Signed-off-by: Guenter Roeck
Cc: Stephane Eranian
Cc: Peter Zijlstra (Intel)
Cc: Ingo Molnar
Cc: Don Zickus
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
housekeeping_mask gathers all the CPUs that aren't part of the nohz_full
set. This is exactly what we want the watchdog to be affine to without
the need to use complicated cpumask operations.Signed-off-by: Frederic Weisbecker
Reviewed-by: Chris Metcalf
Cc: Thomas Gleixner
Cc: Chris Metcalf
Cc: Don Zickus
Cc: Peter Zijlstra
Cc: Ulrich Obergfell
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
It makes the registration cheaper and simpler for the smpboot per-cpu
kthread users that don't need to always update the cpumask after threads
creation.[sfr@canb.auug.org.au: fix for allow passing the cpumask on per-cpu thread registration]
Signed-off-by: Frederic Weisbecker
Reviewed-by: Chris Metcalf
Reviewed-by: Thomas Gleixner
Cc: Chris Metcalf
Cc: Don Zickus
Cc: Peter Zijlstra
Cc: Ulrich Obergfell
Signed-off-by: Stephen Rothwell
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
25 Jun, 2015
1 commit
-
Change the default behavior of watchdog so it only runs on the
housekeeping cores when nohz_full is enabled at build and boot time.
Allow modifying the set of cores the watchdog is currently running on
with a new kernel.watchdog_cpumask sysctl.In the current system, the watchdog subsystem runs a periodic timer that
schedules the watchdog kthread to run. However, nohz_full cores are
designed to allow userspace application code running on those cores to
have 100% access to the CPU. So the watchdog system prevents the
nohz_full application code from being able to run the way it wants to,
thus the motivation to suppress the watchdog on nohz_full cores, which
this patchset provides by default.However, if we disable the watchdog globally, then the housekeeping
cores can't benefit from the watchdog functionality. So we allow
disabling it only on some cores. See Documentation/lockup-watchdogs.txt
for more information.[jhubbard@nvidia.com: fix a watchdog crash in some configurations]
Signed-off-by: Chris Metcalf
Acked-by: Don Zickus
Cc: Ingo Molnar
Cc: Ulrich Obergfell
Cc: Thomas Gleixner
Cc: Peter Zijlstra
Cc: Frederic Weisbecker
Signed-off-by: John Hubbard
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
20 May, 2015
1 commit
-
Commit ab992dc38f9a ("watchdog: Fix merge 'conflict'") has introduced an
obvious deadlock because of a typo. watchdog_proc_mutex should be
unlocked on exit.Thanks to Miroslav Benes who was staring at the code with me and noticed
this.Signed-off-by: Michal Hocko
Duh-by: Peter Zijlstra
Signed-off-by: Linus Torvalds
19 May, 2015
1 commit
-
Two watchdog changes that came through different trees had a non
conflicting conflict, that is, one changed the semantics of a variable
but no actual code conflict happened. So the merge appeared fine, but
the resulting code did not behave as expected.Commit 195daf665a62 ("watchdog: enable the new user interface of the
watchdog mechanism") changes the semantics of watchdog_user_enabled,
which thereafter is only used by the functions introduced by
b3738d293233 ("watchdog: Add watchdog enable/disable all functions").There further appears to be a distinct lack of serialization between
setting and using watchdog_enabled, so perhaps we should wrap the
{en,dis}able_all() things in watchdog_proc_mutex.This patch fixes a s2r failure reported by Michal; which I cannot
readily explain. But this does make the code internally consistent
again.Reported-and-tested-by: Michal Hocko
Signed-off-by: Peter Zijlstra (Intel)
Signed-off-by: Linus Torvalds
15 Apr, 2015
10 commits
-
Merge first patchbomb from Andrew Morton:
- arch/sh updates
- ocfs2 updates
- kernel/watchdog feature
- about half of mm/
* emailed patches from Andrew Morton : (122 commits)
Documentation: update arch list in the 'memtest' entry
Kconfig: memtest: update number of test patterns up to 17
arm: add support for memtest
arm64: add support for memtest
memtest: use phys_addr_t for physical addresses
mm: move memtest under mm
mm, hugetlb: abort __get_user_pages if current has been oom killed
mm, mempool: do not allow atomic resizing
memcg: print cgroup information when system panics due to panic_on_oom
mm: numa: remove migrate_ratelimited
mm: fold arch_randomize_brk into ARCH_HAS_ELF_RANDOMIZE
mm: split ET_DYN ASLR from mmap ASLR
s390: redefine randomize_et_dyn for ELF_ET_DYN_BASE
mm: expose arch_mmap_rnd when available
s390: standardize mmap_rnd() usage
powerpc: standardize mmap_rnd() usage
mips: extract logic for mmap_rnd()
arm64: standardize mmap_rnd() usage
x86: standardize mmap_rnd() usage
arm: factor out mmap ASLR into mmap_rnd
... -
Have kvm_guest_init() use hardlockup_detector_disable() instead of
watchdog_enable_hardlockup_detector(false).Remove the watchdog_hardlockup_detector_is_enabled() and the
watchdog_enable_hardlockup_detector() function which are no longer needed.Signed-off-by: Ulrich Obergfell
Signed-off-by: Don Zickus
Cc: Ingo Molnar
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
Rename the update_timers*() functions to update_watchdog*().
Remove the boolean argument from watchdog_enable_all_cpus() because
update_watchdog_all_cpus() is now a generic function to change the run
state of the lockup detectors and to have the lockup detectors use a new
sample period.Signed-off-by: Ulrich Obergfell
Signed-off-by: Don Zickus
Cc: Ingo Molnar
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
With the current user interface of the watchdog mechanism it is only
possible to disable or enable both lockup detectors at the same time.
This series introduces new kernel parameters and changes the semantics of
some existing kernel parameters, so that the hard lockup detector and the
soft lockup detector can be disabled or enabled individually. With this
series applied, the user interface is as follows.- parameters in /proc/sys/kernel
. soft_watchdog
This is a new parameter to control and examine the run state of
the soft lockup detector.. nmi_watchdog
The semantics of this parameter have changed. It can now be used
to control and examine the run state of the hard lockup detector.. watchdog
This parameter is still available to control the run state of both
lockup detectors at the same time. If this parameter is examined,
it shows the logical OR of soft_watchdog and nmi_watchdog.. watchdog_thresh
The semantics of this parameter are not affected by the patch.- kernel command line parameters
. nosoftlockup
The semantics of this parameter have changed. It can now be used
to disable the soft lockup detector at boot time.. nmi_watchdog=0 or nmi_watchdog=1
Disable or enable the hard lockup detector at boot time. The patch
introduces '=1' as a new option.. nowatchdog
The semantics of this parameter are not affected by the patch. It
is still available to disable both lockup detectors at boot time.Also, remove the proc_dowatchdog() function which is no longer needed.
[dzickus@redhat.com: wrote changelog]
[dzickus@redhat.com: update documentation for kernel params and sysctl]
Signed-off-by: Ulrich Obergfell
Signed-off-by: Don Zickus
Cc: Ingo Molnar
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
If watchdog_nmi_enable() fails to set up the hardware perf event of one
CPU, the entire hard lockup detector is deemed unreliable. Hence, disable
the hard lockup detector and shut down the hardware perf events on all
CPUs.[dzickus@redhat.com: update comments to explain some code]
Signed-off-by: Ulrich Obergfell
Signed-off-by: Don Zickus
Cc: Ingo Molnar
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
Separate handlers for each watchdog parameter in /proc/sys/kernel replace
the proc_dowatchdog() function. Three of those handlers merely call
proc_watchdog_common() with one different argument.Signed-off-by: Ulrich Obergfell
Signed-off-by: Don Zickus
Cc: Ingo Molnar
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
Three of four handlers for the watchdog parameters in /proc/sys/kernel
essentially have to do the same thing.if the parameter is being read {
return the state of the corresponding bit(s) in 'watchdog_enabled'
} else {
set/clear the state of the corresponding bit(s) in 'watchdog_enabled'
update the run state of the lockup detector(s)
}Hence, introduce a common function that can be called by those handlers.
The callers pass a 'bit mask' to this function to indicate which bit(s)
should be set/cleared in 'watchdog_enabled'.This function handles an uncommon race with watchdog_nmi_enable() where a
concurrent update of 'watchdog_enabled' is possible. We use 'cmpxchg' to
detect the concurrency. [This avoids introducing a new spinlock or a
mutex to synchronize updates of 'watchdog_enabled'. Using the same lock
or mutex in watchdog thread context and in system call context needs to be
considered carefully because it can make the code prone to deadlock
situations in connection with parking/unparking the watchdog threads.]Signed-off-by: Ulrich Obergfell
Signed-off-by: Don Zickus
Cc: Ingo Molnar
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
This series removes proc_dowatchdog(). Since multiple new functions need
the 'watchdog_proc_mutex' to serialize access to the watchdog parameters
in /proc/sys/kernel, move the mutex outside of any function.Signed-off-by: Ulrich Obergfell
Signed-off-by: Don Zickus
Cc: Ingo Molnar
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
This series introduces a separate handler for each watchdog parameter in
/proc/sys/kernel. The separate handlers need a common function that they
can call to update the run state of the lockup detectors, or to have the
lockup detectors use a new sample period.Signed-off-by: Ulrich Obergfell
Signed-off-by: Don Zickus
Cc: Ingo Molnar
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
The hardlockup and softockup had always been tied together. Due to the
request of KVM folks, they had a need to have one enabled but not the
other. Internally rework the code to split things apart more cleanly.There is a bunch of churn here, but the end result should be code that
should be easier to maintain and fix without knowing the internals of what
is going on.This patch (of 9):
Introduce new definitions and variables to separate the user interface in
/proc/sys/kernel from the internal run state of the lockup detectors. The
internal run state is represented by two bits in a new variable that is
named 'watchdog_enabled'. This helps simplify the code, for example:- In order to check if any of the two lockup detectors is enabled,
it is sufficient to check if 'watchdog_enabled' is not zero.- In order to enable/disable one or both lockup detectors,
it is sufficient to set/clear one or both bits in 'watchdog_enabled'.- Concurrent updates of 'watchdog_enabled' need not be synchronized via
a spinlock or a mutex. Updates can either be atomic or concurrency can
be detected by using 'cmpxchg'.Signed-off-by: Ulrich Obergfell
Signed-off-by: Don Zickus
Cc: Ingo Molnar
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds