23 May, 2011
4 commits
-
Before the conversion of the NMI watchdog to perf event, the
watchdog timeout was 5 seconds. Now it is 60 seconds. For my
particular application, netbooks, 5 seconds was a better
timeout. With a short timeout, we catch faults earlier and are
able to send back a panic. With a 60 second timeout, the user is
unlikely to wait and will instead hit the power button, causing
us to lose the panic info.This change configures the NMI period to watchdog_thresh and
sets the softlockup_thresh to watchdog_thresh * 2. In addition,
watchdog_thresh was reduced to 10 seconds as suggested by Ingo
Molnar.Signed-off-by: Mandeep Singh Baines
Cc: Marcin Slusarz
Cc: Don Zickus
Cc: Peter Zijlstra
Cc: Frederic Weisbecker
Link: http://lkml.kernel.org/r/1306127423-3347-4-git-send-email-msb@chromium.org
Signed-off-by: Ingo Molnar
LKML-Reference: -
This restores the previous behavior of softlock_thresh.
Currently, setting watchdog_thresh to zero causes the watchdog
kthreads to consume a lot of CPU.In addition, the logic of proc_dowatchdog_thresh and
proc_dowatchdog_enabled has been factored into proc_dowatchdog.Signed-off-by: Mandeep Singh Baines
Cc: Marcin Slusarz
Cc: Don Zickus
Cc: Peter Zijlstra
Cc: Frederic Weisbecker
Link: http://lkml.kernel.org/r/1306127423-3347-3-git-send-email-msb@chromium.org
Signed-off-by: Ingo Molnar
LKML-Reference: -
Don't take any action on an unsuccessful write to /proc.
Signed-off-by: Mandeep Singh Baines
Cc: Marcin Slusarz
Cc: Don Zickus
Cc: Peter Zijlstra
Cc: Frederic Weisbecker
Link: http://lkml.kernel.org/r/1306127423-3347-2-git-send-email-msb@chromium.org
Signed-off-by: Ingo Molnar -
In get_sample_period(), softlockup_thresh is integer divided by
5 before the multiplication by NSEC_PER_SEC. This results in
softlockup_thresh being rounded down to the nearest integer
multiple of 5.For example, a softlockup_thresh of 4 rounds down to 0.
Signed-off-by: Mandeep Singh Baines
Cc: Marcin Slusarz
Cc: Don Zickus
Cc: Peter Zijlstra
Cc: Frederic Weisbecker
Link: http://lkml.kernel.org/r/1306127423-3347-1-git-send-email-msb@chromium.org
Signed-off-by: Ingo Molnar
29 Apr, 2011
1 commit
-
In corner cases where softlockup watchdog is not setup successfully, the
relevant nmi perf event for hardlockup watchdog could be disabled, then
the status of the underlying hardware remains unchanged.Also, if the kthread doesn't start then the hrtimer won't run and the
hardlockup detector will falsely fire.Signed-off-by: Hillf Danton
Signed-off-by: Don Zickus
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
23 Mar, 2011
2 commits
-
This patch addresses a couple of problems. One was the case when the
hardlockup failed to start, it also failed to start the softlockup. There
were valid cases when the hardlockup shouldn't start and that shouldn't
block the softlockup (no lapic, bios controls perf counters).The second problem was when the hardlockup failed to start on boxes (from
a no lapic or bios controlled perf counter case), it reported failure to
the cpu notifier chain. This blocked the notifier from continuing to
start other more critical pieces of cpu bring-up (in our case based on a
2.6.32 fork, it was the mce). As a result, during soft cpu online/offline
testing, the system would panic when a cpu was offlined because the cpu
notifier would succeed in processing a watchdog disable cpu event and
would panic in the mce case as a result of un-initialized variables from a
never executed cpu up event.I realized the hardlockup/softlockup cases are really just debugging aids
and should never impede the progress of a cpu up/down event. Therefore I
modified the code to always return NOTIFY_OK and instead rely on printks
to inform the user of problems.Signed-off-by: Don Zickus
Acked-by: Peter Zijlstra
Reviewed-by: WANG Cong
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
When a cpu is considered stuck, instead of limping along and just printing
a warning, it is sometimes preferred to just panic, let kdump capture the
vmcore and reboot. This gets the machine back into a stable state quickly
while saving the info that got it into a stuck state to begin with.Add a Kconfig option to allow users to set the hardlockup to panic
by default. Also add in a 'nmi_watchdog=nopanic' to override this.[akpm@linux-foundation.org: fix strncmp length]
Signed-off-by: Don Zickus
Acked-by: Peter Zijlstra
Reviewed-by: WANG Cong
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
10 Feb, 2011
1 commit
-
During boot if the hardlockup detector fails to initialize, it
complains very loudly. Some failures should be expected under
certain situations, ie no lapics, or resource in-use. Tone
those error messages down a bit. Keep the rest at a high level.Reported-by: Paul Bolle
Tested-by: Paul Bolle
Signed-off-by: Don Zickus
Cc: Peter Zijlstra
LKML-Reference:
Signed-off-by: Ingo Molnar
31 Jan, 2011
3 commits
-
Signed-off-by: Marcin Slusarz
[ add {}'s to fix a warning ]
Signed-off-by: Don Zickus
Cc: Peter Zijlstra
Cc: Frederic Weisbecker
Cc:
LKML-Reference:
Signed-off-by: Ingo Molnar -
If it was not possible to enable watchdog for any cpu, switch
watchdog_enabled back to 0, because it's visible via
kernel.watchdog sysctl.Signed-off-by: Marcin Slusarz
Signed-off-by: Don Zickus
Cc: Peter Zijlstra
Cc: Frederic Weisbecker
Cc:
LKML-Reference:
Signed-off-by: Ingo Molnar -
Passing nowatchdog to kernel disables 2 things: creation of
watchdog threads AND initialization of percpu watchdog_hrtimer.
As hrtimers are initialized only at boot it's not possible to
enable watchdog later - for me all watchdog threads started to
eat 100% of CPU time, but they could just crash.Additionally, even if these threads would start properly,
watchdog_disable_all_cpus was guarded by no_watchdog check, so
you couldn't disable watchdog.To fix this, remove no_watchdog variable and use already
existing watchdog_enabled variable.Signed-off-by: Marcin Slusarz
[ removed another no_watchdog instance ]
Signed-off-by: Don Zickus
Cc: Stephane Eranian
Cc: Peter Zijlstra
Cc: Frederic Weisbecker
Cc:
LKML-Reference:
Signed-off-by: Ingo Molnar
08 Jan, 2011
1 commit
-
* 'for-2.6.38' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu: (30 commits)
gameport: use this_cpu_read instead of lookup
x86: udelay: Use this_cpu_read to avoid address calculation
x86: Use this_cpu_inc_return for nmi counter
x86: Replace uses of current_cpu_data with this_cpu ops
x86: Use this_cpu_ops to optimize code
vmstat: User per cpu atomics to avoid interrupt disable / enable
irq_work: Use per cpu atomics instead of regular atomics
cpuops: Use cmpxchg for xchg to avoid lock semantics
x86: this_cpu_cmpxchg and this_cpu_xchg operations
percpu: Generic this_cpu_cmpxchg() and this_cpu_xchg support
percpu,x86: relocate this_cpu_add_return() and friends
connector: Use this_cpu operations
xen: Use this_cpu_inc_return
taskstats: Use this_cpu_ops
random: Use this_cpu_inc_return
fs: Use this_cpu_inc_return in buffer.c
highmem: Use this_cpu_xx_return() operations
vmstat: Use this_cpu_inc_return for vm statistics
x86: Support for this_cpu_add, sub, dec, inc_return
percpu: Generic support for this_cpu_add, sub, dec, inc_return
...Fixed up conflicts: in arch/x86/kernel/{apic/nmi.c, apic/x2apic_uv_x.c, process.c}
as per Tejun.
07 Jan, 2011
1 commit
-
…/git/tip/linux-2.6-tip
* 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip: (30 commits)
sched: Change wait_for_completion_*_timeout() to return a signed long
sched, autogroup: Fix reference leak
sched, autogroup: Fix potential access to freed memory
sched: Remove redundant CONFIG_CGROUP_SCHED ifdef
sched: Fix interactivity bug by charging unaccounted run-time on entity re-weight
sched: Move periodic share updates to entity_tick()
printk: Use this_cpu_{read|write} api on printk_pending
sched: Make pushable_tasks CONFIG_SMP dependant
sched: Add 'autogroup' scheduling feature: automated per session task groups
sched: Fix unregister_fair_sched_group()
sched: Remove unused argument dest_cpu to migrate_task()
mutexes, sched: Introduce arch_mutex_cpu_relax()
sched: Add some clock info to sched_debug
cpu: Remove incorrect BUG_ON
cpu: Remove unused variable
sched: Fix UP build breakage
sched: Make task dump print all 15 chars of proc comm
sched: Update tg->shares after cpu.shares write
sched: Allow update_cfs_load() to update global load
sched: Implement demand based update_cfs_load()
...
05 Jan, 2011
2 commits
-
Merge reason: Add the final .37 tree.
Signed-off-by: Ingo Molnar
-
Merge reason: Merge the final .37 tree.
Signed-off-by: Ingo Molnar
03 Jan, 2011
1 commit
-
The error message 'NMI watchdog failed to create perf event...'
does not make it clear that this is a fatal error for the
watchdog. It also currently prints the error value as a
pointer, rather than extracting the error code with PTR_ERR().
Fix that.Add a note to the description of the 'nowatchdog' kernel
parameter to associate it with this message.Reported-by: Cesare Leonardi
Signed-off-by: Ben Hutchings
Cc: 599368@bugs.debian.org
Cc: 608138@bugs.debian.org
Cc: Don Zickus
Cc: Frederic Weisbecker
Cc: # .37.x and later
LKML-Reference:
Signed-off-by: Ingo Molnar
17 Dec, 2010
1 commit
-
__get_cpu_var() can be replaced with this_cpu_read and will then use a
single read instruction with implied address calculation to access the
correct per cpu instance.However, the address of a per cpu variable passed to __this_cpu_read()
cannot be determined (since it's an implied address conversion through
segment prefixes). Therefore apply this only to uses of __get_cpu_var
where the address of the variable is not used.Cc: Pekka Enberg
Cc: Hugh Dickins
Cc: Thomas Gleixner
Acked-by: H. Peter Anvin
Signed-off-by: Christoph Lameter
Signed-off-by: Tejun Heo
10 Dec, 2010
1 commit
-
Originally adapted from Huang Ying's patch which moved the
unknown_nmi_panic to the traps.c file. Because the old nmi
watchdog was deleted before this change happened, the
unknown_nmi_panic sysctl was lost. This re-adds it.Also, the nmi_watchdog sysctl was re-implemented and its
documentation updated accordingly.Patch-inspired-by: Huang Ying
Signed-off-by: Don Zickus
Reviewed-by: Cyrill Gorcunov
Acked-by: Yinghai Lu
Cc: fweisbec@gmail.com
LKML-Reference:
Signed-off-by: Ingo Molnar
26 Nov, 2010
1 commit
-
The perf hardware pmu got initialized at various points in the boot,
some before early_initcall() some after (notably arch_initcall).The problem is that the NMI lockup detector is ran from early_initcall()
and expects the hardware pmu to be present.Sanitize this by moving all architecture hardware pmu implementations to
initialize at early_initcall() and move the lockup detector to an explicit
initcall right after that.Cc: paulus
Cc: davem
Cc: Michael Cree
Cc: Deng-Cheng Zhu
Acked-by: Paul Mundt
Acked-by: Will Deacon
Signed-off-by: Peter Zijlstra
LKML-Reference:
Signed-off-by: Ingo Molnar
18 Nov, 2010
1 commit
-
Merge reason: Move to a .37-rc base.
Signed-off-by: Ingo Molnar
06 Nov, 2010
1 commit
-
Commit d9ca07a05ce1 ("watchdog: Avoid kernel crash when disabling
watchdog") introduces a section mismatch.Now that we reference no_watchdog from non-__init code it can no longer
be __initdata.Signed-off-by: David Daney
Cc: Stephane Eranian
Cc: Peter Zijlstra
Cc: Ingo Molnar
Signed-off-by: Linus Torvalds
23 Oct, 2010
1 commit
-
Andrew Morton pointed out almost all sched_setscheduler() callers are
using fixed parameters and can be converted to static. It reduces runtime
memory use a little.Signed-off-by: KOSAKI Motohiro
Reported-by: Andrew Morton
Acked-by: James Morris
Cc: Ingo Molnar
Cc: Steven Rostedt
Signed-off-by: Andrew Morton
Signed-off-by: Thomas Gleixner
Signed-off-by: Ingo Molnar
22 Oct, 2010
1 commit
-
…git/tip/linux-2.6-tip
* 'perf-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip: (163 commits)
tracing: Fix compile issue for trace_sched_wakeup.c
[S390] hardirq: remove pointless header file includes
[IA64] Move local_softirq_pending() definition
perf, powerpc: Fix power_pmu_event_init to not use event->ctx
ftrace: Remove recursion between recordmcount and scripts/mod/empty
jump_label: Add COND_STMT(), reducer wrappery
perf: Optimize sw events
perf: Use jump_labels to optimize the scheduler hooks
jump_label: Add atomic_t interface
jump_label: Use more consistent naming
perf, hw_breakpoint: Fix crash in hw_breakpoint creation
perf: Find task before event alloc
perf: Fix task refcount bugs
perf: Fix group moving
irq_work: Add generic hardirq context callbacks
perf_events: Fix transaction recovery in group_sched_in()
perf_events: Fix bogus AMD64 generic TLB events
perf_events: Fix bogus context time tracking
tracing: Remove parent recording in latency tracer graph options
tracing: Use one prologue for the preempt irqs off tracer function tracers
...
12 Oct, 2010
1 commit
-
Reason: Pull in the semaphore related changes
Signed-off-by: Thomas Gleixner
15 Sep, 2010
2 commits
-
The kernel perf event creation path shouldn't use find_task_by_vpid()
because a vpid exists in a specific namespace. find_task_by_vpid() uses
current's pid namespace which isn't always the correct namespace to use
for the vpid in all the places perf_event_create_kernel_counter() (and
thus find_get_context()) is called.The goal is to clean up pid namespace handling and prevent bugs like:
https://bugzilla.kernel.org/show_bug.cgi?id=17281
Instead of using pids switch find_get_context() to use task struct
pointers directly. The syscall is responsible for resolving the pid to
a task struct. This moves the pid namespace resolution into the syscall
much like every other syscall that takes pid parameters.Signed-off-by: Matt Helsley
Signed-off-by: Peter Zijlstra
Cc: Robin Green
Cc: Prasad
Cc: Arnaldo Carvalho de Melo
Cc: Steven Rostedt
Cc: Will Deacon
Cc: Mahesh Salgaonkar
LKML-Reference:
Signed-off-by: Ingo Molnar -
In case you boot with the watchdog disabled, i.e., nowatchdog, then,
if you try to disable it via /proc/sys/kernel/watchdog, you get
a kernel crash. The reason is that you are trying to cancel a hrtimer
which has never been initialized.This patch fixes this by skipping execution of
watchdog_disable_all_cpus() when the watchdog is marked
disabled from boot.Signed-off-by: Stephane Eranian
Signed-off-by: Peter Zijlstra
LKML-Reference:
Signed-off-by: Ingo Molnar
10 Sep, 2010
1 commit
-
Merge reason: Pick up pending fixes before applying dependent new changes.
Signed-off-by: Ingo Molnar
01 Sep, 2010
3 commits
-
During my rewrite, the semantics of touch_nmi_watchdog and
touch_softlockup_watchdog changed enough to break some drivers
(mostly over preemptable regions).These are cases where long delays on one CPU (due to
print_delay for example) can cause long delays on other
CPUs - so we must 'touch' the nmi_watchdog flag of those
other CPUs as well.This change brings those touch_*_watchdog() functions back in line
with to how they used to work.Signed-off-by: Don Zickus
Acked-by: Cyrill Gorcunov
Cc: peterz@infradead.org
Cc: fweisbec@gmail.com
LKML-Reference:
Signed-off-by: Ingo Molnar -
The panic notifer in lockup_detector just set did_panic to 1.
But did_panic is not used anywhere so we can just remove it.Signed-off-by: Akinobu Mita
Cc: peterz@infradead.org
Cc: gorcunov@gmail.com
Cc: fweisbec@gmail.com
LKML-Reference:
Signed-off-by: Don Zickus
Signed-off-by: Ingo Molnar -
By the commit e6bde73b07edeb703d4c89c1daabc09c303de11f
("cpu-hotplug: return better errno on cpu hotplug failure"),
the cpu notifier can return encapsulate errno value, resulting
in more meaningful error codes for CPU hotplug failures.This converts the cpu notifier to return encapsulate errno value
for the lockup_detector as well.Signed-off-by: Akinobu Mita
Cc: peterz@infradead.org
Cc: gorcunov@gmail.com
Cc: fweisbec@gmail.com
LKML-Reference:
Signed-off-by: Don Zickus
Signed-off-by: Ingo Molnar
23 Aug, 2010
1 commit
-
Stephane reported that when the machine locks up, the regular ticks,
which are responsible to resetting the throttle count, stop too.Hence the NMI watchdog can end up being throttled before it reports on
the locked up state, and we end up being sad..Cure this by having the watchdog overflow reset its own throttle count.
Reported-by: Stephane Eranian
Tested-by: Stephane Eranian
Cc: Don Zickus
Cc: Frederic Weisbecker
Signed-off-by: Peter Zijlstra
LKML-Reference:
Signed-off-by: Ingo Molnar
20 Aug, 2010
1 commit
-
watchdog_overflow_callback() is only used in kernel/watchdog.c.
Signed-off-by: Lin Ming
Cc: Peter Zijlstra
Cc: Don Zickus
LKML-Reference:
Signed-off-by: Ingo Molnar
07 Jul, 2010
1 commit
-
Variable on the stack is not initialized to zero, do it
explicitly.This bug was found by a compiler warning:
kernel/watchdog.c:463: warning: 'result' may be used uninitialized in this function
Signed-off-by: Kulikov Vasiliy
Acked-by: Don Zickus
Cc: Frederic Weisbecker
Cc: Peter Zijlstra
Cc: Arnaldo Carvalho de Melo
Cc: Frederic Weisbecker
Cc: Paul Mackerras
Cc: Mike Galbraith
Cc: Steven Rostedt
LKML-Reference:
Signed-off-by: Ingo Molnar
19 May, 2010
1 commit
-
Just a bunch of conversions as suggested by Frederic W.
__get_cpu_var() provides preemption disabled checks.Plus it gives more readability as it makes it obvious
we are dealing locally now with these vars.Signed-off-by: Don Zickus
Cc: Ingo Molnar
Cc: Peter Zijlstra
Cc: Don Zickus
Cc: Cyrill Gorcunov
LKML-Reference:
Signed-off-by: Frederic Weisbecker
16 May, 2010
2 commits
-
Combining the softlockup and hardlockup code causes watchdog.c
to build even without the hardlockup detection support.So if an arch, that has the previous and the new nmi watchdog
implementations cohabiting, wants to know if the generic one
is in use, CONFIG_LOCKUP_DETECTOR is not a reliable check.
We need to use CONFIG_HARDLOCKUP_DETECTOR instead.Fixes:
kernel/built-in.o: In function `touch_nmi_watchdog':
(.text+0x449bc): multiple definition of `touch_nmi_watchdog'
arch/sparc/kernel/built-in.o:(.text+0x11b28): first defined hereSigned-off-by: Don Zickus
Cc: Ingo Molnar
Cc: Peter Zijlstra
Cc: Don Zickus
Cc: Cyrill Gorcunov
LKML-Reference:
[ use CONFIG_HARDLOCKUP_DETECTOR instead of CONFIG_PERF_EVENTS_NMI]
Signed-off-by: Frederic Weisbecker -
This new config is deemed to simplify even more the lockup detector
dependencies and can make it easier to bring a smooth sorting
between archs that support the new generic lockup detector and those
that still have their own, especially for those that are in the
middle of this migration.Instead of checking whether we have CONFIG_LOCKUP_DETECTOR +
CONFIG_PERF_EVENTS_NMI each time an arch wants to know if it needs
to build its own lockup detector, take a shortcut with this new
config. It is enabled only if the hardlockup detection part of
the whole lockup detector is on.Signed-off-by: Frederic Weisbecker
Cc: Ingo Molnar
Cc: Peter Zijlstra
Cc: Don Zickus
Cc: Cyrill Gorcunov
13 May, 2010
4 commits
-
There are modules that rely on it:
ERROR: "touch_softlockup_watchdog" [drivers/video/nvidia/nvidiafb.ko] undefined!
Cc: Frederic Weisbecker
Cc: Don Zickus
Cc: Peter Zijlstra
Cc: Cyrill Gorcunov
LKML-Reference:
Signed-off-by: Ingo Molnar -
When I combined the nmi_watchdog (hardlockup) and softlockup code, I
also combined the paths the touch_watchdog and touch_nmi_watchdog took.
This may not be the best idea as pointed out by Frederic W., that the
touch_watchdog case probably should not reset the hardlockup count.Therefore the patch below falls back to the previous idea of keeping
the touch_nmi_watchdog a superset of the touch_watchdog case.Signed-off-by: Don Zickus
Cc: Ingo Molnar
Cc: Peter Zijlstra
Cc: Cyrill Gorcunov
Cc: Eric Paris
Cc: Randy Dunlap
LKML-Reference:
Signed-off-by: Frederic Weisbecker -
Just some code cleanup to make touch_softlockup clearer and remove the
softlockup_tick function as it is no longer needed.Also remove the /proc softlockup_thres call as it has been changed to
watchdog_thres.Signed-off-by: Don Zickus
Cc: Ingo Molnar
Cc: Peter Zijlstra
Cc: Cyrill Gorcunov
Cc: Eric Paris
Cc: Randy Dunlap
LKML-Reference:
Signed-off-by: Frederic Weisbecker -
The new nmi_watchdog (which uses the perf event subsystem) is very
similar in structure to the softlockup detector. Using Ingo's
suggestion, I combined the two functionalities into one file:
kernel/watchdog.c.Now both the nmi_watchdog (or hardlockup detector) and softlockup
detector sit on top of the perf event subsystem, which is run every
60 seconds or so to see if there are any lockups.To detect hardlockups, cpus not responding to interrupts, I
implemented an hrtimer that runs 5 times for every perf event
overflow event. If that stops counting on a cpu, then the cpu is
most likely in trouble.To detect softlockups, tasks not yielding to the scheduler, I used the
previous kthread idea that now gets kicked every time the hrtimer fires.
If the kthread isn't being scheduled neither is anyone else and the
warning is printed to the console.I tested this on x86_64 and both the softlockup and hardlockup paths
work.V2:
- cleaned up the Kconfig and softlockup combination
- surrounded hardlockup cases with #ifdef CONFIG_PERF_EVENTS_NMI
- seperated out the softlockup case from perf event subsystem
- re-arranged the enabling/disabling nmi watchdog from proc space
- added cpumasks for hardlockup failure cases
- removed fallback to soft events if no PMU exists for hard eventsV3:
- comment cleanups
- drop support for older softlockup code
- per_cpu cleanups
- completely remove software clock base hardlockup detector
- use per_cpu masking on hard/soft lockup detection
- #ifdef cleanups
- rename config option NMI_WATCHDOG to LOCKUP_DETECTOR
- documentation additionsV4:
- documentation fixes
- convert per_cpu to __get_cpu_var
- powerpc compile fixesV5:
- split apart warn flags for hard and soft lockupsTODO:
- figure out how to make an arch-agnostic clock2cycles call
(if possible) to feed into perf events as a sample period[fweisbec: merged conflict patch]
Signed-off-by: Don Zickus
Cc: Ingo Molnar
Cc: Peter Zijlstra
Cc: Cyrill Gorcunov
Cc: Eric Paris
Cc: Randy Dunlap
LKML-Reference:
Signed-off-by: Frederic Weisbecker