05 Feb, 2013

1 commit

  • Conflicts:
    kernel/irq_work.c

    Add support for printk in full dynticks CPU.

    * Don't stop tick with irq works pending. This
    fix is generally useful and concerns archs that
    can't raise self IPIs.

    * Flush irq works before CPU offlining.

    * Introduce "lazy" irq works that can wait for the
    next tick to be executed, unless it's stopped.

    * Implement klogd wake up using irq work. This
    removes the ad-hoc printk_tick()/printk_needs_cpu()
    hooks and make it working even in dynticks mode.

    Signed-off-by: Frederic Weisbecker

    Frederic Weisbecker
     

04 Feb, 2013

1 commit

  • As no one is using the return value of irq_work_queue(),
    so it is better to just make it void.

    Signed-off-by: anish kumar
    Acked-by: Steven Rostedt
    [ Fix stale comments, remove now unnecessary __irq_work_queue() intermediate function ]
    Signed-off-by: Frederic Weisbecker
    Link: http://lkml.kernel.org/r/1359925703-24304-1-git-send-email-fweisbec@gmail.com
    Signed-off-by: Ingo Molnar

    anish kumar
     

18 Nov, 2012

4 commits

  • On irq work initialization, let the user choose to define it
    as "lazy" or not. "Lazy" means that we don't want to send
    an IPI (provided the arch can anyway) when we enqueue this
    work but we rather prefer to wait for the next timer tick
    to execute our work if possible.

    This is going to be a benefit for non-urgent enqueuers
    (like printk in the future) that may prefer not to raise
    an IPI storm in case of frequent enqueuing on short periods
    of time.

    Signed-off-by: Frederic Weisbecker
    Acked-by: Steven Rostedt
    Cc: Peter Zijlstra
    Cc: Thomas Gleixner
    Cc: Ingo Molnar
    Cc: Andrew Morton
    Cc: Paul Gortmaker

    Frederic Weisbecker
     
  • If we are in nohz and there's still irq_work to be done when the idle
    task is about to go offline, give a nasty warning. Everything should
    have been flushed from the CPU_DYING notifier already. Further attempts
    to enqueue an irq_work are buggy because irqs are disabled by
    __cpu_disable(). The best we can do is to report the issue to the user.

    Signed-off-by: Steven Rostedt
    Cc: Peter Zijlstra
    Cc: Thomas Gleixner
    Cc: Ingo Molnar
    Cc: Andrew Morton
    Cc: Paul Gortmaker
    Signed-off-by: Frederic Weisbecker

    Steven Rostedt
     
  • In order not to offline a CPU with pending irq works, flush the
    queue from CPU_DYING. The notifier is called by stop_machine on
    the CPU that is going down. The code will not be called from irq context
    (so things like get_irq_regs() wont work) but I'm not sure what the
    requirements are for irq_work in that regard (Peter?). But irqs are
    disabled and the CPU is about to go offline. Might as well flush the work.

    Signed-off-by: Steven Rostedt
    Cc: Peter Zijlstra
    Cc: Thomas Gleixner
    Cc: Ingo Molnar
    Cc: Andrew Morton
    Cc: Paul Gortmaker
    Signed-off-by: Frederic Weisbecker

    Steven Rostedt
     
  • Don't stop the tick if we have pending irq works on the
    queue, otherwise if the arch can't raise self-IPIs, we may not
    find an opportunity to execute the pending works for a while.

    Signed-off-by: Frederic Weisbecker
    Acked-by: Steven Rostedt
    Cc: Peter Zijlstra
    Cc: Thomas Gleixner
    Cc: Ingo Molnar
    Cc: Andrew Morton
    Cc: Paul Gortmaker

    Frederic Weisbecker
     

15 Nov, 2012

2 commits

  • Work claiming wants to be SMP-safe.

    And by the time we try to claim a work, if it is already executing
    concurrently on another CPU, we want to succeed the claiming and queue
    the work again because the other CPU may have missed the data we wanted
    to handle in our work if it's about to complete there.

    This scenario is summarized below:

    CPU 1 CPU 2
    ----- -----
    (flags = 0)
    cmpxchg(flags, 0, IRQ_WORK_FLAGS)
    (flags = 3)
    [...]
    xchg(flags, IRQ_WORK_BUSY)
    (flags = 2)
    func()
    if (flags & IRQ_WORK_PENDING)
    (not true)
    cmpxchg(flags, flags, IRQ_WORK_FLAGS)
    (flags = 3)
    [...]
    cmpxchg(flags, IRQ_WORK_BUSY, 0);
    (fail, pending on CPU 2)

    This state machine is synchronized using [cmp]xchg() on the flags.
    As such, the early IRQ_WORK_PENDING check in CPU 2 above is racy.
    By the time we check it, we may be dealing with a stale value because
    we aren't using an atomic accessor. As a result, CPU 2 may "see"
    that the work is still pending on another CPU while it may be
    actually completing the work function exection already, leaving
    our data unprocessed.

    To fix this, we start by speculating about the value we wish to be
    in the work->flags but we only make any conclusion after the value
    returned by the cmpxchg() call that either claims the work or let
    the current owner handle the pending work for us.

    Changelog-heavily-inspired-by: Steven Rostedt
    Signed-off-by: Frederic Weisbecker
    Acked-by: Steven Rostedt
    Cc: Peter Zijlstra
    Cc: Ingo Molnar
    Cc: Thomas Gleixner
    Cc: Andrew Morton
    Cc: Paul Gortmaker
    Cc: Anish Kumar

    Frederic Weisbecker
     
  • The IRQ_WORK_BUSY flag is set right before we execute the
    work. Once this flag value is set, the work enters a
    claimable state again.

    So if we have specific data to compute in our work, we ensure it's
    either handled by another CPU or locally by enqueuing the work again.
    This state machine is guanranteed by atomic operations on the flags.

    So when we set IRQ_WORK_BUSY without using an xchg-like operation,
    we break this guarantee as in the following summarized scenario:

    CPU 1 CPU 2
    ----- -----
    (flags = 0)
    old_flags = flags;
    (flags = 0)
    cmpxchg(flags, old_flags,
    old_flags | IRQ_WORK_FLAGS)
    (flags = 3)
    [...]
    flags = IRQ_WORK_BUSY
    (flags = 2)
    func()
    (sees flags = 3)
    cmpxchg(flags, old_flags,
    old_flags | IRQ_WORK_FLAGS)
    (give up)

    cmpxchg(flags, 2, 0);
    (flags = 0)

    CPU 1 claims a work and executes it, so it sets IRQ_WORK_BUSY and
    the work is again in a claimable state. Now CPU 2 has new data to process
    and try to claim that work but it may see a stale value of the flags
    and think the work is still pending somewhere that will handle our data.
    This is because CPU 1 doesn't set IRQ_WORK_BUSY atomically.

    As a result, the data expected to be handle by CPU 2 won't get handled.

    To fix this, use xchg() to set IRQ_WORK_BUSY, this way we ensure the CPU 2
    will see the correct value with cmpxchg() using the expected ordering.

    Changelog-heavily-inspired-by: Steven Rostedt
    Signed-off-by: Frederic Weisbecker
    Acked-by: Steven Rostedt
    Cc: Peter Zijlstra
    Cc: Ingo Molnar
    Cc: Thomas Gleixner
    Cc: Andrew Morton
    Cc: Paul Gortmaker
    Cc: Anish Kumar

    Frederic Weisbecker
     

14 Apr, 2012

1 commit


02 Apr, 2012

1 commit

  • Builds of the MIPS platform ip32_defconfig fails as of commit
    0195c00244dc ("Merge tag 'split-asm_system_h ...") because MIPS xchg()
    macro uses BUILD_BUG_ON and it was moved in commit b81947c646bf
    ("Disintegrate asm/system.h for MIPS").

    The root cause is that the system.h split wasn't tested on a baseline
    with commit 6c03438edeb5 ("kernel.h: doesn't explicitly use bug.h, so
    don't include it.")

    Since this file uses BUG code in several other places besides the xchg
    call, simply make the inclusion explicit.

    Signed-off-by: Paul Gortmaker
    Acked-by: David Howells
    Signed-off-by: Linus Torvalds

    Paul Gortmaker
     

31 Oct, 2011

2 commits

  • Up until now, this file was getting percpu.h because nearly every
    file was implicitly getting module.h (and all its sub-includes).
    But we want to clean that up, so call out percpu.h explicitly.
    Otherwise we'll get things like this on an ARM build:

    kernel/irq_work.c:48: error: expected declaration specifiers or '...' before 'irq_work_list'
    kernel/irq_work.c:48: warning: type defaults to 'int' in declaration of 'DEFINE_PER_CPU'

    The same thing was happening for builds on ARM for asm/processor.h

    kernel/irq_work.c: In function 'irq_work_sync':
    kernel/irq_work.c:166: error: implicit declaration of function 'cpu_relax'

    Signed-off-by: Paul Gortmaker

    Paul Gortmaker
     
  • The changed files were only including linux/module.h for the
    EXPORT_SYMBOL infrastructure, and nothing else. Revector them
    onto the isolated export header for faster compile times.

    Nothing to see here but a whole lot of instances of:

    -#include
    +#include

    This commit is only changing the kernel dir; next targets
    will probably be mm, fs, the arch dirs, etc.

    Signed-off-by: Paul Gortmaker

    Paul Gortmaker
     

04 Oct, 2011

2 commits

  • So we don't have to expose the struct list_node member.

    Cc: Huang Ying
    Cc: Andrew Morton
    Signed-off-by: Peter Zijlstra
    Link: http://lkml.kernel.org/r/1315836348.26517.41.camel@twins
    Signed-off-by: Ingo Molnar

    Peter Zijlstra
     
  • Use llist in irq_work instead of the lock-less linked list
    implementation in irq_work to avoid the code duplication.

    Signed-off-by: Huang Ying
    Signed-off-by: Peter Zijlstra
    Link: http://lkml.kernel.org/r/1315461646-1379-6-git-send-email-ying.huang@intel.com
    Signed-off-by: Ingo Molnar

    Huang Ying
     

18 Dec, 2010

1 commit

  • The irq work queue is a per cpu object and it is sufficient for
    synchronization if per cpu atomics are used. Doing so simplifies
    the code and reduces the overhead of the code.

    Before:

    christoph@linux-2.6$ size kernel/irq_work.o
    text data bss dec hex filename
    451 8 1 460 1cc kernel/irq_work.o

    After:

    christoph@linux-2.6$ size kernel/irq_work.o
    text data bss dec hex filename
    438 8 1 447 1bf kernel/irq_work.o

    Cc: Peter Zijlstra
    Signed-off-by: Christoph Lameter

    Christoph Lameter
     

18 Nov, 2010

1 commit

  • The compiler warned us about:

    kernel/irq_work.c: In function 'irq_work_run':
    kernel/irq_work.c:148: warning: value computed is not used

    Dropping the cmpxchg() result is indeed weird, but correct -
    so annotate away the warning.

    Signed-off-by: Sergio Aguirre
    Cc: Huang Ying
    Cc: Martin Schwidefsky
    Cc: Kyle McMartin
    Signed-off-by: Peter Zijlstra
    LKML-Reference:
    Signed-off-by: Ingo Molnar

    Sergio Aguirre
     

19 Oct, 2010

1 commit

  • Provide a mechanism that allows running code in IRQ context. It is
    most useful for NMI code that needs to interact with the rest of the
    system -- like wakeup a task to drain buffers.

    Perf currently has such a mechanism, so extract that and provide it as
    a generic feature, independent of perf so that others may also
    benefit.

    The IRQ context callback is generated through self-IPIs where
    possible, or on architectures like powerpc the decrementer (the
    built-in timer facility) is set to generate an interrupt immediately.

    Architectures that don't have anything like this get to do with a
    callback from the timer tick. These architectures can call
    irq_work_run() at the tail of any IRQ handlers that might enqueue such
    work (like the perf IRQ handler) to avoid undue latencies in
    processing the work.

    Signed-off-by: Peter Zijlstra
    Acked-by: Kyle McMartin
    Acked-by: Martin Schwidefsky
    [ various fixes ]
    Signed-off-by: Huang Ying
    LKML-Reference:
    Signed-off-by: Ingo Molnar

    Peter Zijlstra