04 Dec, 2015

1 commit


02 Dec, 2015

1 commit

  • This module allows to insert errors in some of netdevice's notifier
    events. All network drivers use these notifiers to signal various events
    and to check if they are allowed, e.g. PRECHANGEMTU and CHANGEMTU
    afterwards. Until recently I had to run failure tests by injecting
    a custom module, but now this infrastructure makes it trivial to test
    these failure paths. Some of the recent bugs I fixed were found using
    this module.
    Here's an example:
    $ cd /sys/kernel/debug/notifier-error-inject/netdev
    $ echo -22 > actions/NETDEV_CHANGEMTU/error
    $ ip link set eth0 mtu 1024
    RTNETLINK answers: Invalid argument

    CC: Akinobu Mita
    CC: "David S. Miller"
    CC: netdev
    Signed-off-by: Nikolay Aleksandrov
    Signed-off-by: David S. Miller

    Nikolay Aleksandrov
     

20 Jul, 2015

1 commit

  • Although futexes are well known for being a royal pita,
    we really have very little debugging capabilities - except
    for relying on tglx's eye half the time.

    By simply making use of the existing fault-injection machinery,
    we can improve this situation, allowing generating artificial
    uaddress faults and deadlock scenarios. Of course, when this is
    disabled in production systems, the overhead for failure checks
    is practically zero -- so this is very cheap at the same time.
    Future work would be nice to now enhance trinity to make use of
    this.

    There is a special tunable 'ignore-private', which can filter
    out private futexes. Given the tsk->make_it_fail filter and
    this option, pi futexes can be narrowed down pretty closely.

    Signed-off-by: Davidlohr Bueso
    Cc: Peter Zijlstra
    Cc: Darren Hart
    Cc: Davidlohr Bueso
    Link: http://lkml.kernel.org/r/1435645562-975-3-git-send-email-dave@stgolabs.net
    Signed-off-by: Thomas Gleixner

    Davidlohr Bueso
     

19 Nov, 2012

1 commit


31 Jul, 2012

2 commits

  • This adds tools/testing/fault-injection/failcmd.sh to run a command while
    injecting slab/page allocation failures via fault injection.

    Example:

    Run a command "make -C tools/testing/selftests/ run_tests" with
    injecting slab allocation failure.

    # ./tools/testing/fault-injection/failcmd.sh \
    -- make -C tools/testing/selftests/ run_tests

    Same as above except to specify 100 times failures at most instead of
    one time at most by default.

    # ./tools/testing/fault-injection/failcmd.sh --times=100 \
    -- make -C tools/testing/selftests/ run_tests

    Same as above except to inject page allocation failure instead of slab
    allocation failure.

    # env FAILCMD_TYPE=fail_page_alloc \
    ./tools/testing/fault-injection/failcmd.sh --times=100 \
    -- make -C tools/testing/selftests/ run_tests

    Signed-off-by: Akinobu Mita
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Akinobu Mita
     
  • This patchset provides kernel modules that can be used to test the error
    handling of notifier call chain failures by injecting artifical errors to
    the following notifier chain callbacks.

    * CPU notifier
    * PM notifier
    * memory hotplug notifier
    * powerpc pSeries reconfig notifier

    Example: Inject CPU offline error (-1 == -EPERM)

    # cd /sys/kernel/debug/notifier-error-inject/cpu
    # echo -1 > actions/CPU_DOWN_PREPARE/error
    # echo 0 > /sys/devices/system/cpu/cpu1/online
    bash: echo: write error: Operation not permitted

    The patchset also adds cpu and memory hotplug tests to
    tools/testing/selftests These tests first do simple online and offline
    test and then do fault injection tests if notifier error injection
    module is available.

    This patch:

    The notifier error injection provides the ability to inject artifical
    errors to specified notifier chain callbacks. It is useful to test the
    error handling of notifier call chain failures.

    This adds common basic functions to define which type of events can be
    fail and to initialize the debugfs interface to control what error code
    should be returned and which event should be failed.

    Signed-off-by: Akinobu Mita
    Cc: Pavel Machek
    Cc: "Rafael J. Wysocki"
    Cc: Greg KH
    Cc: Benjamin Herrenschmidt
    Cc: Paul Mackerras
    Cc: Michael Ellerman
    Cc: Dave Jones
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Akinobu Mita
     

27 Oct, 2011

2 commits


04 Aug, 2011

1 commit

  • init_fault_attr_dentries() is used to export fault_attr via debugfs.
    But it can only export it in debugfs root directory.

    Per Forlin is working on mmc_fail_request which adds support to inject
    data errors after a completed host transfer in MMC subsystem.

    The fault_attr for mmc_fail_request should be defined per mmc host and
    export it in debugfs directory per mmc host like
    /sys/kernel/debug/mmc0/mmc_fail_request.

    init_fault_attr_dentries() doesn't help for mmc_fail_request. So this
    introduces fault_create_debugfs_attr() which is able to create a
    directory in the arbitrary directory and replace
    init_fault_attr_dentries().

    [akpm@linux-foundation.org: extraneous semicolon, per Randy]
    Signed-off-by: Akinobu Mita
    Tested-by: Per Forlin
    Cc: Jens Axboe
    Cc: Christoph Lameter
    Cc: Pekka Enberg
    Cc: Matt Mackall
    Cc: Randy Dunlap
    Cc: Stephen Rothwell
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Akinobu Mita
     

07 Mar, 2010

1 commit

  • Add adds a debugfs interface and additional failure modes to LKDTM to
    provide similar functionality to the provoke-crash driver submitted here:

    http://lwn.net/Articles/371208/

    Crashes can now be induced either through module parameters (as before)
    or through the debugfs interface as in provoke-crash.

    The patch also provides a new "direct" interface, where KPROBES are not
    used, i.e., the crash is invoked directly upon write to the debugfs
    file. When built without KPROBES configured, only this mode is available.

    Signed-off-by: Simon Kagstrom
    Cc: M. Mohan Kumar
    Cc: Americo Wang
    Cc: David Woodhouse
    Cc: Ingo Molnar
    Cc: "Eric W. Biederman" ,
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Simon Kagstrom
     

03 Feb, 2010

1 commit


16 Jun, 2009

1 commit

  • Many developers use "/debug/" or "/debugfs/" or "/sys/kernel/debug/"
    directory name to mount debugfs filesystem for ftrace according to
    ./Documentation/tracers/ftrace.txt file.

    And, three directory names(ex:/debug/, /debugfs/, /sys/kernel/debug/) is
    existed in kernel source like ftrace, DRM, Wireless, Documentation,
    Network[sky2]files to mount debugfs filesystem.

    debugfs means debug filesystem for debugging easy to use by greg kroah
    hartman. "/sys/kernel/debug/" name is suitable as directory name
    of debugfs filesystem.
    - debugfs related reference: http://lwn.net/Articles/334546/

    Fix inconsistency of directory name to mount debugfs filesystem.

    * From Steven Rostedt
    - find_debugfs() and tracing_files() in this patch.

    Signed-off-by: GeunSik Lim
    Acked-by : Inaky Perez-Gonzalez
    Reviewed-by : Steven Rostedt
    Reviewed-by : James Smart
    CC: Jiri Kosina
    CC: David Airlie
    CC: Peter Osterlund
    CC: Ananth N Mavinakayanahalli
    CC: Anil S Keshavamurthy
    CC: Masami Hiramatsu
    Signed-off-by: Greg Kroah-Hartman

    GeunSik Lim
     

17 Jul, 2007

2 commits

  • Fix and cleanup example scripts in fault injection documentation.

    1. Eliminate broken oops() shell function.

    2. Fold failcmd.sh and failmodule.sh into example scripts. It makes
    the example scripts work independent of current working directory.

    3. Set "space" parameter to 0 to start injecting errors immediately.

    4. Use /sys/module//sections/.data as upper bound of
    .text section. Because some module doesn't have .exit.text section.

    Signed-off-by: Akinobu Mita
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Akinobu Mita
     
  • Limiting smaller allocation failures by fault injection helps to find real
    possible bugs. Because higher order allocations are likely to fail and
    zero-order allocations are not likely to fail.

    This patch adds min-order parameter to fail_page_alloc. It specifies the
    minimum page allocation order to be injected failures.

    Signed-off-by: Akinobu Mita
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Akinobu Mita
     

09 Dec, 2006

4 commits

  • Correct, disambiguate, and reformat documentation.

    Signed-off-by: Don Mullis
    Cc: Akinobu Mita
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Don Mullis
     
  • This patch provides stacktrace filtering feature.

    The stacktrace filter allows failing only for the caller you are
    interested in.

    For example someone may want to inject kmalloc() failures into
    only e100 module. they want to inject not only direct kmalloc() call,
    but also indirect allocation, too.

    - e100_poll --> netif_receive_skb --> packet_rcv_spkt --> skb_clone
    --> kmem_cache_alloc

    This patch enables to detect function calls like this by stacktrace
    and inject failures. The script Documentaion/fault-injection/failmodule.sh
    helps it.

    The range of text section of loaded e100 is expected to be
    [/sys/module/e100/sections/.text, /sys/module/e100/sections/.exit.text)

    So failmodule.sh stores these values into /debug/failslab/address-start
    and /debug/failslab/address-end. The maximum stacktrace depth is specified
    by /debug/failslab/stacktrace-depth.

    Please see the example that demonstrates how to inject slab allocation
    failures only for a specific module
    in Documentation/fault-injection/fault-injection.txt

    [dwm@meer.net: reject failure if any caller lies within specified range]
    Signed-off-by: Akinobu Mita
    Signed-off-by: Don Mullis
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Akinobu Mita
     
  • This patch provides fault-injection capability for kmalloc.

    Boot option:

    failslab=,,,

    -- specifies the interval of failures.

    -- specifies how often it should fail in percent.

    -- specifies the size of free space where memory can be
    allocated safely in bytes.

    -- specifies how many times failures may happen at most.

    Debugfs:

    /debug/failslab/interval
    /debug/failslab/probability
    /debug/failslab/specifies
    /debug/failslab/times
    /debug/failslab/ignore-gfp-highmem
    /debug/failslab/ignore-gfp-wait

    Example:

    failslab=10,100,0,-1

    slab allocation (kmalloc(), kmem_cache_alloc(),..) fails once per 10 times.

    Cc: Pekka Enberg
    Signed-off-by: Akinobu Mita
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Akinobu Mita
     
  • This patch set provides some fault-injection capabilities.

    - kmalloc() failures

    - alloc_pages() failures

    - disk IO errors

    We can see what really happens if those failures happen.

    In order to enable these fault-injection capabilities:

    1. Enable relevant config options (CONFIG_FAILSLAB, CONFIG_PAGE_ALLOC,
    CONFIG_MAKE_REQUEST) and if you want to configure them via debugfs,
    enable CONFIG_FAULT_INJECTION_DEBUG_FS.

    2. Build and boot with this kernel

    3. Configure fault-injection capabilities behavior by boot option or debugfs

    - Boot option

    failslab=
    fail_page_alloc=
    fail_make_request=

    - Debugfs

    /debug/failslab/*
    /debug/fail_page_alloc/*
    /debug/fail_make_request/*

    Please refer to the Documentation/fault-injection/fault-injection.txt
    for details.

    4. See what really happens.

    Signed-off-by: Akinobu Mita
    Signed-off-by: Don Mullis
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Akinobu Mita