01 Jun, 2012

40 commits

  • We already check the mq attr struct if it's passed in, but now that the
    admin can set system wide defaults separate from maximums, it's actually
    possible to set the defaults to something that would overflow. So, if
    there is no attr struct passed in to the open call, check the default
    values.

    While we are at it, simplify mq_attr_ok() by making it return 0 or an
    error condition, so that way if we add more tests to it later, we have the
    option of what error should be returned instead of the calling location
    having to pick a possibly inaccurate error code.

    [akpm@linux-foundation.org: s/ENOMEM/EOVERFLOW/]
    Signed-off-by: Doug Ledford
    Cc: Stephen Rothwell
    Cc: Manfred Spraul
    Acked-by: KOSAKI Motohiro
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Doug Ledford
     
  • While working on the other parts of the mqueue stuff, I noticed that the
    calculation for overflow in mq_attr_ok didn't actually match reality (this
    is especially true since my last patch which changed how we account memory
    slightly).

    In particular, we used to test for overflow using:
    msgs * msgsize + msgs * sizeof(struct msg_msg *)

    That was never really correct because each message we allocate via
    load_msg() is actually a struct msg_msg followed by the data for the
    message (and if struct msg_msg + data exceeds PAGE_SIZE we end up
    allocating struct msg_msgseg structs too, but accounting for them would
    get really tedious, so let's ignore those...they're only a pointer in size
    anyway). This patch updates the calculation to be more accurate in
    regards to maximum possible memory consumption by the mqueue.

    [akpm@linux-foundation.org: add a local to simplify overflow-checking expression]
    Signed-off-by: Doug Ledford
    Cc: Stephen Rothwell
    Cc: Manfred Spraul
    Acked-by: KOSAKI Motohiro
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Doug Ledford
     
  • The existing implementation of the POSIX message queue send and recv
    functions is, well, abysmal. Even worse than abysmal. I submitted a
    patch to increase the maximum POSIX message queue limit to 65536 due to
    customer needs, however, upon looking over the send/recv implementation, I
    realized that my customer needs help with that too even if they don't know
    it. The basic problem is that, given the fairly typical use case scenario
    for a large queue of queueing lots of messages all at the same priority (I
    verified with my customer that this is indeed what their app does), the
    msg_insert routine is basically a frikkin' bubble sort. I mean, whoa,
    that's *so* middle school.

    OK, OK, to not slam the original author too much, I'm sure they didn't
    envision a queue depth of 50,000+ messages. No one would think that
    moving elements in an array, one at a time, and dereferencing each pointer
    in that array to check priority of the message being pointed too, again
    one at a time, for 50,000+ times would be good. So let's assume that, as
    is typical, the users have found a way to break our code simply by using
    it in a way we didn't envision. Fair enough.

    "So, just how broken is it?", you ask. I wondered the same thing, so I
    wrote an app to let me know. It's my next patch. It gave me some
    interesting results. Here's what it tested:

    Interference with other apps - In continuous mode, the app just sits there
    and hits a message queue forever, while you go do something productive on
    another terminal using other CPUs. You then measure how long it takes you
    to do that something productive. Then you restart the app in fake
    continuous mode, and it sits in a tight loop on a CPU while you repeat
    your tests. The whole point of this is to keep one CPU tied up (so it
    can't be used in your other work) but in one case tied up hitting the
    mqueue code so we can see the effect of walking that 65,528 element array
    one pointer at a time on the global CPU cache. If it's bad, then it will
    slow down your app on the other CPUs just by polluting cache mercilessly.
    In the fake case, it will be in a tight loop, but not polluting cache.
    Testing the mqueue subsystem directly - Here we just run a number of tests
    to see how the mqueue subsystem performs under different conditions. A
    couple conditions are known to be worst case for the old system, and some
    routines, so this tests all of them.

    So, on to the results already:

    Subsystem/Test Old New

    Time to compile linux
    kernel (make -j12 on a
    6 core CPU)
    Running mqueue test user 49m10.744s user 45m26.294s
    sys 5m51.924s sys 4m59.894s
    total 55m02.668s total 50m26.188s

    Running fake test user 45m32.686s user 45m18.552s
    sys 5m12.465s sys 4m56.468s
    total 50m45.151s total 50m15.020s

    % slowdown from mqueue
    cache thrashing ~8% ~.5%

    Avg time to send/recv (in nanoseconds per message)
    when queue empty 305/288 349/318
    when queue full (65528 messages)
    constant priority 526589/823 362/314
    increasing priority 403105/916 495/445
    decreasing priority 73420/594 482/409
    random priority 280147/920 546/436

    Time to fill/drain queue (65528 messages, in seconds)
    constant priority 17.37/.12 .13/.12
    increasing priority 4.14/.14 .21/.18
    decreasing priority 12.93/.13 .21/.18
    random priority 8.88/.16 .22/.17

    So, I think the results speak for themselves. It's possible this
    implementation could be improved by cacheing at least one priority level
    in the node tree (that would bring the queue empty performance more in
    line with the old implementation), but this works and is *so* much better
    than what we had, especially for the common case of a single priority in
    use, that further refinements can be in follow on patches.

    [akpm@linux-foundation.org: fix typo in comment, remove stray semicolon]
    [levinsasha928@gmail.com: use correct gfp flags in msg_insert]
    Signed-off-by: Doug Ledford
    Cc: Stephen Rothwell
    Cc: Manfred Spraul
    Acked-by: KOSAKI Motohiro
    Signed-off-by: Sasha Levin
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Doug Ledford
     
  • Add a directory to house POSIX message queue subsystem specific tests.
    Add first test which checks the operation of mq_open() under various
    corner conditions.

    Signed-off-by: Doug Ledford
    Cc: KOSAKI Motohiro
    Cc: Doug Ledford
    Cc: Joe Korty
    Cc: Amerigo Wang
    Cc: Serge E. Hallyn
    Cc: Jiri Slaby
    Cc: Manfred Spraul
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Doug Ledford
     
  • Commit b231cca4381e ("message queues: increase range limits") changed
    mqueue default value when attr parameter is specified NULL from hard
    coded value to fs.mqueue.{msg,msgsize}_max sysctl value.

    This made large side effect. When user need to use two mqueue
    applications 1) using !NULL attr parameter and it require big message
    size and 2) using NULL attr parameter and only need small size message,
    app (1) require to raise fs.mqueue.msgsize_max and app (2) consume large
    memory size even though it doesn't need.

    Doug Ledford propsed to switch back it to static hard coded value.
    However it also has a compatibility problem. Some applications might
    started depend on the default value is tunable.

    The solution is to separate default value from maximum value.

    Signed-off-by: KOSAKI Motohiro
    Signed-off-by: Doug Ledford
    Acked-by: Doug Ledford
    Acked-by: Joe Korty
    Cc: Amerigo Wang
    Acked-by: Serge E. Hallyn
    Cc: Jiri Slaby
    Cc: Manfred Spraul
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    KOSAKI Motohiro
     
  • KMALLOC_MAX_SIZE is not a good threshold. It is extremely high and
    problematic. Unfortunately, some silly drivers depend on this and we
    can't change it. But any new code needn't use such extreme ugly high
    order allocations. It brings us awful fragmentation issues and system
    slowdown.

    Signed-off-by: KOSAKI Motohiro
    Acked-by: Doug Ledford
    Acked-by: Joe Korty
    Cc: Amerigo Wang
    Cc: Serge E. Hallyn
    Cc: Jiri Slaby
    Cc: Joe Korty
    Cc: Manfred Spraul
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    KOSAKI Motohiro
     
  • Mqueue limitation is slightly naieve parameter likes other ipcs because
    unprivileged user can consume kernel memory by using ipcs.

    Thus, too aggressive raise bring us security issue. Example, current
    setting allow evil unprivileged user use 256GB (= 256 * 1024 * 1024*1024)
    and it's enough large to system will belome unresponsive. Don't do that.

    Instead, every admin should adjust the knobs for their own systems.

    Signed-off-by: KOSAKI Motohiro
    Acked-by: Doug Ledford
    Acked-by: Joe Korty
    Cc: Amerigo Wang
    Acked-by: Serge E. Hallyn
    Cc: Jiri Slaby
    Cc: Manfred Spraul
    Cc: Dave Hansen
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    KOSAKI Motohiro
     
  • Commit b231cca4381e ("message queues: increase range limits") changed the
    maximum size of a message in a message queue from INT_MAX to 8192*128.
    Unfortunately, we had customers that relied on a size much larger than
    8192*128 on their production systems. After reviewing POSIX, we found
    that it is silent on the maximum message size. We did find a couple other
    areas in which it was not silent. Fix up the mqueue maximums so that the
    customer's system can continue to work, and document both the POSIX and
    real world requirements in ipc_namespace.h so that we don't have this
    issue crop back up.

    Also, commit 9cf18e1dd74cd0 ("ipc: HARD_MSGMAX should be higher not lower
    on 64bit") fiddled with HARD_MSGMAX without realizing that the number was
    intentionally in place to limit the msg queue depth to one that was small
    enough to kmalloc an array of pointers (hence why we divided 128k by
    sizeof(long)). If we wish to meet POSIX requirements, we have no choice
    but to change our allocation to a vmalloc instead (at least for the large
    queue size case). With that, it's possible to increase our allowed
    maximum to the POSIX requirements (or more if we choose).

    [sfr@canb.auug.org.au: using vmalloc requires including vmalloc.h]
    Signed-off-by: Doug Ledford
    Cc: Serge E. Hallyn
    Cc: Amerigo Wang
    Cc: Joe Korty
    Cc: Jiri Slaby
    Acked-by: KOSAKI Motohiro
    Cc: Manfred Spraul
    Signed-off-by: Stephen Rothwell
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Doug Ledford
     
  • In two places we don't enforce the hard limits for CAP_SYS_RESOURCE apps.
    In preparation for making more reasonable hard limits, start enforcing
    them even on CAP_SYS_RESOURCE.

    Signed-off-by: Doug Ledford
    Cc: Serge E. Hallyn
    Cc: Amerigo Wang
    Cc: Joe Korty
    Cc: Jiri Slaby
    Acked-by: KOSAKI Motohiro
    Cc: Manfred Spraul
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Doug Ledford
     
  • Commit b231cca4381e ("message queues: increase range limits") changed
    how we create a queue that does not include an attr struct passed to
    open so that it creates the queue with whatever the maximum values are.
    However, if the admin has set the maximums to allow flexibility in
    creating a queue (aka, both a large size and large queue are allowed,
    but combined they create a queue too large for the RLIMIT_MSGQUEUE of
    the user), then attempts to create a queue without an attr struct will
    fail. Switch back to using acceptable defaults regardless of what the
    maximums are.

    Note: so far, we only know of a few applications that rely on this
    behavior (specifically, set the maximums in /proc, then run the
    application which calls mq_open() without passing in an attr struct, and
    the application expects the newly created message queue to have the
    maximum sizes that were set in /proc used on the mq_open() call, and all
    of those applications that we know of are actually part of regression
    test suites that were coded to do something like this:

    for size in 4096 65536 $((1024 * 1024)) $((16 * 1024 * 1024)); do
    echo $size > /proc/sys/fs/mqueue/msgsize_max
    mq_open || echo "Error opening mq with size $size"
    done

    These test suites that depend on any behavior like this are broken. The
    concept that programs should rely upon the system wide maximum in order
    to get their desired results instead of simply using a attr struct to
    specify what they want is fundamentally unfriendly programming practice
    for any multi-tasking OS.

    Fixing this will break those few apps that we know of (and those app
    authors recognize the brokenness of their code and the need to fix it).
    However, the following patch "mqueue: separate mqueue default value"
    allows a workaround in the form of new knobs for the default msg queue
    creation parameters for any software out there that we don't already
    know about that might rely on this behavior at the moment.

    Signed-off-by: Doug Ledford
    Cc: Serge E. Hallyn
    Cc: Amerigo Wang
    Cc: Joe Korty
    Cc: Jiri Slaby
    Acked-by: KOSAKI Motohiro
    Cc: Manfred Spraul
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Doug Ledford
     
  • Since commit b231cca4381e ("message queues: increase range limits") on
    Oct 18, 2008, calls to mq_open() that did not pass in an attribute
    struct and expected to get default values for the size of the queue and
    the max message size now get the system wide maximums instead of
    hardwired defaults like they used to get.

    This was uncovered when one of the earlier patches in this patch set
    increased the default system wide maximums at the same time it increased
    the hard ceiling on the system wide maximums (a customer specifically
    needed the hard ceiling brought back up, the new ceiling that commit
    b231cca4381e introduced was too low for their production systems). By
    increasing the default maximums and not realising they were tied to any
    attempt to create a message queue without an attribute struct, I had
    inadvertently made it such that all message queue creation attempts
    without an attribute struct were failing because the new default
    maximums would create a queue that exceeded the default rlimit for
    message queue bytes.

    As a result, the system wide defaults were brought back down to their
    previous levels, and the system wide ceilings on the maximums were
    raised to meet the customer's needs. However, the fact that the no
    attribute struct behavior of mq_open() could be broken by changing the
    system wide maximums for message queues was seen as fundamentally broken
    itself. So we hardwired the no attribute case back like it used to be.
    But, then we realized that on the very off chance that some piece of
    software in the wild depended on that behavior, we could work around
    that issue by adding two new knobs to /proc that allowed setting the
    defaults for message queues created without an attr struct separately
    from the system wide maximums.

    What is not an option IMO is to leave the current behavior in place. No
    piece of software should ever rely on setting the system wide maximums
    in order to get a desired message queue. Such a reliance would be so
    fundamentally multitasking OS unfriendly as to not really be tolerable.
    Fortunately, we don't know of any software in the wild that uses this
    except for a regression test program that caught the issue in the first
    place. If there is though, we have made accommodations with the two new
    /proc knobs (and that's all the accommodations such fundamentally broken
    software can be allowed)..

    This patch:

    The various defines for minimums and maximums of the sysctl controllable
    mqueue values are scattered amongst different files and named
    inconsistently. Move them all into ipc_namespace.h and make them have
    consistent names. Additionally, make the number of queues per namespace
    also have a minimum and maximum and use the same sysctl function as the
    other two settable variables.

    Signed-off-by: Doug Ledford
    Acked-by: Serge E. Hallyn
    Cc: Amerigo Wang
    Cc: Joe Korty
    Cc: Jiri Slaby
    Acked-by: KOSAKI Motohiro
    Cc: Manfred Spraul
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Doug Ledford
     
  • Add userspace definitions, guard all relevant kernel structures. While at
    it document stuff and remove now useless userspace hint.

    It is easy to add the relevant system call to respective libc's, but it
    seems pointless to have to duplicate the data structures.

    This is based on the kexec-tools headers, with the exception of just using
    int on return (succes or failure) and using size_t instead of 'unsigned
    long int' for the number of segments argument of kexec_load().

    Signed-off-by: maximilian attems
    Cc: Simon Horman
    Cc: Vivek Goyal
    Cc: Haren Myneni
    Cc: "Eric W. Biederman"
    Cc: Martin Schwidefsky
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    maximilian attems
     
  • Add more comments on clear_tasks_mm_cpumask, plus adds a runtime check:
    the function is only suitable for offlined CPUs, and if called
    inappropriately, the kernel should scream aloud.

    [akpm@linux-foundation.org: tweak comment: s/walks up/walks/, use 80 cols]
    Suggested-by: Andrew Morton
    Suggested-by: Peter Zijlstra
    Signed-off-by: Anton Vorontsov
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Anton Vorontsov
     
  • kill_off_processes() might miss a valid process, this is because checking
    for process->mm is not enough. Process' main thread may exit or detach
    its mm via use_mm(), but other threads may still have a valid mm.

    To catch this we use find_lock_task_mm(), which walks up all threads and
    returns an appropriate task (with task lock held).

    Suggested-by: Oleg Nesterov
    Signed-off-by: Anton Vorontsov
    Cc: Richard Weinberger
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Anton Vorontsov
     
  • Checking for task->mm is dangerous as ->mm might disappear (exit_mm()
    assigns NULL under task_lock(), so tasklist lock is not enough).

    We can't use get_task_mm()/mmput() pair as mmput() might sleep, so let's
    take the task lock while we care about its mm.

    Note that we should also use find_lock_task_mm() to check all process'
    threads for a valid mm, but for uml we'll do it in a separate patch.

    Signed-off-by: Anton Vorontsov
    Cc: Richard Weinberger
    Cc: Oleg Nesterov
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Anton Vorontsov
     
  • Traversing the tasks requires holding tasklist_lock, otherwise it is
    unsafe.

    p.s. However, I'm not sure that calling os_kill_ptraced_process() in the
    atomic context is correct. It seem to work, but please take a closer
    look.

    Signed-off-by: Anton Vorontsov
    Cc: Richard Weinberger
    Cc: Oleg Nesterov
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Anton Vorontsov
     
  • Oleg Nesterov found an interesting deadlock possibility:

    > sysrq_showregs_othercpus() does smp_call_function(showacpu)
    > and showacpu() show_stack()->decode_address(). Now suppose that IPI
    > interrupts the task holding read_lock(tasklist).

    To fix this, blackfin should not grab the write_ variant of the
    tasklist lock, read_ one is enough.

    Suggested-by: Oleg Nesterov
    Signed-off-by: Anton Vorontsov
    Cc: Mike Frysinger
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Anton Vorontsov
     
  • The patch fixes two problems:

    1. Working with task->mm w/o getting mm or grabing the task lock is
    dangerous as ->mm might disappear (exit_mm() assigns NULL under
    task_lock(), so tasklist lock is not enough).

    We can't use get_task_mm()/mmput() pair as mmput() might sleep,
    so we have to take the task lock while handle its mm.

    2. Checking for process->mm is not enough because process' main
    thread may exit or detach its mm via use_mm(), but other threads
    may still have a valid mm.

    To catch this we use find_lock_task_mm(), which walks up all
    threads and returns an appropriate task (with task lock held).

    Suggested-by: Oleg Nesterov
    Signed-off-by: Anton Vorontsov
    Cc: Mike Frysinger
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Anton Vorontsov
     
  • Checking for process->mm is not enough because process' main thread may
    exit or detach its mm via use_mm(), but other threads may still have a
    valid mm.

    To fix this we would need to use find_lock_task_mm(), which would walk up
    all threads and returns an appropriate task (with task lock held).

    clear_tasks_mm_cpumask() has the issue fixed, so let's use it.

    Suggested-by: Oleg Nesterov
    Signed-off-by: Anton Vorontsov
    Cc: Paul Mundt
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Anton Vorontsov
     
  • Current CPU hotplug code has some task->mm handling issues:

    1. Working with task->mm w/o getting mm or grabing the task lock is
    dangerous as ->mm might disappear (exit_mm() assigns NULL under
    task_lock(), so tasklist lock is not enough).

    We can't use get_task_mm()/mmput() pair as mmput() might sleep,
    so we must take the task lock while handle its mm.

    2. Checking for process->mm is not enough because process' main
    thread may exit or detach its mm via use_mm(), but other threads
    may still have a valid mm.

    To fix this we would need to use find_lock_task_mm(), which would
    walk up all threads and returns an appropriate task (with task
    lock held).

    clear_tasks_mm_cpumask() has all the issues fixed, so let's use it.

    Suggested-by: Oleg Nesterov
    Signed-off-by: Anton Vorontsov
    Acked-by: Benjamin Herrenschmidt
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Anton Vorontsov
     
  • Checking for process->mm is not enough because process' main thread may
    exit or detach its mm via use_mm(), but other threads may still have a
    valid mm.

    To fix this we would need to use find_lock_task_mm(), which would walk up
    all threads and returns an appropriate task (with task lock held).

    clear_tasks_mm_cpumask() has this issue fixed, so let's use it.

    Suggested-by: Oleg Nesterov
    Signed-off-by: Anton Vorontsov
    Cc: Russell King
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Anton Vorontsov
     
  • Many architectures clear tasks' mm_cpumask like this:

    read_lock(&tasklist_lock);
    for_each_process(p) {
    if (p->mm)
    cpumask_clear_cpu(cpu, mm_cpumask(p->mm));
    }
    read_unlock(&tasklist_lock);

    Depending on the context, the code above may have several problems,
    such as:

    1. Working with task->mm w/o getting mm or grabing the task lock is
    dangerous as ->mm might disappear (exit_mm() assigns NULL under
    task_lock(), so tasklist lock is not enough).

    2. Checking for process->mm is not enough because process' main
    thread may exit or detach its mm via use_mm(), but other threads
    may still have a valid mm.

    This patch implements a small helper function that does things
    correctly, i.e.:

    1. We take the task's lock while whe handle its mm (we can't use
    get_task_mm()/mmput() pair as mmput() might sleep);

    2. To catch exited main thread case, we use find_lock_task_mm(),
    which walks up all threads and returns an appropriate task
    (with task lock held).

    Also, Per Peter Zijlstra's idea, now we don't grab tasklist_lock in
    the new helper, instead we take the rcu read lock. We can do this
    because the function is called after the cpu is taken down and marked
    offline, so no new tasks will get this cpu set in their mm mask.

    Signed-off-by: Anton Vorontsov
    Cc: Richard Weinberger
    Cc: Oleg Nesterov
    Cc: Peter Zijlstra
    Cc: Russell King
    Cc: Benjamin Herrenschmidt
    Cc: Mike Frysinger
    Cc: Paul Mundt
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Anton Vorontsov
     
  • Child should wake up the parent from vfork() only after finishing all
    operations with shared mm. There is no sense in using
    CLONE_CHILD_CLEARTID together with CLONE_VFORK, but it looks more accurate
    now.

    Signed-off-by: Konstantin Khlebnikov
    Cc: Oleg Nesterov
    Cc: Hugh Dickins
    Cc: KAMEZAWA Hiroyuki
    Cc: Konstantin Khlebnikov
    Cc: Markus Trippelsdorf
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Konstantin Khlebnikov
     
  • Currently, nonlinear mappings can not be distinguished from ordinary
    mappings. This patch adds into /proc/pid/smaps line "Nonlinear:
    kB", where size is amount of nonlinear ptes in vma, this line appears only
    if VM_NONLINEAR is set. This information may be useful not only for
    checkpoint/restore project.

    Requested by Pavel Emelyanov.

    Signed-off-by: Konstantin Khlebnikov
    Cc: Andi Kleen
    Cc: Pavel Emelyanov
    Cc: Alexey Dobriyan
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Konstantin Khlebnikov
     
  • Currently smaps reports migration entries as "swap", as result "swap" can
    appears in shared mapping.

    This patch converts migration entries into pages and handles them as usual.

    Signed-off-by: Konstantin Khlebnikov
    Cc: Andi Kleen
    Cc: Pavel Emelyanov
    Cc: Alexey Dobriyan
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Konstantin Khlebnikov
     
  • This is an implementation of Andrew's proposal to extend the pagemap file
    bits to report what is missing about tasks' working set.

    The problem with the working set detection is multilateral. In the criu
    (checkpoint/restore) project we dump the tasks' memory into image files
    and to do it properly we need to detect which pages inside mappings are
    really in use. The mincore syscall I though could help with this did not.
    First, it doesn't report swapped pages, thus we cannot find out which
    parts of anonymous mappings to dump. Next, it does report pages from page
    cache as present even if they are not mapped, and it doesn't make that has
    not been cow-ed.

    Note, that issue with swap pages is critical -- we must dump swap pages to
    image file. But the issues with file pages are optimization -- we can
    take all file pages to image, this would be correct, but if we know that a
    page is not mapped or not cow-ed, we can remove them from dump file. The
    dump would still be self-consistent, though significantly smaller in size
    (up to 10 times smaller on real apps).

    Andrew noticed, that the proc pagemap file solved 2 of 3 above issues --
    it reports whether a page is present or swapped and it doesn't report not
    mapped page cache pages. But, it doesn't distinguish cow-ed file pages
    from not cow-ed.

    I would like to make the last unused bit in this file to report whether the
    page mapped into respective pte is PageAnon or not.

    [comment stolen from Pavel Emelyanov's v1 patch]

    Signed-off-by: Konstantin Khlebnikov
    Cc: Pavel Emelyanov
    Cc: Matt Mackall
    Cc: Hugh Dickins
    Cc: Rik van Riel
    Acked-by: KOSAKI Motohiro
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Konstantin Khlebnikov
     
  • - use int fpr priority and nice, since task_nice()/task_prio() return that

    - field 24: get_mm_rss() returns unsigned long

    Signed-off-by: Jan Engelhardt
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Jan Engelhardt
     
  • Pass "fd" directly, not via pointer -- one less memory read.

    Signed-off-by: Alexey Dobriyan
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Alexey Dobriyan
     
  • rcu_read_lock()/rcu_read_unlock() is nop for TINY_RCU, but is not a nop
    for, say, PREEMPT_RCU.

    proc_fill_cache() is called without RCU lock, there is no need to
    lock/unlock on error path, simply jump out of the loop.

    Signed-off-by: Alexey Dobriyan
    Cc: "Paul E. McKenney"
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Alexey Dobriyan
     
  • mm_access() handles this much better, and avoids some race conditions.

    Signed-off-by: Cong Wang
    Cc: Oleg Nesterov
    Cc: Alexey Dobriyan
    Cc: Hugh Dickins
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Cong Wang
     
  • mm_for_maps() is a simple wrapper for mm_access(), and the name is
    misleading, so just remove it and use mm_access() directly.

    Signed-off-by: Cong Wang
    Cc: Oleg Nesterov
    Cc: Alexey Dobriyan
    Acked-by: Hugh Dickins
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Cong Wang
     
  • Similar to e268337dfe26 ("proc: clean up and fix /proc//mem
    handling"), move the check of permission to open(), this will simplify
    read() code.

    [akpm@linux-foundation.org: checkpatch fixes]
    Signed-off-by: Cong Wang
    Cc: Oleg Nesterov
    Cc: Alexey Dobriyan
    Cc: Hugh Dickins
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Cong Wang
     
  • In embedded systems, sometimes the same program (busybox) is the cause of
    multiple warnings. Outputting the pid with the program name in the
    warning printk helps distinguish which instances of a program are using
    the stack most.

    This is a small patch, but useful.

    Signed-off-by: Tim Bird
    Cc: Oleg Nesterov
    Cc: Frederic Weisbecker
    Cc: Andrea Arcangeli
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Tim Bird
     
  • Commit 8f92054e7ca1 ("CRED: Fix __task_cred()'s lockdep check and banner
    comment"):

    add the following validation condition:

    task->exit_state >= 0

    to permit the access if the target task is dead and therefore
    unable to change its own credentials.

    OK, but afaics currently this can only help wait_task_zombie() which calls
    __task_cred() without rcu lock.

    Remove this validation and change wait_task_zombie() to use task_uid()
    instead. This means we do rcu_read_lock() only to shut up the lockdep,
    but we already do the same in, say, wait_task_stopped().

    task_is_dead() should die, task->exit_state != 0 means that this task has
    passed exit_notify(), only do_wait-like code paths should use this.

    Unfortunately, we can't kill task_is_dead() right now, it has already
    acquired buggy users in drivers/staging. The fix already exists.

    Signed-off-by: Oleg Nesterov
    Reviewed-by: "Eric W. Biederman"
    Acked-by: David Howells
    Cc: Jiri Olsa
    Cc: Paul E. McKenney
    Cc: James Morris
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Oleg Nesterov
     
  • Warning(kernel/kmod.c:419): No description found for parameter 'depth'

    Signed-off-by: Randy Dunlap
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Randy Dunlap
     
  • If we move call_usermodehelper_fns() to kmod.c file and EXPORT_SYMBOL it
    we can avoid exporting all it's helper functions:
    call_usermodehelper_setup
    call_usermodehelper_setfns
    call_usermodehelper_exec
    And make all of them static to kmod.c

    Since the optimizer will see all these as a single call site it will
    inline them inside call_usermodehelper_fns(). So we loose the call to
    _fns but gain 3 calls to the helpers. (Not that it matters)

    Signed-off-by: Boaz Harrosh
    Cc: Oleg Nesterov
    Cc: Tetsuo Handa
    Cc: Ingo Molnar
    Cc: Peter Zijlstra
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Boaz Harrosh
     
  • Both kernel/sys.c && security/keys/request_key.c where inlining the exact
    same code as call_usermodehelper_fns(); So simply convert these sites to
    directly use call_usermodehelper_fns().

    Signed-off-by: Boaz Harrosh
    Cc: Oleg Nesterov
    Cc: Tetsuo Handa
    Cc: Ingo Molnar
    Cc: Peter Zijlstra
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Boaz Harrosh
     
  • call_usermodehelper_freeinfo() is not used outside of kmod.c. So unexport
    it, and make it static to kmod.c

    Signed-off-by: Boaz Harrosh
    Cc: Oleg Nesterov
    Cc: Tetsuo Handa
    Cc: Ingo Molnar
    Cc: Peter Zijlstra
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Boaz Harrosh
     
  • If an application tries to lookup (opendir/readdir/stat) 5000 files on a
    fatfs USB device and the device is unplugged, many message occur, shown
    below. This makes the application slow. So use the new
    fat_msg_ratelimit() decrease the messaging rate.

    #> ./file_lookup_testcase ./files_directory/
    usb 2-1.4: USB disconnect, device number 4
    FAT-fs (sda1): FAT read failed (blocknr 2631)
    FAT-fs (sda1): Directory bread(block 396816) failed
    FAT-fs (sda1): Directory bread(block 396817) failed
    FAT-fs (sda1): Directory bread(block 396818) failed
    FAT-fs (sda1): Directory bread(block 396819) failed
    FAT-fs (sda1): Directory bread(block 396820) failed
    FAT-fs (sda1): Directory bread(block 396821) failed
    FAT-fs (sda1): Directory bread(block 396822) failed
    FAT-fs (sda1): Directory bread(block 396823) failed
    FAT-fs (sda1): Directory bread(block 406824) failed
    FAT-fs (sda1): Directory bread(block 406825) failed
    FAT-fs (sda1): Directory bread(block 406826) failed
    FAT-fs (sda1): Directory bread(block 406827) failed
    FAT-fs (sda1): Directory bread(block 406828) failed
    FAT-fs (sda1): Directory bread(block 406829) failed
    FAT-fs (sda1): Directory bread(block 406830) failed
    FAT-fs (sda1): Directory bread(block 406831) failed
    FAT-fs (sda1): Directory bread(block 417696) failed
    FAT-fs (sda1): Directory bread(block 417697) failed
    FAT-fs (sda1): Directory bread(block 417698) failed
    FAT-fs (sda1): Directory bread(block 417699) failed
    FAT-fs (sda1): Directory bread(block 417700) failed
    FAT-fs (sda1): Directory bread(block 417701) failed
    FAT-fs (sda1): Directory bread(block 417702) failed
    FAT-fs (sda1): Directory bread(block 417703) failed
    FAT-fs (sda1): FAT read failed (blocknr 2631)
    FAT-fs (sda1): Directory bread(block 396816) failed
    FAT-fs (sda1): Directory bread(block 396817) failed
    FAT-fs (sda1): Directory bread(block 396818) failed
    FAT-fs (sda1): Directory bread(block 396819) failed
    FAT-fs (sda1): Directory bread(block 396820) failed
    FAT-fs (sda1): Directory bread(block 396821) failed

    Signed-off-by: Namjae Jeon
    Signed-off-by: Amit Sahrawat
    Acked-by: OGAWA Hirofumi
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Namjae Jeon
     
  • Add a fat_msg_ratelimit() to limit the message generation rate.

    Signed-off-by: Namjae Jeon
    Signed-off-by: Amit Sahrawat
    Acked-by: OGAWA Hirofumi
    Signed-off-by: Andrew Morton
    Signed-off-by: Linus Torvalds

    Namjae Jeon