Commit 907aed48f65efeecf91575397e3d79335d93a466

Authored by Mel Gorman
Committed by Linus Torvalds
1 parent b37f1dd0f5

mm: allow PF_MEMALLOC from softirq context

This is needed to allow network softirq packet processing to make use of
PF_MEMALLOC.

Currently softirq context cannot use PF_MEMALLOC due to it not being
associated with a task, and therefore not having task flags to fiddle with
- thus the gfp to alloc flag mapping ignores the task flags when in
interrupts (hard or soft) context.

Allowing softirqs to make use of PF_MEMALLOC therefore requires some
trickery.  This patch borrows the task flags from whatever process happens
to be preempted by the softirq.  It then modifies the gfp to alloc flags
mapping to not exclude task flags in softirq context, and modify the
softirq code to save, clear and restore the PF_MEMALLOC flag.

The save and clear, ensures the preempted task's PF_MEMALLOC flag doesn't
leak into the softirq.  The restore ensures a softirq's PF_MEMALLOC flag
cannot leak back into the preempted process.  This should be safe due to
the following reasons

Softirqs can run on multiple CPUs sure but the same task should not be
	executing the same softirq code. Neither should the softirq
	handler be preempted by any other softirq handler so the flags
	should not leak to an unrelated softirq.

Softirqs re-enable hardware interrupts in __do_softirq() so can be
	preempted by hardware interrupts so PF_MEMALLOC is inherited
	by the hard IRQ. However, this is similar to a process in
	reclaim being preempted by a hardirq. While PF_MEMALLOC is
	set, gfp_to_alloc_flags() distinguishes between hard and
	soft irqs and avoids giving a hardirq the ALLOC_NO_WATERMARKS
	flag.

If the softirq is deferred to ksoftirq then its flags may be used
        instead of a normal tasks but as the softirq cannot be preempted,
        the PF_MEMALLOC flag does not leak to other code by accident.

[davem@davemloft.net: Document why PF_MEMALLOC is safe]
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
Cc: David Miller <davem@davemloft.net>
Cc: Neil Brown <neilb@suse.de>
Cc: Mike Christie <michaelc@cs.wisc.edu>
Cc: Eric B Munson <emunson@mgebm.net>
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Christoph Lameter <cl@linux.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

Showing 3 changed files with 21 additions and 1 deletions Side-by-side Diff

include/linux/sched.h
... ... @@ -1894,6 +1894,13 @@
1894 1894  
1895 1895 #endif
1896 1896  
  1897 +static inline void tsk_restore_flags(struct task_struct *task,
  1898 + unsigned long orig_flags, unsigned long flags)
  1899 +{
  1900 + task->flags &= ~flags;
  1901 + task->flags |= orig_flags & flags;
  1902 +}
  1903 +
1897 1904 #ifdef CONFIG_SMP
1898 1905 extern void do_set_cpus_allowed(struct task_struct *p,
1899 1906 const struct cpumask *new_mask);
... ... @@ -210,7 +210,15 @@
210 210 __u32 pending;
211 211 int max_restart = MAX_SOFTIRQ_RESTART;
212 212 int cpu;
  213 + unsigned long old_flags = current->flags;
213 214  
  215 + /*
  216 + * Mask out PF_MEMALLOC s current task context is borrowed for the
  217 + * softirq. A softirq handled such as network RX might set PF_MEMALLOC
  218 + * again if the socket is related to swap
  219 + */
  220 + current->flags &= ~PF_MEMALLOC;
  221 +
214 222 pending = local_softirq_pending();
215 223 account_system_vtime(current);
216 224  
... ... @@ -265,6 +273,7 @@
265 273  
266 274 account_system_vtime(current);
267 275 __local_bh_enable(SOFTIRQ_OFFSET);
  276 + tsk_restore_flags(current, old_flags, PF_MEMALLOC);
268 277 }
269 278  
270 279 #ifndef __ARCH_HAS_DO_SOFTIRQ
... ... @@ -2296,7 +2296,11 @@
2296 2296 if (likely(!(gfp_mask & __GFP_NOMEMALLOC))) {
2297 2297 if (gfp_mask & __GFP_MEMALLOC)
2298 2298 alloc_flags |= ALLOC_NO_WATERMARKS;
2299   - else if (likely(!(gfp_mask & __GFP_NOMEMALLOC)) && !in_interrupt())
  2299 + else if (in_serving_softirq() && (current->flags & PF_MEMALLOC))
  2300 + alloc_flags |= ALLOC_NO_WATERMARKS;
  2301 + else if (!in_interrupt() &&
  2302 + ((current->flags & PF_MEMALLOC) ||
  2303 + unlikely(test_thread_flag(TIF_MEMDIE))))
2300 2304 alloc_flags |= ALLOC_NO_WATERMARKS;
2301 2305 }
2302 2306