mm: page allocator: drain per-cpu lists after direct reclaim allocation fails

When under significant memory pressure, a process enters direct reclaim and immediately afterwards tries to allocate a page. If it fails and no further progress is made, it's possible the system will go OOM. However, on systems with large amounts of memory, it's possible that a significant number of pages are on per-cpu lists and inaccessible to the calling process. This leads to a process entering direct reclaim more often than it should increasing the pressure on the system and compounding the problem. This patch notes that if direct reclaim is making progress but allocations are still failing that the system is already under heavy pressure. In this case, it drains the per-cpu lists and tries the allocation a second time before continuing. Signed-off-by: Mel Gorman <mel@csn.ul.ie> Reviewed-by: Minchan Kim <minchan.kim@gmail.com> Reviewed-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> Reviewed-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com> Reviewed-by: Christoph Lameter <cl@linux.com> Cc: Dave Chinner <david@fromorbit.com> Cc: Wu Fengguang <fengguang.wu@intel.com> Cc: David Rientjes <rientjes@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

mm: page allocator: drain per-cpu lists after direct reclaim allocation fails
When under significant memory pressure, a process enters direct reclaim and immediately afterwards tries to allocate a page. If it fails and no further progress is made, it's possible the system will go OOM. However, on systems with large amounts of memory, it's possible that a significant number of pages are on per-cpu lists and inaccessible to the calling process. This leads to a process entering direct reclaim more often than it should increasing the pressure on the system and compounding the problem. This patch notes that if direct reclaim is making progress but allocations are still failing that the system is already under heavy pressure. In this case, it drains the per-cpu lists and tries the allocation a second time before continuing. Signed-off-by: Mel Gorman <mel@csn.ul.ie> Reviewed-by: Minchan Kim <minchan.kim@gmail.com> Reviewed-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> Reviewed-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com> Reviewed-by: Christoph Lameter <cl@linux.com> Cc: Dave Chinner <david@fromorbit.com> Cc: Wu Fengguang <fengguang.wu@intel.com> Cc: David Rientjes <rientjes@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Mel Gorman · Linus Torvalds
1 parent aa45484031
Showing 1 changed file with 16 additions and 4 deletions Side-by-side Diff
mm/page_alloc.c
@@ -1847,6 +1847,7 @@
 	struct page *page = NULL;
 	struct reclaim_state reclaim_state;
 	struct task_struct *p = current;
+	bool drained = false;
  
 	cond_resched();
  
  
  
@@ -1865,14 +1866,25 @@
  
 	cond_resched();
  
-	if (order != 0)
-		drain_all_pages();
+	if (unlikely(!(*did_some_progress)))
+		return NULL;
  
-	if (likely(*did_some_progress))
-		page = get_page_from_freelist(gfp_mask, nodemask, order,
+retry:
+	page = get_page_from_freelist(gfp_mask, nodemask, order,
 					zonelist, high_zoneidx,
 					alloc_flags, preferred_zone,
 					migratetype);
+
+	/*
+	 * If an allocation failed after direct reclaim, it could be because
+	 * pages are pinned on the per-cpu lists. Drain them and try again
+	 */
+	if (!page && !drained) {
+		drain_all_pages();
+		drained = true;
+		goto retry;
+	}
+
 	return page;
 }
...	...	@@ -1847,6 +1847,7 @@
1847	1847	struct page *page = NULL;
1848	1848	struct reclaim_state reclaim_state;
1849	1849	struct task_struct *p = current;
	1850	+ bool drained = false;
1850	1851
1851	1852	cond_resched();
1852	1853
1853	1854
1854	1855
...	...	@@ -1865,14 +1866,25 @@
1865	1866
1866	1867	cond_resched();
1867	1868
1868		- if (order != 0)
1869		- drain_all_pages();
	1869	+ if (unlikely(!(*did_some_progress)))
	1870	+ return NULL;
1870	1871
1871		- if (likely(*did_some_progress))
1872		- page = get_page_from_freelist(gfp_mask, nodemask, order,
	1872	+retry:
	1873	+ page = get_page_from_freelist(gfp_mask, nodemask, order,
1873	1874	zonelist, high_zoneidx,
1874	1875	alloc_flags, preferred_zone,
1875	1876	migratetype);
	1877	+
	1878	+ /*
	1879	+ * If an allocation failed after direct reclaim, it could be because
	1880	+ * pages are pinned on the per-cpu lists. Drain them and try again
	1881	+ */
	1882	+ if (!page && !drained) {
	1883	+ drain_all_pages();
	1884	+ drained = true;
	1885	+ goto retry;
	1886	+ }
	1887	+
1876	1888	return page;
1877	1889	}
1878	1890