mm, thp: allow fallback when pte_alloc_one() fails for huge pmd

The transparent hugepages feature is careful to not invoke the oom killer when a hugepage cannot be allocated. pte_alloc_one() failing in __do_huge_pmd_anonymous_page(), however, currently results in VM_FAULT_OOM which invokes the pagefault oom killer to kill a memory-hogging task. This is unnecessary since it's possible to drop the reference to the hugepage and fallback to allocating a small page. Signed-off-by: David Rientjes <rientjes@google.com> Cc: Andrea Arcangeli <aarcange@redhat.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

mm, thp: allow fallback when pte_alloc_one() fails for huge pmd
The transparent hugepages feature is careful to not invoke the oom killer when a hugepage cannot be allocated. pte_alloc_one() failing in __do_huge_pmd_anonymous_page(), however, currently results in VM_FAULT_OOM which invokes the pagefault oom killer to kill a memory-hogging task. This is unnecessary since it's possible to drop the reference to the hugepage and fallback to allocating a small page. Signed-off-by: David Rientjes <rientjes@google.com> Cc: Andrea Arcangeli <aarcange@redhat.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
David Rientjes · Linus Torvalds
1 parent aa2e878efa
Showing 1 changed file with 8 additions and 5 deletions Side-by-side Diff
mm/huge_memory.c
@@ -640,11 +640,8 @@
  
 	VM_BUG_ON(!PageCompound(page));
 	pgtable = pte_alloc_one(mm, haddr);
-	if (unlikely(!pgtable)) {
-		mem_cgroup_uncharge_page(page);
-		put_page(page);
+	if (unlikely(!pgtable))
 		return VM_FAULT_OOM;
-	}
  
 	clear_huge_page(page, haddr, HPAGE_PMD_NR);
 	__SetPageUptodate(page);
  
@@ -723,8 +720,14 @@
 			put_page(page);
 			goto out;
 		}
+		if (unlikely(__do_huge_pmd_anonymous_page(mm, vma, haddr, pmd,
+							  page))) {
+			mem_cgroup_uncharge_page(page);
+			put_page(page);
+			goto out;
+		}
  
-		return __do_huge_pmd_anonymous_page(mm, vma, haddr, pmd, page);
+		return 0;
 	}
 out:
 	/*
...	...	@@ -640,11 +640,8 @@
640	640
641	641	VM_BUG_ON(!PageCompound(page));
642	642	pgtable = pte_alloc_one(mm, haddr);
643		- if (unlikely(!pgtable)) {
644		- mem_cgroup_uncharge_page(page);
645		- put_page(page);
	643	+ if (unlikely(!pgtable))
646	644	return VM_FAULT_OOM;
647		- }
648	645
649	646	clear_huge_page(page, haddr, HPAGE_PMD_NR);
650	647	__SetPageUptodate(page);
651	648
...	...	@@ -723,8 +720,14 @@
723	720	put_page(page);
724	721	goto out;
725	722	}
	723	+ if (unlikely(__do_huge_pmd_anonymous_page(mm, vma, haddr, pmd,
	724	+ page))) {
	725	+ mem_cgroup_uncharge_page(page);
	726	+ put_page(page);
	727	+ goto out;
	728	+ }
726	729
727		- return __do_huge_pmd_anonymous_page(mm, vma, haddr, pmd, page);
	730	+ return 0;
728	731	}
729	732	out:
730	733	/*