mm, thp: avoid excessive compaction latency during fault

Synchronous memory compaction can be very expensive: it can iterate an
enormous amount of memory without aborting, constantly rescheduling,
waiting on page locks and lru_lock, etc, if a pageblock cannot be
defragmented.

Unfortunately, it's too expensive for transparent hugepage page faults and
it's much better to simply fallback to pages.  On 128GB machines, we find
that synchronous memory compaction can take O(seconds) for a single thp
fault.

Now that async compaction remembers where it left off without strictly
relying on sync compaction, this makes thp allocations best-effort without
causing egregious latency during fault.  We still need to retry async
compaction after reclaim, but this won't stall for seconds.

Signed-off-by: David Rientjes <rientjes@google.com>
Acked-by: Mel Gorman <mgorman@suse.de>
Cc: Greg Thelen <gthelen@google.com>
Cc: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 8cac6912bba3c86c1b61d1dad42428df3c4b339d..1f8100c747b3636ec466dfd870c26e62ddc94d86 100644 (file)
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2575,7 +2575,17 @@ rebalance:
                                        &did_some_progress);
        if (page)
                goto got_pg;
-       migration_mode = MIGRATE_SYNC_LIGHT;
+
+       if (gfp_mask & __GFP_NO_KSWAPD) {
+               /*
+                * Khugepaged is allowed to try MIGRATE_SYNC_LIGHT, the latency
+                * of this allocation isn't critical.  Everything else, however,
+                * should only be allowed to do MIGRATE_ASYNC to avoid excessive
+                * stalls during fault.
+                */
+               if ((current->flags & (PF_KTHREAD | PF_KSWAPD)) == PF_KTHREAD)
+                       migration_mode = MIGRATE_SYNC_LIGHT;
+       }
 
        /*
         * If compaction is deferred for high-order allocations, it is because