sched: Introduce staged average NUMA faults

The current way of building the p->numa_faults[2][node] faults
statistics has a sampling artifact:

The continuous and immediate nature of propagating new fault
stats to the numa_faults array creates a 'pulsating' dynamic,
that starts at the average value at the beginning of the scan,
increases monotonically until we finish the scan to about twice
the average, and then drops back to half of its value due to
the running average.

Since we rely on these values to balance tasks, the pulsating
nature resulted in false migrations and general noise in the
stats.

To solve this, introduce buffering of the current scan via
p->task_numa_faults_curr[]. The array is co-allocated with the
p->task_numa[] for efficiency reasons, but it is otherwise an
ordinary separate array.

At the end of the scan we propagate the latest stats into the
average stats value. Most of the balancing code stays unmodified.

The cost of this change is that we delay the effects of the latest
round of faults by 1 scan - but using the partial faults info was
creating artifacts.

This instantly stabilized the page fault stats and improved
numa02-alike workloads by making them faster to converge.

Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Hugh Dickins <hughd@google.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 8f6532382c8cb67fd014c810d51a230d915ee5a9..92b41b49751b9c601e43a51eae2cd60a9a58f6fa 100644 (file)
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1511,6 +1511,7 @@ struct task_struct {
        u64 node_stamp;                 /* migration stamp  */
        unsigned long numa_weight;
        unsigned long *numa_faults;
+       unsigned long *numa_faults_curr;
        struct callback_head numa_work;
 #endif /* CONFIG_NUMA_BALANCING */
 

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 9c46b459925b26941e469741796c1e5ea202044b..1ab11be0a851f78d39121701074de996f44fc2a6 100644 (file)
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -852,12 +852,26 @@ static void task_numa_placement(struct task_struct *p)
 
        p->numa_scan_seq = seq;
 
+       /*
+        * Update the fault average with the result of the latest
+        * scan:
+        */
        for (node = 0; node < nr_node_ids; node++) {
                faults = 0;
                for (priv = 0; priv < 2; priv++) {
-                       faults += p->numa_faults[2*node + priv];
-                       total[priv] += p->numa_faults[2*node + priv];
-                       p->numa_faults[2*node + priv] /= 2;
+                       unsigned int new_faults;
+                       unsigned int idx;
+
+                       idx = 2*node + priv;
+                       new_faults = p->numa_faults_curr[idx];
+                       p->numa_faults_curr[idx] = 0;
+
+                       /* Keep a simple running average: */
+                       p->numa_faults[idx] += new_faults;
+                       p->numa_faults[idx] /= 2;
+
+                       faults += p->numa_faults[idx];
+                       total[priv] += p->numa_faults[idx];
                }
                if (faults > max_faults) {
                        max_faults = faults;