cpu: Wait for RCU grace periods concurrently

In kernels built with CONFIG_PREEMPT, _cpu_down() waits for RCU and
RCU-sched grace periods back-to-back, incurring quite a bit more latency
than required.  This commit therefore uses the new synchronize_rcu_mult()
to allow waiting for both grace periods concurrently.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

diff --git a/kernel/cpu.c b/kernel/cpu.c
index 9c9c9fab16cc3610afa76a6c467780482b35b0be..d63b062b62679d46af89d9028154f3e0b9d6e0ab 100644 (file)
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -380,14 +380,14 @@ static int __ref _cpu_down(unsigned int cpu, int tasks_frozen)
         * will observe it.
         *
         * For CONFIG_PREEMPT we have preemptible RCU and its sync_rcu() might
-        * not imply sync_sched(), so explicitly call both.
+        * not imply sync_sched(), so wait for both.
         *
         * Do sync before park smpboot threads to take care the rcu boost case.
         */
-#ifdef CONFIG_PREEMPT
-       synchronize_sched();
-#endif
-       synchronize_rcu();
+       if (IS_ENABLED(CONFIG_PREEMPT))
+               synchronize_rcu_mult(call_rcu, call_rcu_sched);
+       else
+               synchronize_rcu();
 
        smpboot_park_threads(cpu);