ARM: smpboot: Enable irqs on secondary CPU after marking it online/active

ARM: smpboot: Enable irqs on secondary CPU after marking it online/active

Patch is the last version from tglx on Oct 7.

Discussion is at: http://comments.gmane.org/gmane.linux.ports.arm.kernel/131919

The original commit message for the first patch version:

Frank Rowand reported:

 I have a consistent (every boot) hang on boot with the RT patches.
 With a few hacks to get console output, I get:

  rcu_preempt_state detected stalls on CPUs/tasks

 I have also replicated the problem on the ARM RealView (in tree) and
 without the RT patches.

 The problem ended up being caused by the allowed cpus mask being set
 to all possible cpus for the ksoftirqd on the secondary processors.
 So the RCU softirq was never executing on the secondary cpu.

 The problem was that ksoftirqd was woken on the secondary processors before
 the secondary processors were online. This led to allowed cpus being set
 to all cpus.

   wake_up_process()
      try_to_wake_up()
         select_task_rq()
            if (... || !cpu_online(cpu))
               select_fallback_rq(task_cpu(p), p)
                  ...
                  /* No more Mr. Nice Guy. */
                  dest_cpu = cpuset_cpus_allowed_fallback(p)
                     do_set_cpus_allowed(p, cpu_possible_mask)
                        #  Thus ksoftirqd can now run on any cpu...
</report>

The reason is that the ARM SMP boot code for the secondary CPUs enables
interrupts before the newly brought up CPU is marked online and
active.

That causes a wakeup of ksoftirqd or a wakeup of any other kernel
thread which is affine to the brought up CPU break that threads
affinity and therefor being scheduled on already online CPUs.

This problem has been observed on x86 before and the only solution is
to mark the CPU online and wait for the CPU active bit before the
point where interrupts are enabled.

Change-Id: If948ef52d434191579e1ca95d18d0c50e91a03b9
Signed-off-by: Dima Zavin <dima@android.com>