anolis: sched: Fix the performence regression because of update_rq_on…

…_expel()

ANBZ: torvalds#696

The performence of online task regressed because of update_rq_on_expel()
and __update_rq_on_expel() are not inlined in the compilation phase,
althrogh they have the attribute "inline".

And this patch is to fix this problem.

Here follows the test scenario and test result:

Online task: Floating-point number arithmetic task with 3 threads,
	calculating the square root from 1 to 400000000, and the bvt
	of its cgroup is 2.
Offline task: stress-ng -c $nr_cpus -l 70, and the bvt of its cgroup
	is -1.

1. Only online task running, online task performence:
quantile 0.5:  3.8557952085e+09 ns
quantile 0.90:  3.8581308506e+09 ns
quantile 0.95:  3.8589838853e+09 ns
quantile 0.99:  3.860188924e+09 ns

2. Online task + offline task before applying this patch, online task
performence:
quantile 0.5:  3.9636566725e+09 ns
quantile 0.90:  4.0008950969e+09 ns
quantile 0.95:  4.0098206868e+09 ns
quantile 0.99:  4.196525352e+09 ns

3. Online task + offline task after applying this patch, online task
performence:
quantile 0.5:  3.8602548115e+09 ns
quantile 0.90:  3.8633850429e+09 ns
quantile 0.95:  3.8649974241e+09 ns
quantile 0.99:  3.869420975e+09 ns

This patch meets our expection.

Fixes: 1d45c19 ("sched: make ID_LOOSE_EXPEL defined with CONFIG_GROUP_IDENTITY on")
Signed-off-by: Cruz Zhao <[email protected]>
Acked-by: Michael Wang <[email protected]>

kernel/sched/fair.c

@@ -673,7 +673,7 @@ static inline bool need_expel(int this_cpu)
 	return false;
 }
 
-static inline void __update_rq_on_expel(struct rq *rq)
+static __always_inline void __update_rq_on_expel(struct rq *rq)
 {
 	bool ret = need_expel(rq->cpu);
 
@@ -693,18 +693,16 @@ static inline void __update_rq_on_expel(struct rq *rq)
  * irq is disabled, so an underclass task is picked to run on this cpu until
  * reschedule, which will degrade the performence of highclass task.
  */
-static inline void update_rq_on_expel(struct rq *rq)
+static __always_inline void update_rq_on_expel(struct rq *rq)
 {
-	if (!sched_feat(ID_LOOSE_EXPEL))
-		goto update;
-
-	if (!rq->nr_under_running
-	    || !time_after(jiffies, rq->next_expel_update))
-		return;
-	rq->next_expel_update = jiffies +
-		msecs_to_jiffies(sysctl_sched_expel_update_interval);
+	if (sched_feat(ID_LOOSE_EXPEL)) {
+		if (!rq->nr_under_running
+		    || !time_after(jiffies, rq->next_expel_update))
+			return;
 
-update:
+		rq->next_expel_update = jiffies +
+			msecs_to_jiffies(sysctl_sched_expel_update_interval);
+	}
 	__update_rq_on_expel(rq);
 }