Make controller's os.RemoveAll single-threaded

[Warashi]

What this PR does / why we need it:

This PR modifies the planner/scheduler cleanup to be single-threaded in order to reduce the file IO loads.
I did a load test with the following scenario and discovered that os.RemoveAll creates a high load for piped:

• A single piped instance manages 400 apps
• All 400 apps are triggered by one commit

In this scenario, the piped requires about 20 minutes to complete all deployments.
After this PR, the piped requires about 15 minutes to complete.

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?: Yes

• How are users affected by this change:

  • The piped requires larger disk space.
  • The piped performance becomes better

• Is this breaking change:

• How to migrate (if breaking change):

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 18 lines in your changes missing coverage. Please review.

Project coverage is 22.81%. Comparing base (5b9d417) to head (efa4e6d).
Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/app/piped/controller/controller.go	0.00%	18 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5217      +/-   ##
==========================================
- Coverage   22.83%   22.81%   -0.03%     
==========================================
  Files         420      420              
  Lines       45302    45316      +14     
==========================================
- Hits        10344    10338       -6     
- Misses      34163    34183      +20     
  Partials      795      795              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ffjlabo]

Thank you for the fix! I left a comment.

[ffjlabo]

[Ask] It looks like there is nothing to close c.workingDirRemovalCh. Does this lead to the goroutine leak?

[Warashi]

I'm not sure if we would call this pattern a goroutine leak.
This goroutine's lifetime is the same as that of the piped.
As there is only one controller for each piped, there is also only one instance of this goroutine.
I believe this code is acceptable because the number of goroutines doesn't increase.

[ffjlabo]

@Warashi

This goroutine's lifetime is the same as that of the piped.
As there is only one controller for each piped, there is also only one instance of this goroutine.

I see. I got your point 👍
But I think it would be better to make the lifetime of the channel explicit.

It would be better for the person calling controller.Run to not be aware of that.

WDYT?

[Warashi]

It would be better for the person calling controller.Run to not be aware of that.

I got your point. I'll try to make the lifetime explicit.

[Warashi]

Fixed in this commit
efa4e6d

[ffjlabo]

LGTM 🚀

[khanhtc1202]

LGTM 👍