Change default queue settings to be low go-routines

[zeripath]

This PR suggests a change to the default configuration for queues:

• Use a common DATADIR for the queues
• Set starting workers to 0 and make boost a single worker

This will change the baseline default go-routine number to 31.

Fixes #15542

⚠️ BREAKING ⚠️

• As a result of this PR instead of each queue having their own level db by default, the queues will use a common level db. It is recommended to ensure that queues are empty before upgrading using gitea manager flush-queues or on the admin pages.
• Starting workers are now 0 with boost workers at 1. If you have explicitly set BOOST_WORKERS = 0 you will need to explicitly set WORKERS to at least 1. Administrators of busy sites should tune their WORKERS, BOOST_WORKERS, & DATADIR parameters as needed.

Signed-off-by: Andrew Thornton [email protected]

[codecov-commenter]

Codecov Report

Merging #15964 (a58b499) into main (6021fbf) will decrease coverage by 0.03%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main   #15964      +/-   ##
==========================================
- Coverage   44.07%   44.04%   -0.04%     
==========================================
  Files         682      682              
  Lines       82408    82408              
==========================================
- Hits        36322    36295      -27     
- Misses      40176    40210      +34     
+ Partials     5910     5903       -7     

Impacted Files	Coverage Δ
modules/setting/indexer.go	91.42% <100.00%> (ø)
modules/setting/queue.go	67.92% <100.00%> (ø)
models/repo_mirror.go	16.66% <0.00%> (-11.91%)	⬇️
modules/notification/mail/mail.go	38.77% <0.00%> (-3.07%)	⬇️
modules/cron/tasks_basic.go	85.43% <0.00%> (-2.92%)	⬇️
modules/log/event.go	58.96% <0.00%> (-2.84%)	⬇️
modules/queue/manager.go	65.53% <0.00%> (-2.83%)	⬇️
modules/charset/charset.go	71.71% <0.00%> (-2.03%)	⬇️
services/gitdiff/gitdiff.go	72.35% <0.00%> (-1.42%)	⬇️
modules/queue/workerpool.go	53.81% <0.00%> (-1.15%)	⬇️
... and 10 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6021fbf...a58b499. Read the comment docs.

[lunny]

This is a break change. If user hasn't set the value of ISSUE_INDEXER_QUEUE_DIR and the previous queue directory still have data, they will lost if they didn't rename that directory. I think we should give some documents about this.

[jpraet]

I now get this error on Windows:

2021/05/24 19:27:52 ...eue/queue_wrapped.go:229:Run() [F] Unable to set the internal queue for issue_indexer-wrapper Error: Unable to create queue level for issue_indexer with cfg {"Name":"issue_indexer","DataDir":"C:/gitea/data/queues/common","QueueLength":20,"BatchLength":20,"ConnectionString":"","Type":"level","Network":"","Addresses":"","Password":"","QueueName":"issue_indexer_queue","SetName":"","DBIndex":0,"WrapIfNecessary":true,"MaxAttempts":10,"Timeout":90000000000,"Workers":0,"MaxWorkers":10,"BlockTimeout":1000000000,"BoostTimeout":300000000000,"BoostWorkers":1} by max attempts: error: The process cannot access the file because it is being used by another process.

[zeripath]

Yeah that's why I wrote breaking change. Should have been clearer...

@jpraet that's interesting because this should be the same file...

Try:

[queue]
DATADIR=C:/gitea/data/queues/common

I wonder if the common db is not normalizing the file names properly?

[zeripath]

Ah so the problem is the use of filepath.Join in modules/settings/queue.go:96, and then lack of an equivalent in the indexer.

I'll pop up a PR that does filepath.ToSlash(filepath.Join(...)) in the three places.

[zeripath]

@lunny the recommendation would be to simply run queue flush prior to shutdown and restart until the queues are empty, or set the settings as appropriate.