feat: support traffic.max_concurrency for api server and runner

[frostming]

Signed-off-by: Frost Ming [email protected]

What does this PR address?

This PR adds support for traffic configuration for both API server and runners.

The previous timeout setting is moved under traffic field plus a new configuration max_concurrency, which defaults to infinite.

Before submitting:

• Does the Pull Request follow Conventional Commits specification naming? Here are GitHub's
  guide on how to create a pull request.
• Does the code follow BentoML's code style, both make format and make lint script have passed (instructions)?
• Did you read through contribution guidelines and follow development guidelines?
• Did your changes require updates to the documentation? Have you updated
  those accordingly? Here are documentation guidelines and tips on writting docs.
• Did you write tests to cover your changes?

[codecov]

Codecov Report

Merging #3864 (b5f5a72) into main (bd56fa9) will not change coverage.
The diff coverage is 0.00%.

@@          Coverage Diff          @@
##            main   #3864   +/-   ##
=====================================
  Coverage   0.00%   0.00%           
=====================================
  Files        155     155           
  Lines      12878   12915   +37     
=====================================
- Misses     12878   12915   +37     

Impacted Files	Coverage Δ
src/bentoml/_internal/configuration/containers.py	0.00% <ø> (ø)
src/bentoml/_internal/configuration/v1/__init__.py	0.00% <0.00%> (ø)
...c/bentoml/_internal/runner/runner_handle/remote.py	0.00% <0.00%> (ø)
src/bentoml/_internal/server/base_app.py	0.00% <0.00%> (ø)
src/bentoml/_internal/server/http/traffic.py	0.00% <0.00%> (ø)
src/bentoml/_internal/server/http_app.py	0.00% <0.00%> (ø)
src/bentoml/_internal/server/runner_app.py	0.00% <0.00%> (ø)

[aarnphm]

I think the test is pretty minuscule, but this is LGTM

[aarnphm]

Also I suspect the tests might got cancelled because of the timeout. So you might need to increase the default timeout for our github actions.

[bojiang]

Here we see it will be a breaking change over our configuration.
Since there is just an adjustment over the configuration structure, we may not bother our users.
We may make the original timeout optional and do some adapting and migration.