feat: expconf flag to force scheduling on a single node/container/pod

[ioga]

Description

• Add an expconf flag to allow forcing command-like scheduling for experiments (https://hpe-aiatscale.slack.com/archives/C04TUNDGKMF/p1706031288718059)
• Fix command capacity checks on static pools (https://hpe-aiatscale.slack.com/archives/C02PV33GSN5/p1706067629265619)

Test Plan

• Added an autotest to ensure flagged experiments don't get distributed across multiple nodes in e2e_cpu_2a.
• Scrapped/reworked some older tests which don't make sense any more.

Commentary (optional)

currently not supported by slurm due to https://github.com/determined-ai/determined-ee/blob/64ae6c75eb1d5072cb60b1572f394219c0210f57/master/pkg/tasks/dispatcher_task.go#L568

Checklist

• Changes have been manually QA'd
• User-facing API changes need the "User-facing API Change" label.
• Release notes should be added as a separate file under docs/release-notes/.
  See Release Note for details.
• Licenses should be included for new code which was copied and/or modified from any external code.

Ticket

[netlify]

✅ Deploy Preview for determined-ui canceled.

Name	Link
🔨 Latest commit	9b4cf74
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/65b186cf8f73940008db14ad

[codecov]

Codecov Report

Attention: 14 lines in your changes are missing coverage. Please review.

Comparison is base (b71da7a) 47.28% compared to head (9b4cf74) 47.27%.
Report is 8 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8743      +/-   ##
==========================================
- Coverage   47.28%   47.27%   -0.01%     
==========================================
  Files        1045     1045              
  Lines      166785   166800      +15     
  Branches     2244     2244              
==========================================
- Hits        78857    78855       -2     
- Misses      87770    87787      +17     
  Partials      158      158              

Flag	Coverage Δ
backend	40.90% <26.31%> (-0.03%)	⬇️
harness	64.35% <ø> (ø)
web	42.57% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
master/internal/core_experiment.go	63.15% <100.00%> (+0.16%)	⬆️
master/pkg/model/command_config.go	56.52% <100.00%> (+1.97%)	⬆️
master/pkg/model/experiment_config.go	39.24% <ø> (ø)
master/pkg/schemas/expconf/experiment_config.go	64.21% <ø> (ø)
master/pkg/schemas/zgen_schemas.go	1.24% <ø> (ø)
master/internal/trial.go	41.38% <66.66%> (+0.11%)	⬆️
master/pkg/model/compat.go	15.06% <0.00%> (-0.21%)	⬇️
...er/pkg/schemas/expconf/zgen_resources_config_v0.go	3.22% <0.00%> (-0.23%)	⬇️
master/internal/rm/agentrm/resource_pool.go	25.84% <0.00%> (-1.80%)	⬇️

... and 4 files with indirect coverage changes

[corban-beaird]

much cleaner!

[corban-beaird]

Looks good from the go & e2e side

[szewaiyuen6]

LGTM, thanks for the quick turnaround!