Bench nomad exec #4852

fmaste · 2023-01-26T19:09:32Z

Exec task driver and standalone Nomad Server and Clients (no -dev mode)

Local/testing environment
- Bye bye -dev mode, now both Nomad Servers and Clients are started independently
- Can automate as many Nomad Clients as needed (wb backend pass nomad server|client start|stop NAME)
- Optional stateful Nomad Agents instead of starting and stopping on every run
- Only when using the Nomad exec Task Driver Clients are run as root
- Create symlinks to every Task's stdout and stderr file for easier/live monitoring and debugging
Cloud deployments
- Add a new mode called nomadexec that uses SRE's patched version of Nomad
- As part of the switch-over to directly-executable derivations, the Nomad Job specification file is self contained. Every script and config file is included as a template stanza only leaving outside the genesis folder.
- Created an ad-hoc "container-specs.json" file with GitHub references (towards being able to benchmark any cardano-node commit from any workbench version).
- The workbench checks that the commit being tested is publicly available on GitHub
- Fetch all Nomad (some dynamically) generated start.sh and config.json files for log and debugging purposes
Both local and cloud
- Control and monitoring of Nomad Jobs was refactored in line with the important role it has
- Removal of all locality assumptions
- Removal of all 1-1 allocation ID-task name relation assumptions
Others
- Fixed how wb backend calls backend specific subcommands

Makefile

nix/workbench/service/nodes.nix

deepfire

@fmaste, just one comment/question.

Thank you! : -)

fmaste · 2023-03-14T01:45:06Z

bors merge

4852: Bench nomad exec r=fmaste a=fmaste Exec task driver and standalone Nomad Server and Clients (no -dev mode) - Local/testing environment - - Bye bye `-dev` mode, now both Nomad Servers and Clients are started independently - - Can automate as many Nomad Clients as needed (`wb backend pass nomad server|client start|stop NAME`) - - Optional stateful Nomad Agents instead of starting and stopping on every run - - Only when using the Nomad `exec` Task Driver Clients are run as root - - Create symlinks to every Task's `stdout` and `stderr` file for easier/live monitoring and debugging - Cloud deployments - - Add a new mode called `nomadexec` that uses SRE's patched version of Nomad - - As part of the switch-over to directly-executable derivations, the Nomad Job specification file is self contained. Every script and config file is included as a template stanza only leaving outside the genesis folder. - - Created an ad-hoc "container-specs.json" file with GitHub references (towards being able to benchmark any cardano-node commit from any workbench version). - - The workbench checks that the commit being tested is publicly available on GitHub - - Fetch all Nomad (some dynamically) generated `start.sh` and `config.json` files for log and debugging purposes - Both local and cloud - - Control and monitoring of Nomad Jobs was refactored in line with the important role it has - - Removal of all locality assumptions - - Removal of all 1-1 `allocation ID`-`task name` relation assumptions - Others - - Fixed how `wb backend` calls backend specific subcommands Co-authored-by: Federico Mastellone <[email protected]>

iohk-bors · 2023-03-14T01:46:10Z

This PR was included in a batch that successfully built, but then failed to merge into master. It will not be retried.

Additional information:

{"message":"Validation Failed","documentation_url":"https://docs.github.com/articles/about-protected-branches"}

fmaste · 2023-03-14T11:53:42Z

bors merge

4852: Bench nomad exec r=fmaste a=fmaste Exec task driver and standalone Nomad Server and Clients (no -dev mode) - Local/testing environment - - Bye bye `-dev` mode, now both Nomad Servers and Clients are started independently - - Can automate as many Nomad Clients as needed (`wb backend pass nomad server|client start|stop NAME`) - - Optional stateful Nomad Agents instead of starting and stopping on every run - - Only when using the Nomad `exec` Task Driver Clients are run as root - - Create symlinks to every Task's `stdout` and `stderr` file for easier/live monitoring and debugging - Cloud deployments - - Add a new mode called `nomadexec` that uses SRE's patched version of Nomad - - As part of the switch-over to directly-executable derivations, the Nomad Job specification file is self contained. Every script and config file is included as a template stanza only leaving outside the genesis folder. - - Created an ad-hoc "container-specs.json" file with GitHub references (towards being able to benchmark any cardano-node commit from any workbench version). - - The workbench checks that the commit being tested is publicly available on GitHub - - Fetch all Nomad (some dynamically) generated `start.sh` and `config.json` files for log and debugging purposes - Both local and cloud - - Control and monitoring of Nomad Jobs was refactored in line with the important role it has - - Removal of all locality assumptions - - Removal of all 1-1 `allocation ID`-`task name` relation assumptions - Others - - Fixed how `wb backend` calls backend specific subcommands Co-authored-by: Federico Mastellone <[email protected]>

iohk-bors · 2023-03-14T11:54:46Z

This PR was included in a batch that successfully built, but then failed to merge into master. It will not be retried.

Additional information:

{"message":"Validation Failed","documentation_url":"https://docs.github.com/articles/about-protected-branches"}

fmaste · 2023-03-14T12:47:17Z

bors merge

4852: Bench nomad exec r=fmaste a=fmaste Exec task driver and standalone Nomad Server and Clients (no -dev mode) - Local/testing environment - - Bye bye `-dev` mode, now both Nomad Servers and Clients are started independently - - Can automate as many Nomad Clients as needed (`wb backend pass nomad server|client start|stop NAME`) - - Optional stateful Nomad Agents instead of starting and stopping on every run - - Only when using the Nomad `exec` Task Driver Clients are run as root - - Create symlinks to every Task's `stdout` and `stderr` file for easier/live monitoring and debugging - Cloud deployments - - Add a new mode called `nomadexec` that uses SRE's patched version of Nomad - - As part of the switch-over to directly-executable derivations, the Nomad Job specification file is self contained. Every script and config file is included as a template stanza only leaving outside the genesis folder. - - Created an ad-hoc "container-specs.json" file with GitHub references (towards being able to benchmark any cardano-node commit from any workbench version). - - The workbench checks that the commit being tested is publicly available on GitHub - - Fetch all Nomad (some dynamically) generated `start.sh` and `config.json` files for log and debugging purposes - Both local and cloud - - Control and monitoring of Nomad Jobs was refactored in line with the important role it has - - Removal of all locality assumptions - - Removal of all 1-1 `allocation ID`-`task name` relation assumptions - Others - - Fixed how `wb backend` calls backend specific subcommands Co-authored-by: Federico Mastellone <[email protected]>

iohk-bors · 2023-03-14T12:48:26Z

This PR was included in a batch that successfully built, but then failed to merge into master. It will not be retried.

Additional information:

{"message":"Required status check \"build-complete\" is expected.","documentation_url":"https://docs.github.com/articles/about-protected-branches"}

fmaste force-pushed the bench-nomad-exec branch 3 times, most recently from a015319 to 8898923 Compare February 1, 2023 14:04

deepfire force-pushed the bench-nomad-exec branch from 8898923 to 8454b0d Compare February 1, 2023 15:20

fmaste force-pushed the bench-nomad-exec branch 2 times, most recently from 41f05c8 to 177b9ce Compare February 8, 2023 05:57

fmaste force-pushed the bench-nomad-exec branch from a1c2c8a to 3998d2c Compare February 15, 2023 21:21

fmaste force-pushed the bench-nomad-exec branch 13 times, most recently from 528fc5e to e9dfef8 Compare March 1, 2023 13:58

fmaste force-pushed the bench-nomad-exec branch 3 times, most recently from a897d37 to 5230b79 Compare March 8, 2023 20:02

fmaste marked this pull request as ready for review March 9, 2023 14:09

fmaste requested review from a team, deepfire, mgmeier, dcoutts, Jimbo4350 and erikd as code owners March 9, 2023 14:09

fmaste force-pushed the bench-nomad-exec branch from 2604801 to fd0a93d Compare March 13, 2023 13:11

deepfire reviewed Mar 13, 2023

View reviewed changes

Makefile Show resolved Hide resolved

deepfire reviewed Mar 13, 2023

View reviewed changes

nix/workbench/service/nodes.nix Show resolved Hide resolved

deepfire approved these changes Mar 13, 2023

View reviewed changes

fmaste force-pushed the bench-nomad-exec branch from fd0a93d to 723e9ab Compare March 14, 2023 02:34

fmaste added 6 commits March 14, 2023 12:46

workbench: Post massive refactoring fixes/cleanup

f2e5363

workbench: Fix backend specific subcommands calling error

ae14921

workbench: Use XDG dirs

8a11781

workbench: wb publish comments and small fixes

99c0dcd

workbench: Stateful and multiple independent Nomad Clients

f99c579

workbench: Nomad exec task driver

6dc3f9e

fmaste force-pushed the bench-nomad-exec branch from 723e9ab to 6dc3f9e Compare March 14, 2023 12:46

fmaste enabled auto-merge March 14, 2023 12:53

fmaste disabled auto-merge March 14, 2023 13:05

fmaste enabled auto-merge March 14, 2023 13:06

fmaste added this pull request to the merge queue Mar 14, 2023

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 14, 2023

fmaste added this pull request to the merge queue Mar 14, 2023

Merged via the queue into master with commit a16c198 Mar 14, 2023

iohk-bors bot deleted the bench-nomad-exec branch March 14, 2023 14:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bench nomad exec #4852

Bench nomad exec #4852

fmaste commented Jan 26, 2023 •

edited

Loading

deepfire left a comment

fmaste commented Mar 14, 2023

iohk-bors bot commented Mar 14, 2023

fmaste commented Mar 14, 2023

iohk-bors bot commented Mar 14, 2023

fmaste commented Mar 14, 2023

iohk-bors bot commented Mar 14, 2023

Bench nomad exec #4852

Bench nomad exec #4852

Conversation

fmaste commented Jan 26, 2023 • edited Loading

deepfire left a comment

Choose a reason for hiding this comment

fmaste commented Mar 14, 2023

iohk-bors bot commented Mar 14, 2023

fmaste commented Mar 14, 2023

iohk-bors bot commented Mar 14, 2023

fmaste commented Mar 14, 2023

iohk-bors bot commented Mar 14, 2023

fmaste commented Jan 26, 2023 •

edited

Loading