reusable infrastructure for benchmark measurements #14464
Comments
rpoyner-tri
added a commit
to rpoyner-tri/drake
that referenced
this issue
Dec 16, 2020
Relevant to: RobotLocomotion#14464 Gather the sharable bits of "record_results" and port the cassie benchmark to use it.
|
https://github.com/rpoyner-tri/drake/tree/performance-tools has some work in progress. |
|
This would be very useful for hydroelastics too. Thank you! |
rpoyner-tri
added a commit
to rpoyner-tri/drake
that referenced
this issue
Jan 7, 2021
Relevant to: RobotLocomotion#14464 Gather the sharable bits of "record_results" and port the cassie benchmark to use it.
rpoyner-tri
added a commit
to rpoyner-tri/drake
that referenced
this issue
Jan 7, 2021
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a WIP branch, exploring the idea of factoring some experiment-execution machinery out of the cassie_benchmark example. I'm not greatly sold on how it looks so far, though I think it is potentially useful enough and solves enough distracting subproblems to be worth some sort of better expression.
rpoyner-tri
added a commit
to rpoyner-tri/drake
that referenced
this issue
Jan 7, 2021
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a WIP branch, exploring the idea of factoring some experiment-execution machinery out of the cassie_benchmark example. I'm not greatly sold on how it looks so far, though I think it is potentially useful enough and solves enough distracting subproblems to be worth some sort of better expression.
rpoyner-tri
added a commit
to rpoyner-tri/drake
that referenced
this issue
Jan 25, 2021
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a WIP branch, exploring the idea of factoring some experiment-execution machinery out of the cassie_benchmark example. I'm not greatly sold on how it looks so far, though I think it is potentially useful enough and solves enough distracting subproblems to be worth some sort of better expression.
rpoyner-tri
added a commit
to rpoyner-tri/drake
that referenced
this issue
Jan 26, 2021
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a WIP branch, exploring the idea of factoring some experiment-execution machinery out of the cassie_benchmark example. I'm not greatly sold on how it looks so far, though I think it is potentially useful enough and solves enough distracting subproblems to be worth some sort of better expression.
rpoyner-tri
added a commit
to rpoyner-tri/drake
that referenced
this issue
Jan 26, 2021
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
rpoyner-tri
added a commit
to rpoyner-tri/drake
that referenced
this issue
Jan 27, 2021
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
rpoyner-tri
added a commit
to rpoyner-tri/drake
that referenced
this issue
Jan 27, 2021
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
rpoyner-tri
added a commit
to rpoyner-tri/drake
that referenced
this issue
Jan 27, 2021
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
rpoyner-tri
added a commit
to rpoyner-tri/drake
that referenced
this issue
Jan 27, 2021
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
rpoyner-tri
added a commit
to rpoyner-tri/drake
that referenced
this issue
Jan 28, 2021
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
rpoyner-tri
added a commit
to rpoyner-tri/drake
that referenced
this issue
Jan 28, 2021
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
rpoyner-tri
added a commit
to rpoyner-tri/drake
that referenced
this issue
Jan 28, 2021
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
rpoyner-tri
added a commit
to rpoyner-tri/drake
that referenced
this issue
Jan 28, 2021
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
rpoyner-tri
added a commit
to rpoyner-tri/drake
that referenced
this issue
Jan 28, 2021
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
rpoyner-tri
added a commit
that referenced
this issue
Jan 28, 2021
* tools/performance: Split useful tooling out from cassie_bench Relevant to: #14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
|
The main reuse PR has merged. I'll keep this open to track at least one new client. Optionally, it would be nice to figure out a convenient way to better manage the compare.py script from googlebenchmark. Right now, I'm using it from bazel-drake/externals, but since that is bazel-managed cache, the file disappears at random times. Running any rule that depends on googlebenchmark will bring it back, but the workflow is hardly optimal. |
josephsnyder
pushed a commit
to EricCousineau-TRI/drake
that referenced
this issue
Mar 16, 2021
…motion#14505) * tools/performance: Split useful tooling out from cassie_bench Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
|
The prospects for a new client of this infrastructure don't seem imminent, and there is a TODO regarding compare.py usage. I don't think there is much value in holding this ticket open. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Development of the
cassie_benchbenchmark involved adding a fair amount of infrastructure to control experiment conditions, manage result data, etc. Most of this would be useful for benchmarks and performance-improvement campaigns elsewhere in Drake.This issue proposes to repackage that infrastructure for reuse. A prerequisite might be to show 3 or so examples where it would be useful. Follow-up would include outfitting those examples with the reusable machinery.
The text was updated successfully, but these errors were encountered: