-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
reusable infrastructure for benchmark measurements #14464
Comments
Relevant to: RobotLocomotion#14464 Gather the sharable bits of "record_results" and port the cassie benchmark to use it.
https://github.com/rpoyner-tri/drake/tree/performance-tools has some work in progress. |
This would be very useful for hydroelastics too. Thank you! |
Relevant to: RobotLocomotion#14464 Gather the sharable bits of "record_results" and port the cassie benchmark to use it.
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a WIP branch, exploring the idea of factoring some experiment-execution machinery out of the cassie_benchmark example. I'm not greatly sold on how it looks so far, though I think it is potentially useful enough and solves enough distracting subproblems to be worth some sort of better expression.
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a WIP branch, exploring the idea of factoring some experiment-execution machinery out of the cassie_benchmark example. I'm not greatly sold on how it looks so far, though I think it is potentially useful enough and solves enough distracting subproblems to be worth some sort of better expression.
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a WIP branch, exploring the idea of factoring some experiment-execution machinery out of the cassie_benchmark example. I'm not greatly sold on how it looks so far, though I think it is potentially useful enough and solves enough distracting subproblems to be worth some sort of better expression.
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a WIP branch, exploring the idea of factoring some experiment-execution machinery out of the cassie_benchmark example. I'm not greatly sold on how it looks so far, though I think it is potentially useful enough and solves enough distracting subproblems to be worth some sort of better expression.
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
* tools/performance: Split useful tooling out from cassie_bench Relevant to: #14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
The main reuse PR has merged. I'll keep this open to track at least one new client. Optionally, it would be nice to figure out a convenient way to better manage the compare.py script from googlebenchmark. Right now, I'm using it from bazel-drake/externals, but since that is bazel-managed cache, the file disappears at random times. Running any rule that depends on googlebenchmark will bring it back, but the workflow is hardly optimal. |
…motion#14505) * tools/performance: Split useful tooling out from cassie_bench Relevant to: RobotLocomotion#14464 Gather the sharable bits of experiment execution scripting and port the cassie benchmark to use it. This is a first pass at factoring out the common stuff from the cassie_benchmark tree, for use by other benchmarks. It hoists the basic techniques for data collection and variance reduction. As this gathers more uses, it can be polished more. Also missing from this patch is any documentation or tooling to help with using the compare.py script from the googlebenchmark package.
The prospects for a new client of this infrastructure don't seem imminent, and there is a TODO regarding compare.py usage. I don't think there is much value in holding this ticket open. |
Development of the
cassie_bench
benchmark involved adding a fair amount of infrastructure to control experiment conditions, manage result data, etc. Most of this would be useful for benchmarks and performance-improvement campaigns elsewhere in Drake.This issue proposes to repackage that infrastructure for reuse. A prerequisite might be to show 3 or so examples where it would be useful. Follow-up would include outfitting those examples with the reusable machinery.
The text was updated successfully, but these errors were encountered: