-
-
Notifications
You must be signed in to change notification settings - Fork 638
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] Cached local process execution #7171
[WIP] Cached local process execution #7171
Conversation
- compiles - runs - may not work at all
Notes from @illicitonion on slack on how to get this mergeable:
|
I believe @hrfuller completed this work! |
WIP because this is truly massive and also addresses the tower protos migration a bit by converting
Action
andCommand
inremote.rs
into the new protos (that usage was also moved tocached_execution.rs
). That part wasn't intentional, but it happens to make all of the below a lot simpler and avoid converting between different proto types completely. I can clean this up if this is an appropriate scope.Problem
Resolves #6898.
(explain the context of the problem and why you're making this change. include
references to all relevant github issues.)
Solution
lmdb::Database
tofs::local::ByteStore
as a k/v store forbazel_protos::build::bazel::remote::execution::v2::Action{,Result}
(from the new tower protos)fs::Store
cached_execution.rs
which containsCacheableExecuteProcessRe{quest,sult}
analogs toExecuteProcessRequest
andFallibleExecuteProcessResult
, but which either contain caching information or don't contain information relevant to a specific process invocation, so that they can deterministically be serialized intoAction{,Result}
into theByteStore
remote.rs
intocached_execution.rs
CachingCommandRunner
whichimpl
sprocess_execution::CommandRunner
and serializes process invocations intoAction
s, and when complete, stores the result into anActionResult
in the process execution k/v db added tofs::local::ByteStore
earlier.bazel_protos::remote_execution::{Action,Command}
inremote.rs
to the new tower protos(describe the modifications you've made.)
Result
./pants -ldebug cloc ::
is instantaneous with pantsd on, for one. This alone is a fantastic reason for tasks to consider switching to v2 rules, or at the very least a v2 process invocation (which is easier to keep cacheable when the process request is created using v2 rules). The only remainingbazel_protos::remote_execution::*
protos I saw wereDirectory
andFile
infs::Store
-- this is likely not hard to finish, as that remaining usage was still 100% orthogonal to this PR.(describe how your changes affect the end-user behavior of the system. this section is
optional, and should generally be summarized in the title of the pull request.)