Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regression on scalafmt: Error lifting Process: Failed to merge input digests for process #14858

Closed
alonsodomin opened this issue Mar 21, 2022 · 3 comments · Fixed by #14889
Closed

Regression on scalafmt: Error lifting Process: Failed to merge input digests for process #14858

alonsodomin opened this issue Mar 21, 2022 · 3 comments · Fixed by #14889
Assignees
@stuhood
Labels
backend: JVM JVM backend-related issues bug
Milestone
2.11.x

Comments

@alonsodomin
Copy link
Contributor

alonsodomin commented Mar 21, 2022 •
edited
Loading

Describe the bug
Pants 2.11 RC0 shows a regression bug when running ScalaFMT on subset of the files in the monorepo. The final error displays as follows:

Engine traceback:
  in select
  in pants.core.goals.fmt.fmt
  in pants.core.goals.fmt.fmt_language
  in pants.backend.scala.lint.scalafmt.rules.scalafmt_fmt (scalafmt)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (no traceback):
  <pants native internals>
Exception: Error lifting Process: Failed to merge input digests for process: String("Computed digest for Snapshot loaded from store mismatched: Digest { hash: Fingerprint<f30734b515303ba317d4bf44dc68d10aec43510c4d8a4a57512e1dd16d0e1a52>, size_bytes: 172 } vs Digest { hash: Fingerprint<6d66001a88fdeb90927dbc77a5b6916d277e8728ed92e3fbc9b0d7dcc9d9c204>, size_bytes: 172 }")

The error happens when running ./pants fmt over a fairly big subset of the repository (or in the whole repository). Running the same command over smaller subsets of targets works fine.

Attaching a DEBUG level log of the relevant parts. It seems the error happens in the scalafmt_fmt goal when running the JVM processes for each of the partitions. It also looks like it managed to run one partition successfuly.

pants.log.txt

Pants version
2.11rc0

OS
MacOS 12.3 on Intel

Additional info
Add any other information about the problem here, such as attachments or links to gists, if relevant.
@stuhood stuhood added this to the 2.11.x milestone Mar 21, 2022
@stuhood
Copy link
Member

stuhood commented Mar 21, 2022

Thanks for the report! Marked as blocking 2.11.0.

@stuhood stuhood self-assigned this Mar 21, 2022
@stuhood
Copy link
Member

stuhood commented Mar 22, 2022

I have a repro on this one: will try to get a patch out tomorrow.

@tdyas tdyas added the backend: JVM JVM backend-related issues label Mar 22, 2022
@stuhood stuhood mentioned this issue Mar 23, 2022
Merged
stuhood added a commit that referenced this issue Mar 23, 2022
@stuhood
Unify glob-matching implementations to fix malformed snapshot created…
246bc9e 
… by subsetting (#14889)

#14858 reported that an invalid `Snapshot` was being created, and reproducing in debug mode triggered `debug_assertions` related to the validity of `remexec::Directory`s created by the subset matching code.

Although porting the existing subset-matching code to `DigestTrie` _and_ fixing the bug would be one option, we've wanted to remove duplication between the "apply globs to a `Snapshot`" and "apply globs to the filesystem" codepaths for a long time (see #9967), and fixing the bug by unifying the codepaths kills two birds with one stone.

This change ports to implementing subset matching using `Vfs::expand_globs`, followed by the creation of a new `Snapshot` from the matches. This is definitely not as optimized as the direct subset matching was (`./cargo bench -p store -- subset` reports a change from ~20ms to ~100ms for 10k files), but it opens the door to unified optimization of _both_ our glob-expansion code and in-memory glob matching in parallel: see #14890.

Fixes #9967, fixes #14858, fixes #12462, and fixes #13112.
stuhood added a commit to stuhood/pants that referenced this issue Mar 23, 2022
@stuhood
Unify glob-matching implementations to fix malformed snapshot created…
ca0ade5 
… by subsetting (pantsbuild#14889)

pantsbuild#14858 reported that an invalid `Snapshot` was being created, and reproducing in debug mode triggered `debug_assertions` related to the validity of `remexec::Directory`s created by the subset matching code.

Although porting the existing subset-matching code to `DigestTrie` _and_ fixing the bug would be one option, we've wanted to remove duplication between the "apply globs to a `Snapshot`" and "apply globs to the filesystem" codepaths for a long time (see pantsbuild#9967), and fixing the bug by unifying the codepaths kills two birds with one stone.

This change ports to implementing subset matching using `Vfs::expand_globs`, followed by the creation of a new `Snapshot` from the matches. This is definitely not as optimized as the direct subset matching was (`./cargo bench -p store -- subset` reports a change from ~20ms to ~100ms for 10k files), but it opens the door to unified optimization of _both_ our glob-expansion code and in-memory glob matching in parallel: see pantsbuild#14890.

Fixes pantsbuild#9967, fixes pantsbuild#14858, fixes pantsbuild#12462, and fixes pantsbuild#13112.
@stuhood stuhood mentioned this issue Mar 23, 2022
Merged
@stuhood
Copy link
Member

stuhood commented Mar 23, 2022

The fix should go into the next 2.11.x release via #14896: thanks again for the report!

stuhood added a commit that referenced this issue Mar 24, 2022
@stuhood
Unify glob-matching implementations to fix malformed snapshot created…
280e556 
… by subsetting (cherrypick of #14889) (#14896)

#14858 reported that an invalid `Snapshot` was being created, and reproducing in debug mode triggered `debug_assertions` related to the validity of `remexec::Directory`s created by the subset matching code.

Although porting the existing subset-matching code to `DigestTrie` _and_ fixing the bug would be one option, we've wanted to remove duplication between the "apply globs to a `Snapshot`" and "apply globs to the filesystem" codepaths for a long time (see #9967), and fixing the bug by unifying the codepaths kills two birds with one stone.

This change ports to implementing subset matching using `Vfs::expand_globs`, followed by the creation of a new `Snapshot` from the matches. This is definitely not as optimized as the direct subset matching was (`./cargo bench -p store -- subset` reports a change from ~20ms to ~100ms for 10k files), but it opens the door to unified optimization of _both_ our glob-expansion code and in-memory glob matching in parallel: see #14890.

Fixes #9967, fixes #14858, fixes #12462, and fixes #13112.

[ci skip-build-wheels]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stuhood stuhood

Labels
backend: JVM JVM backend-related issues bug
Projects
None yet
Milestone
2.11.x
Development

Successfully merging a pull request may close this issue.

Unify glob-matching implementations to fix malformed snapshot created by subsetting stuhood/pants
3 participants
@stuhood @tdyas @alonsodomin