Skip formatting generated files #4296
Conversation
| assert!(is_comment_with_generated_notation("// @generated")); | ||
| assert!(is_comment_with_generated_notation("//@generated\n\n")); | ||
| assert!(is_comment_with_generated_notation("\n// @generated")); | ||
| assert!(is_comment_with_generated_notation("/* @generated")); |
There was a problem hiding this comment.
consider adding a test for string w/o @generated or not on the first line.
| .and_then(|s| s.lines().next()) | ||
| .map(str::to_owned) | ||
| .unwrap_or("".to_owned()), | ||
| FileName::Real(ref path) => fs::File::open(path) |
There was a problem hiding this comment.
Not ideal to have another file system read (IIRC we're up to at least 3 due with this plus the parser and one for newline checks) 🤔
Would it be feasible to get the first line without the file system hit, maybe via the parse session data? Pperhaps it won't matter that much given the default is false, and/or could potentially be updated later
There was a problem hiding this comment.
(forgive me for the intrusion, i was recently doing work in this area and learned this: unless you're very low on memory, recently-accessed files are cached by the kernel and effectively free to access; the slowest part will be parsing)
There was a problem hiding this comment.
forgive me for the intrusion
no worries at all!
the slowest part will be parsing
agreed, that's basically a given, and understand the main point about not strictly hitting the fs. though the intent of my original question wasn't really about a parsing/AST construction vs. additional file reads, it's whether the additional file read is necessary given that the file's already been read and loaded from the preceding parsing stage and available for use
unless you're very low on memory, recently-accessed files are cached by the kernel and effectively free to access;
do you know if that (especially the effectively free bit) holds true everywhere, for example Windows, CI environments, some of the tier 2/3 targets, etc?
I'm not terribly familiar in that space, so entirely possible this approach is better than an alternative like using the span from the associated mod to get the first line from the snippet for the file, either more performant/negligible difference due to the caching or other reasons, and/or simpler from a code readability and management perspective 🤷♂️
Also readily admit such a question may be unnecessary/premature optimization, but figured I'd ask anyway 😄
minor edits for clarity
There was a problem hiding this comment.
yeah, I agree that reusing data that is already and easily available is preferable. I just wanted to note that the performance implications of rereading a file may not be as high as it may appear, and that there are probably bigger bottlenecks. But I totally agree with you, no need to do extra work if you don't have to.
Re environments: most modern kernels (~last 20 years) have a page cache; this applies to windows as well
There was a problem hiding this comment.
Re environments: most modern kernels (~last 20 years) have a page cache; this applies to windows as well
Sure, but I suppose I should've phrased my question better.
Was wondering more so if you thought there'd always be sufficient available memory to guarantee/near-guaranatee cache hit in all scenarios. I know even on my laptops I can get really memory constrained (as I tend to use lower end machines and still go overboard with multi tasking 😆), but a lot of CI environments can be pretty low on resources and one of the things I learned during the conversations on the compiler frontend for rls/ra is that there's plenty of folks doing their inner dev loop on some pretty resource constrainted environments, and even some remote workflows.
I genuinely don't know, but wouldn't be surprised if running in some of the more resource constrained environments (or those that purged more aggressively) and/or larger workspace projects could result in some cache misses and i/o hits.
I'm quickly getting out my depth though and will happily defer to @topecongiro on whether it's worth considering an alternative 😄
There was a problem hiding this comment.
I first tried by using the span of ast::Mod, though it turned out that the span may not contain the comment at the beginning or the end of the file. I think that there are ways to get a correct span, though I opted for a simpler implementation this time.
This is a copy of rust-lang#4296 with these changes: * file is not reopened again to find if the file is generated * first five lines are scanned for `@generated` marker instead of one * no attempt is made to only search for marker in comments `@generated` marker is used by certain tools to understand that the file is generated, so it should be treated differently than a file written by a human: * linters should not be invoked on these files, * diffs in these files are less important, * and these files should not be reformatted. This PR proposes builtin support for `@generated` marker. I have not found a standard for a generated file marker, but: * Facebook [uses `@generated` marker](https://tinyurl.com/fb-generated) * Phabricator tool which was spawned from Facebook internal tool [also understands `@generated` marker](https://git.io/JnVHa) * Cargo inserts `@generated` marker into [generated Cargo.lock files](https://git.io/JnVHP) My personal story is that rust-protobuf project which I maintain was broken twice because of incompatibilities/bugs in rustfmt marker handling: [one](stepancheg/rust-protobuf#493), [two](stepancheg/rust-protobuf#551). (Also, rust-protobuf started generating `@generated` marker [6 years ago](https://git.io/JnV5h)). While rustfmt AST markers are useful to apply to a certain AST elements, disable whole-file-at-once all-tools-at-once text level marker might be easier to use and more reliable for generated code.
This is a copy of rust-lang#4296 with these changes: * file is not reopened again to find if the file is generated * first five lines are scanned for `@generated` marker instead of one * no attempt is made to only search for marker in comments `@generated` marker is used by certain tools to understand that the file is generated, so it should be treated differently than a file written by a human: * linters should not be invoked on these files, * diffs in these files are less important, * and these files should not be reformatted. This PR proposes builtin support for `@generated` marker. I have not found a standard for a generated file marker, but: * Facebook [uses `@generated` marker](https://tinyurl.com/fb-generated) * Phabricator tool which was spawned from Facebook internal tool [also understands `@generated` marker](https://git.io/JnVHa) * Cargo inserts `@generated` marker into [generated Cargo.lock files](https://git.io/JnVHP) My personal story is that rust-protobuf project which I maintain was broken twice because of incompatibilities/bugs in rustfmt marker handling: [one](stepancheg/rust-protobuf#493), [two](stepancheg/rust-protobuf#551). (Also, rust-protobuf started generating `@generated` marker [6 years ago](https://git.io/JnV5h)). While rustfmt AST markers are useful to apply to a certain AST elements, disable whole-file-at-once all-tools-at-once text level marker might be easier to use and more reliable for generated code.
This is a copy of #4296 with these changes: * file is not reopened again to find if the file is generated * first five lines are scanned for `@generated` marker instead of one * no attempt is made to only search for marker in comments `@generated` marker is used by certain tools to understand that the file is generated, so it should be treated differently than a file written by a human: * linters should not be invoked on these files, * diffs in these files are less important, * and these files should not be reformatted. This PR proposes builtin support for `@generated` marker. I have not found a standard for a generated file marker, but: * Facebook [uses `@generated` marker](https://tinyurl.com/fb-generated) * Phabricator tool which was spawned from Facebook internal tool [also understands `@generated` marker](https://git.io/JnVHa) * Cargo inserts `@generated` marker into [generated Cargo.lock files](https://git.io/JnVHP) My personal story is that rust-protobuf project which I maintain was broken twice because of incompatibilities/bugs in rustfmt marker handling: [one](stepancheg/rust-protobuf#493), [two](stepancheg/rust-protobuf#551). (Also, rust-protobuf started generating `@generated` marker [6 years ago](https://git.io/JnV5h)). While rustfmt AST markers are useful to apply to a certain AST elements, disable whole-file-at-once all-tools-at-once text level marker might be easier to use and more reliable for generated code.
This is a copy of rust-lang#4296 with these changes: * file is not reopened again to find if the file is generated * first five lines are scanned for `@generated` marker instead of one * no attempt is made to only search for marker in comments `@generated` marker is used by certain tools to understand that the file is generated, so it should be treated differently than a file written by a human: * linters should not be invoked on these files, * diffs in these files are less important, * and these files should not be reformatted. This PR proposes builtin support for `@generated` marker. I have not found a standard for a generated file marker, but: * Facebook [uses `@generated` marker](https://tinyurl.com/fb-generated) * Phabricator tool which was spawned from Facebook internal tool [also understands `@generated` marker](https://git.io/JnVHa) * Cargo inserts `@generated` marker into [generated Cargo.lock files](https://git.io/JnVHP) My personal story is that rust-protobuf project which I maintain was broken twice because of incompatibilities/bugs in rustfmt marker handling: [one](stepancheg/rust-protobuf#493), [two](stepancheg/rust-protobuf#551). (Also, rust-protobuf started generating `@generated` marker [6 years ago](https://git.io/JnV5h)). While rustfmt AST markers are useful to apply to a certain AST elements, disable whole-file-at-once all-tools-at-once text level marker might be easier to use and more reliable for generated code.
This is a copy of #4296 with these changes: * file is not reopened again to find if the file is generated * first five lines are scanned for `@generated` marker instead of one * no attempt is made to only search for marker in comments `@generated` marker is used by certain tools to understand that the file is generated, so it should be treated differently than a file written by a human: * linters should not be invoked on these files, * diffs in these files are less important, * and these files should not be reformatted. This PR proposes builtin support for `@generated` marker. I have not found a standard for a generated file marker, but: * Facebook [uses `@generated` marker](https://tinyurl.com/fb-generated) * Phabricator tool which was spawned from Facebook internal tool [also understands `@generated` marker](https://git.io/JnVHa) * Cargo inserts `@generated` marker into [generated Cargo.lock files](https://git.io/JnVHP) My personal story is that rust-protobuf project which I maintain was broken twice because of incompatibilities/bugs in rustfmt marker handling: [one](stepancheg/rust-protobuf#493), [two](stepancheg/rust-protobuf#551). (Also, rust-protobuf started generating `@generated` marker [6 years ago](https://git.io/JnV5h)). While rustfmt AST markers are useful to apply to a certain AST elements, disable whole-file-at-once all-tools-at-once text level marker might be easier to use and more reliable for generated code.
|
Seems like this was already backported in #4990 |
Related: rust-lang/rustfmt#4296 (cherry picked from commit 3c4bec1)
This PR adds
format_generated_files. When this value isfalse, rustfmt skips formatting generated files. If the value istrue, rustfmt formats generated files.format_generated_filesis set tofalseby default.rustfmt considers files as generated if the file starts with a comment which includes
@generated( e.g.,// @generated).Close #3958.