Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip formatting generated files #4296

Merged
merged 1 commit into from
Jul 10, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions Configurations.md
Original file line number Diff line number Diff line change
Expand Up @@ -964,6 +964,14 @@ fn add_one(x: i32) -> i32 {
}
```

## `format_generated_files`

Format generated files. A file is considered as generated if the file starts with `// @generated`.
calebcartwright marked this conversation as resolved.
Show resolved Hide resolved

- **Default value**: `false`
- **Possible values**: `true`, `false`
- **Stable**: No

## `format_macro_matchers`

Format the metavariable matching patterns in macros.
Expand Down
2 changes: 2 additions & 0 deletions src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,7 @@ create_config! {
inline_attribute_width: usize, 0, false,
"Write an item and its attribute on the same line \
if their combined width is below a threshold";
format_generated_files: bool, false, false, "Format generated files";

// Options that can change the source code beyond whitespace/blocks (somewhat linty things)
merge_derives: bool, true, true, "Merge multiple `#[derive(...)]` into a single one";
Expand Down Expand Up @@ -615,6 +616,7 @@ blank_lines_upper_bound = 1
blank_lines_lower_bound = 0
edition = "2018"
inline_attribute_width = 0
format_generated_files = false
merge_derives = true
use_try_shorthand = false
use_field_init_shorthand = false
Expand Down
7 changes: 6 additions & 1 deletion src/formatting.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ pub(crate) use syntux::session::ParseSess;
use crate::config::{Config, FileName};
use crate::formatting::{
comment::{CharClasses, FullCodeCharKind},
generated::is_generated_file,
modules::{FileModMap, Module},
newline_style::apply_newline_style,
report::NonFormattedRange,
Expand All @@ -30,6 +31,7 @@ mod chains;
mod closures;
mod comment;
mod expr;
mod generated;
mod imports;
mod items;
mod lists;
Expand Down Expand Up @@ -125,7 +127,10 @@ fn format_project(
parse_session.set_silent_emitter();

for (path, module) in &files {
let should_ignore = !input_is_stdin && parse_session.ignore_file(&path);
let should_ignore = (!input_is_stdin && parse_session.ignore_file(&path))
|| (!config.format_generated_files()
&& is_generated_file(&path, original_snippet.as_ref()));

if (!operation_setting.recursive && path != &main_file) || should_ignore {
continue;
}
Expand Down
40 changes: 40 additions & 0 deletions src/formatting/generated.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
use std::{
fs,
io::{self, BufRead},
};

use crate::config::file_lines::FileName;
use crate::formatting::comment::contains_comment;

/// Returns `true` if the given span is a part of generated files.
pub(super) fn is_generated_file(file_name: &FileName, original_snippet: Option<&String>) -> bool {
let first_line = match file_name {
FileName::Stdin => original_snippet
.and_then(|s| s.lines().next())
.map(str::to_owned)
.unwrap_or("".to_owned()),
FileName::Real(ref path) => fs::File::open(path)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not ideal to have another file system read (IIRC we're up to at least 3 due with this plus the parser and one for newline checks) 🤔

Would it be feasible to get the first line without the file system hit, maybe via the parse session data? Pperhaps it won't matter that much given the default is false, and/or could potentially be updated later

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(forgive me for the intrusion, i was recently doing work in this area and learned this: unless you're very low on memory, recently-accessed files are cached by the kernel and effectively free to access; the slowest part will be parsing)

Copy link
Member

@calebcartwright calebcartwright Jul 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

forgive me for the intrusion

no worries at all!

the slowest part will be parsing

agreed, that's basically a given, and understand the main point about not strictly hitting the fs. though the intent of my original question wasn't really about a parsing/AST construction vs. additional file reads, it's whether the additional file read is necessary given that the file's already been read and loaded from the preceding parsing stage and available for use

unless you're very low on memory, recently-accessed files are cached by the kernel and effectively free to access;

do you know if that (especially the effectively free bit) holds true everywhere, for example Windows, CI environments, some of the tier 2/3 targets, etc?

I'm not terribly familiar in that space, so entirely possible this approach is better than an alternative like using the span from the associated mod to get the first line from the snippet for the file, either more performant/negligible difference due to the caching or other reasons, and/or simpler from a code readability and management perspective 🤷‍♂️

Also readily admit such a question may be unnecessary/premature optimization, but figured I'd ask anyway 😄

minor edits for clarity

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I agree that reusing data that is already and easily available is preferable. I just wanted to note that the performance implications of rereading a file may not be as high as it may appear, and that there are probably bigger bottlenecks. But I totally agree with you, no need to do extra work if you don't have to.

Re environments: most modern kernels (~last 20 years) have a page cache; this applies to windows as well

Copy link
Member

@calebcartwright calebcartwright Jul 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re environments: most modern kernels (~last 20 years) have a page cache; this applies to windows as well

Sure, but I suppose I should've phrased my question better.

Was wondering more so if you thought there'd always be sufficient available memory to guarantee/near-guaranatee cache hit in all scenarios. I know even on my laptops I can get really memory constrained (as I tend to use lower end machines and still go overboard with multi tasking 😆), but a lot of CI environments can be pretty low on resources and one of the things I learned during the conversations on the compiler frontend for rls/ra is that there's plenty of folks doing their inner dev loop on some pretty resource constrainted environments, and even some remote workflows.

I genuinely don't know, but wouldn't be surprised if running in some of the more resource constrained environments (or those that purged more aggressively) and/or larger workspace projects could result in some cache misses and i/o hits.

I'm quickly getting out my depth though and will happily defer to @topecongiro on whether it's worth considering an alternative 😄

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I first tried by using the span of ast::Mod, though it turned out that the span may not contain the comment at the beginning or the end of the file. I think that there are ways to get a correct span, though I opted for a simpler implementation this time.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me!

.ok()
.and_then(|f| io::BufReader::new(f).lines().next()?.ok())
.unwrap_or("".to_owned()),
};

is_comment_with_generated_notation(&first_line)
}

fn is_comment_with_generated_notation(s: &str) -> bool {
contains_comment(&s) && s.contains("@generated")
}

#[cfg(test)]
mod test {
#[test]
fn is_comment_with_generated_notation() {
use super::is_comment_with_generated_notation;

assert!(is_comment_with_generated_notation("// @generated"));
assert!(is_comment_with_generated_notation("//@generated\n\n"));
assert!(is_comment_with_generated_notation("\n// @generated"));
assert!(is_comment_with_generated_notation("/* @generated"));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider adding a test for string w/o @generated or not on the first line.

}
}
2 changes: 1 addition & 1 deletion src/test/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -744,7 +744,7 @@ fn read_significant_comments(file_name: &Path) -> HashMap<String, String> {
reader
.lines()
.map(|line| line.expect("failed getting line"))
.take_while(|line| line_regex.is_match(line))
.filter(|line| line_regex.is_match(line))
.filter_map(|line| {
regex.captures_iter(&line).next().map(|capture| {
(
Expand Down
8 changes: 8 additions & 0 deletions tests/source/configs/format_generated_files/false.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
// @generated
// rustfmt-format_generated_files: false

fn main()
{
println!("hello, world")
;
}
8 changes: 8 additions & 0 deletions tests/source/configs/format_generated_files/true.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
// @generated
// rustfmt-format_generated_files: true

fn main()
{
println!("hello, world")
;
}
8 changes: 8 additions & 0 deletions tests/target/configs/format_generated_files/false.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
// @generated
// rustfmt-format_generated_files: false

fn main()
{
println!("hello, world")
;
}
6 changes: 6 additions & 0 deletions tests/target/configs/format_generated_files/true.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
// @generated
// rustfmt-format_generated_files: true

fn main() {
println!("hello, world");
}