track NoN files when configured dirs are used, clean before each new run

[lindsaylevine]

this is a pass at tracking NoN-moved or NoN-generated files between runs of NoN and cleaning them before a new run.

i don't see a way we can add a "manual" clean option for plugin CLI users. for example, if they build on the CLI four consecutive times, this change should allow them to do that without having to change our file overwrite logic at all. BUT, when they're done building/deploying on the CLI, they will be left with all the NoN files in their .git, as @FinnWoelm pointed out in the original issue that has since been transferred internally. because the plugin will only run on netlify build, i don't think adding a "clean" flag/command to NoN will make a difference; there'd be no way for the plugin to access it outside of a build. these users will have to stash or manually delete these files after finishing their builds/deploys. open to ideas here. this may be a greater CLI issue.

to-do:

• test with plugin
• write cleanup script

[lindsaylevine]

addresses netlify/next-runtime#74

[lindsaylevine]

currently this doesnt work in these ways:

1. if node_modules or node_modules/.cache gets purged between cli builds
   • only way around this right now would be to purge your NoN stuff from your configured dirs; not great
2. as mentioned in the PR desc, NoN files are tracked by .git unless they're deleted after cli builds
   • somehow find a "clean up" option via the plugin
   • could possibly find a way to clean after a deploy? but if they're just "build"ing and not deploying then that would only be a bandaid

[FinnWoelm]

Awesome!! 😊

Your approach is interesting! Initially, when I thought about this feature, I imagined that we could a custom copySync function, that tracks each added file, since (almost?) our logic uses copySync, e.g. Netlify Function setup and Public files setup. copySync could just be a wrapper for fs-extra.copySync(), but extend that functionality with also writing the target file into the cache (or pushing it into an array that gets written to file at the end, although then we need to handle interruptions/crashes that occur during next-on-netlify build — an issue that we don't have when we write to the cache file right away).

But this approach is really neat!! It seems like one downside of this approach vs copySync wrapper is that we might mistakenly tag some files as NoN if they are manually put into the functions folder while NoN is running. But that's an edge case. Downside of copySync wrapper is, of course, that it requires extra work/custom wrapper for each IO operation, such as writeFile, etc... (as we already do for _redirects).

My personal thoughts regarding

1. Loss of cache when purging node_modules: I think we can live with that. Seems like it'll probably be a rare edge case.
2. Clean up option: Would be good to add, I think. For now, this could just be part of next-on-netlify and plugin users could install next-on-netlify if they want/need to take advantage of a dedicated cleanup function. But we could also add this in a second step, IMO. Doesn't necessarily need to be part of this PR :)

[lindsaylevine]

@FinnWoelm issue with your cleanup script idea. we purposely fail the plugin if anyone has next-on-netlify manually installed in their project (https://github.com/netlify/netlify-plugin-nextjs/blob/main/index.js#L42). we could change this, but after playing around with writing a custom script, it seems like an unpleasant bandaid that will need urgent replacing.

[lindsaylevine]

i will be starting conversations internally about improving the CLI experience because i do believe now this is a greater CLI issue and one that could impact other plugins. i think the CLI shouldn't be making direct changes to a project's directory. the entire "build" should run in/be output into a tmp directory to avoid messing with a project directly.

[cassidoo]

This seems like a clever approach!! :)