❎ Exclude files listed in project config from myst build and watch

[fwkoch]

Problems:

• Without an explicit _toc.yml file, the project build pulls in all md/ipynb files from the project directory
• During myst start all files within the directory are watched (potentially an arbitrarily large number of files... enough to break the watch function)

Current state:

• Project config already has an exclude field for a list of files to "exclude" - however, nothing respects those excluded files...

What this pr does:

• Files listed under exclude in the project config are treated as glob patterns. These come into play in two places:
  1. During project build, any md/ipynb that matches an exclude pattern will not be added to the project
  2. During myst start, everything matched by exclude is ignored (so you can ignore folders with lots of unrelated files)
• To make this work, there were some changes to the watch function - primarily triggering a project reload on myst.yml changes
  • (there was also some refactoring to consolidate to a single watch "processorFn" rather than duplicating logic across the old "fileProcessor" and "siteProcessor")

Future steps

• Currently you may only build your project with (1) explicit _toc.yml or (2) implicit directory contents with explicitly excluded files. It would be nice to also be able to have a simpler include or something in the project config to define the toc.
• Further, if you exclude a file, it is not built at all; there is no way to build a file as part of the project but not include it in the project's toc (this is useful for notebook-generated figures - we want to keep the source notebook but not really have it as part of the project toc...)

[fwkoch]

Gotta make sure we update this with await downstream...!

[rowanc1]

Added in curvenote here:
curvenote/curvenote#490

[fwkoch]

There is an edge case here:

When you update exclude in the project config, it will rebuild the project correctly with the new exclude value respected. However, chokidar.watch is not updated, so if you stop excluding a file you will not start watching it, and if you exclude a previously watched file, it will continue to be watched. I don't think this is a big deal... worst case scenario is you just kill your myst start process and call it again to re-watch the correct stuff. (And if it is a problem, we can probably fix it....)

[rowanc1]

Yep, I think it shouldn't be that big a deal that the watch is lagging.

[rowanc1]

Nice, works well!

I think it would be good to have a pass in the future where files that are included are added to the watch. For example, I am working with a lot of tex docs right now, and I would love an auto-generated TOC but also want to see the changes live when we update an included document like a figure or table.