semanticdb files are deleted on every compilation

[lolgab]

When using metals I noticed that the semanticdb files where deleted on every compilation and the out/semanticDbData.dest/data/META-INF/semanticdb/src/main/scala/app contained only the file from the last compilation.
I don't know if this has negative consequences for Metals but it doesn't seem right.
Files are deleted here

[lefou]

IIRC, that target dir gets recreated (by copying) from another directory, which is managed by zinc. So, even when compiling incrementally, it should contain the full set of files. I might be mistaken though. I'll review the code and the tests. Maybe, something (seemingly unrelated) has changed?

[lefou]

To check:

• Is recreation of the directory an issue? Can we add/replace/remove per files instead by the whole directory?
• Is the set of semanticDb files complete, even after incremental compilation?
• Do Java / Scala / Mixed modules behave equally?

[lefou]

@lolgab Do you already have a reproducer by any chance?

[lefou]

I wrote incremental tests, but couldn't reproduce any issues with missing files yet.

• See Fixed semanticdb file copying issue, added more tests #3080

[lefou]

Looks like my tests never compile incrementally, although they should.

[lolgab]

This also breaks Scalafix when used together with incremental compilation. It works again after running mill clean.

[lefou]

I identified some issue but also found some possible bugs, e.g. we moved files instead of copied them. I fixed the copy issue in PR #3080.

Here are the other issues:

Here are some issues I identified with our current approach to generate semanticdb data in a Scala 2.13 module:

• When we stop scalac after phase semanticdb-typer, zinc is not able to incrementally compile the next round, therefore we currently always generate all semanticdb files in each run

• If we don't stop scalac after phase semanticdb-typer, we can generate semanticdb data incrementally, but we also unnecessarily compile all scala files to class files

• Incremental compilation doesn't seem to clean semanticdb files for removed sources. So, we need to manually remove those that belong to non-existing source files