Tracker and TODOs for Mill 0.11

[lefou]

This thread is meant to collect and track all changes we want to apply before we release Mill 0.11.
We already postphoned some changes to avoid binary compatibility issue. Ideally, we can collect them here, so we don't forget them before the next major release.

[lefou]

• make MultiLogger an non-case class, and make debugEnabled an ctr property

[lefou]

• Don't include resources into compileClasspath. #1811

[lefou]

• Review all places where @deprectated annotations have a since before Mill 0.10.0 and remove then when appropriate

[lefou]

@lolgab I think you mean the rather large deprecated set in scalajslib. I agree, that we can clean that one more drastically. Scala.js is still such a fast moving target, and users will probably accept, that we can't go with all the newer features without sacrifices. Do you mind to create e PR with the deprecated removal of the old worker? I also think, we forgot to deprecate the fastOpt and fullOpt targets in favor of the new fastLinkJs and fullLinkJs targets. We can still deprecate them in 0.10.11.

[lolgab]

I'm also investigating if we can use Ammonite to report deprecation warnings on compilation with something like this.
It would be also nice if we could update the macros to print deprecation messages when you execute a deprecated target from the cli.

[lefou]

We might want to integrate the deprecation info into our Scaladoc annotation. Then, the user can see it with mill inspect.

[lefou]

• Capture @deprecated annotations in Scaladoc mill-moduledefs#4

[lefou]

• ScalaJSModule: Deprecate fastOpt and fullOpt #2324

[lefou]

• Consolidated artifact and platform suffix handling #2025

[lefou]

• Review dependencies
  • utest >=0.8.0 was previously discarded because it dropped support for Scala.js 0.6

[lefou]

• Update utest to 0.8.1 #2253

[lefou]

• Drop support for Scala.js 0.6

[lefou]

@lolgab Can you send a PR?

[lefou]

I started a PR but one tests seems to work differently when run with Scala.JS 1

• Drop Scala.js 0.6 support and removed some deprecated API #2218

[lefou]

Refactor the command line options configuration class.

• Do not inherit from Ammonite config(s)
• Do not use a case class, but a simple class, to be easily extensible in a bin-compat way
• Review options and drop unsupported options
• Review No longer inherit ammonite cli parser config #1960

[lefou]

Consider releasing most mill artifacts with the mill platform, e.g. mill-scalalib_mill0.11_2.13

[lolgab]

What is the reasoning behind this?

[lefou]

The reasoning is, we want to make sure they work within the same Mill ABI. This is probably redundant, as long as we ship all mill packages with each release and include them in the classpath by default. But someday Mill core might be more stable than the specific modules, so we could decouple release cycles, which makes the extra API version necessary. We could also stop bringing them all in the classpath by-default. Once, we want them to load, we could use the same loading mechanism as for external plugins.

On the other side, most ABI is defined by the specific modules, and we most likely will never decouple them from core. I'm unsure. It was just a cause for thought, I guess.

[lolgab]

Consider creating package mill.javalib to contain JavaModule and other Java specific code.

[lefou]

Haha, yeah, Mill already is a great Java build tool. But it uses Zinc under the hood, to compile Java code, so we are using the typical Scala tool chain to build. And placing the Zinc worker into a javalib, with all it's Scala intricacies, wouldn't be that ideal either.

But for example Gradle [...] is one of the most loved Java build tools.

Also, I tend to disagree. Gradle has a large user base, of course, but only a small part of it is loving it. Gradle is really hard to grasp, esp. when things go wrong. There is no static mental model one can build or inspect.

I think a unique selling point of Mill it the easy correspondence to the host language and how easy it is to find the implementation for anything in the build model.

[lefou]

What we probably want is a landing page for Java developers or those looking for a good Java build tool. Beside some examples, we could show some comparisons for those projects in different build tools, so those users who have used one of the others before, get a quick idea how it may feel like to use Mill.

[lolgab]

What we probably want is a landing page for Java developers or those looking for a good Java build tool. Beside some examples, we could show some comparisons for those projects in different build tools, so those users who have used one of the others before, get a quick idea how it may feel like to use Mill.

This is a great idea!

[lolgab]

Would you consider a package object with type aliases for JavaModule and other java specific code under mill.javalib?
It would be nice to have no mention of Scala in java projects :)

[lefou]

Ugh, that is for sure a little maintenance hell. I'd like to avoid it, but on the other side, I'm not sure how many entries it will have. The greatest issue I see is, that this has potential for great confusion. You could provide a POC though, but I'm not convinced. I agree it would be nice to have pure java stuff in a javalib, but this would mean, we produce lots of source incompatibilities.

[lolgab]

• Make ScalaNativeModule use Seq instead of Array for user exposed targets.
  Update Scala Native API and worker #2193

ScalaNativeModule used Array[String] for things like nativeCompileOptions or nativeLinkingOptions because the worker was written in Scala 2.12. Actually it still is using Scala 2.12 for Scala Native 0.4.[0,1,2] but we already did separate the worker internal API from the user facing API in ScalaJSModule and it worked out pretty well. We can do the same for Scala Native so we can expose more idiomatic types like Seq[String]

[lolgab]

I want to propose to drop support for Scala Native 0.4.[0,1,2]. This will simplify the implementation, also I don't think anybody still uses Scala Native 0.4.2 now that there is 0.4.9. If we do I can convert the worker API from Java to Scala, use Scala types directly and reduce the conversions. Also the implementation will look more similar to the Scala.js one.

[lefou]

Yeah, sounds reasonable.

[lefou]

• Move mill.module.Jvm.JarManifest to mill.api.JarManifest, but keep some forwarders (val for object, type for class) to keep source compatibility. Also remove the case from class.

[lefou]

• Move JarManifest to mill.api #2325

[lefou]

The main branch is now open for more fundamental changes, including binary breakers. We now publish snapshot-releases with the 0.11.0-M0 version and will increase the counter after the M (which stands for Milestone) when needed.

Now is the time to update old open PRs and submit new with code cleanups and changes. Feel free to use this tracker discussion to link to ideas or issues, which we should include in Mill 0.11.

[lefou]

• Implement a script runner, that can run a pre-compiled Mill buildfile without Ammonite

[lefou]

Not a blocker for 0.11.

[lefou]

• Realized in Remove Ammonite as a dependency, handle script running and bootstrapping ourselves #2377

[lefou]

• Re-integrate --watch support back into Mill. (We migrated it into Ammonite for Mill 0.10.0.)

[lefou]

Not a blocker for 0.11.

[lefou]

This will be a part of PR

• Remove Ammonite as a dependency, handle script running and bootstrapping ourselves #2377

[lefou]

• Do not change the millSourcePath in Cross modules by default
  • Maybe `Cross` should not change `millSourcePath` #2252

[lefou]

• Do not add Cross matrix items to millSourcePath #2347

[lefou]

• Write a migration / upgrade guide

Topics below:

[lefou]

• Changed default for millSourcePath in Cross modules

[lefou]

• Changed signatures (TODO: review and list them here)

[lefou]

• Changes / improvements in inter-module resolution but API changes in recursiveIvyDeps and trnsitiveIvyDeps, which now use BoundDep instead of Dep
• Changes in CoursierModule.resolveDeps which now wants BoundDeps.

[lefou]

• Review Remove Ammonite as a dependency, handle script running and bootstrapping ourselves #2377 for documentation updates and migration information

[lefou]

• Review Make Target type abstract to allow overriding by different concrete implementations #2402

[lefou]

• Support recovery from failed task dependencies #1779

Probably unrealistic, to get this one into 0.11 as it's a harder task, but very useful. It will definitely break compatibility.

[lefou]

• Move mill.modules.Assembly to mill.scalalib.Assembly and keep source-compat forwarders.

[lefou]

• See Kill mill.modules package #2513

[lefou]

• Unify MillBuildModule and MillBuildRootModule
  • Unify MillBuildModule with MillBuildRootModule #2414
  • Re-use MillBuildRootModule for BSP #2415

[lihaoyi]

@lefou what do you think about tagging 0.11.0-M8 and re-bootstrapping the com-lihaoyi/mill repo on top of it soon? I would like to begin dogfooding the changes, especially #2377, to try and surface any issues that may not get caught by our test suite. It would also let us clean up our build.sc and reset mill-bootstrap.patch, which is growing a bit large (especially once #2425 lands)

• Release Mill 0.11.0-M8
• Re-Bootstrap Mill build based on Mill 0.11.0-M8

[lefou]

That is the plan, but I wanted to merge all known regressions first.

• Fix BSP regressions and improved resource file handling #2420

[lihaoyi]

Sounds good. Looking forward

[lefou]

• Prepare changelog

[lefou]

• Restore the ability to run Mill with custom repositories which require authentication (we removed predef script support)

[lihaoyi]

@lefou seems Coursier now supports COURSIER_CREDENTIALS as an environment variable and a ~/.config/coursier/credentials.properties config file

• https://get-coursier.io/docs/other-credentials

Maybe we can just tell people to use that when they have custom repositories that need auth?

[lefou]

Would be nice, if someone could verify, that this really works. There are other issue with configuring coursier, e.g.

• can not config repository mirror via ~/.config/coursier/mirror.properties #2312

so we should check it and explicitly document, what's supposed to work and what not.

[lihaoyi]

I did some local testing using reposilite; it appears to be successful at redirecting to a local mirror and using credentials:

lihaoyi 1-simple-scala$ COURSIER_CREDENTIALS="localhost test-name:tst-secret" COURSIER_REPOSITORIES="http://localhost:8080/private" /Users/lihaoyi/.cache/mill/download/0.11.0-M8-34-19ec00-DIRTY56d084cd -i foo.compile
[build.sc] [4/49] mill.scalalib.ZincWorkerModule.classpath
1 targets failed
mill.scalalib.ZincWorkerModule.classpath
Resolution failed for 1 modules:
--------------------------------------------
  com.lihaoyi:mill-scalalib-worker_2.13:0.11.0-M8-34-19ec00-DIRTY56d084cd
	unauthorized: http://localhost:8080/private/com/lihaoyi/mill-scalalib-worker_2.13/0.11.0-M8-34-19ec00-DIRTY56d084cd/mill-scalalib-worker_2.13-0.11.0-M8-34-19ec00-DIRTY56d084cd.pom (Reposilite)

--------------------------------------------

For additional information on library dependencies, see the docs at
https://com-lihaoyi.github.io/mill/mill/Library_Dependencies.html

lihaoyi 1-simple-scala$ COURSIER_CREDENTIALS="localhost test-name:test-secret" COURSIER_REPOSITORIES="http://localhost:8080/private" /Users/lihaoyi/.cache/mill/download/0.11.0-M8-34-19ec00-DIRTY56d084cd -i foo.compile
[build.sc] [4/49] mill.scalalib.ZincWorkerModule.classpath
1 targets failed
mill.scalalib.ZincWorkerModule.classpath
Resolution failed for 1 modules:
--------------------------------------------
  com.lihaoyi:mill-scalalib-worker_2.13:0.11.0-M8-34-19ec00-DIRTY56d084cd
	not found: http://localhost:8080/private/com/lihaoyi/mill-scalalib-worker_2.13/0.11.0-M8-34-19ec00-DIRTY56d084cd/mill-scalalib-worker_2.13-0.11.0-M8-34-19ec00-DIRTY56d084cd.pom

--------------------------------------------

For additional information on library dependencies, see the docs at
https://com-lihaoyi.github.io/mill/mill/Library_Dependencies.html

Will need to try again post-release to verify it actually can download, but I don't have any reason to believe it won't. Given the change in error message from unauthorized to not found, it's clear that the two environment variables are taking effect

[lefou]

• Make meta builds (mill-build/build.sc) opt-in and configurable #2461

[lihaoyi]

@lefou I think I've gotten in most of the major changes I wanted to get in for 0.11.0. Shall we tag 0.11.0-M9? Then we can rebootstrap, roll it out across com-lihaoyi, and let it bake a bit before 0.11.0 final

[lefou]

This is the way. 😉

[lihaoyi]

Ok, I pushed tag 0.11.0-M9

[lihaoyi]

@lolgab @lefou 0.11.0-M9 should be out, if you can help publish mill-mima and mill-vcs- version we can rebootstrap

[lolgab]

mill-mima 0.0.20 was released with support for Mill 0.11.0-M9

[lihaoyi]

@lefou @lolgab I tagged 0.11.0-M10, with another set of fixes for rough edges I discovered when trying to roll out 0.11.0-M9 throughout com-lihaoyi. If you could publish mill-mima and mill-vcs-version, then I'll proceed with updating the rest of com-lihaoyi to 0.11.0-M10

[lefou]

• Update Mill 0.11 API to 0.11.0-M10 lefou/mill-vcs-version#112 => Version 0.3.1-8-37c08a

[lolgab]

• Update Mill 0.11 to 0.11.0-M10 lolgab/mill-mima#109 => Version 0.0.21

[DamianReeves]

I've been playing with 0.11.0-M9 and I like the changes. I know @lolgab mentioned it in the PR for the cross Platform stuff, but I want to reiterate the enable functionality like what mill-cross-platform provides would be really beneficial.

[lihaoyi]

I agree, I'd like to get that in somehow. Last I looked I couldn't figure out a way to make it fit nicely, but maybe with some more thought I can find a way

[lihaoyi]

Last few PRs for 0.11.0 appear to be the ones I merged below

1. issue-408 - server timeout setting configurable through envs #2550
2. Update thirdparty tests, make subprocess testrunner run user code in root classloader #2561
3. Enable buildInfoStaticCompiled on ScalaJSModule and ScalaNativeModule #2562

I'll also try to get a PR in to upload a doc-site bundle, for the versioned-documentation support in 0.11+.

One thing that is worth reciewing before the release is the packages for which we provide compatibility guarantees for. I listed them out in https://github.com/com-lihaoyi/mill#compatibility-stability, rather than have them be implicitly derived from the build configuration. If we want to take bincompat seriously, we should be explicit what we are (and are not) promising.

I'm going to tag 0.11.0-M11 on f944ceb96565385953ba7fe6869100da24025b03 Sun Jun 4 06:43:06 2023 -0700 Update thirdparty tests, make subprocess testrunner run user code in root classloader (#2561), and ask that @lefou and @lolgab release mill-vcs-version and mill-mima again for that milestone. I will then update all the mill-upgrade PRs for the other com.lihaoyi libraries to use the latest code before we tag 0.11.0 final, giving me a chance to catch any last-minute issues as well as helping me draft a migration guide given the com-lihaoyi libraries as examples. Especially the removal of Tests traits will require some updates to the existing mill-upgrade PRs

Still aiming for 0.11.0 final to be released on Tuesday morning PST (6 June), so I can advertise it at Scaladays Seattle (7 June), as discussed in Discord.

[lefou]

mill-vcs-version version 0.3.1-14-7e2bd2 supports Mill 0.11.0-M11.

[lolgab]

mill-mima version 0.0.22 supports Mill 0.11.0-M11

[lihaoyi]

Thanks! I'll update the open mill-upgrade PRs to use M11

[lihaoyi]

Merged the last few PRs and tagged 0.11.0.

• Remove mill._ magic import from builds #2418
• Bump Scala to 2.13.11 and use pre-published bridges #2575
• Add mill.define.DynamicModule to allow custom overriding of millModuleDirectChildren #2574
• Renamings for conciseness: Scala*ModuleTests -> Scala*Tests, InnerCrossModule -> CrossValue #2572

If no flakiness, it should go out in a few hours after it makes its way through CI

[lihaoyi]

@lefou @lolgab I started writing Major changes that may require migration notes in the release https://github.com/com-lihaoyi/mill/releases/tag/0.11.0. Please help flesh it out with anything I may have missed, and we can use that as the source of truth for the migration notes.

What do you guys think of moving the changelog out of readme.adoc into the Github Releases pages? That's what most projects seem to be doing nowadays, and it may be more familiar to users and give them a single page we can link to for all release-related information

[lefou]

Well, as a user GitHub Release Pages are nice. From a long time maintenance perspective, I prefer a document in the repository. I'm doing Open Source for 20+ years and I've seen many hosting platforms come an go. I also migrated many project over different platform. Keeping essential info in the repo makes those migrations much easier and older/historic information for users more accessible. Currently GitHub is free to use, but nobody can guarantee it.

The nice thing is, that we can link from the release pages to a changelog or readme file, and even Scala Steward nowadays supports dedicated changelog documents. Also, we can easily render the changelog as part of our documentation.