SIP-51 - Drop Forwards Binary Compatibility of the Scala 2.13 Standard Library

[lrytz]

No description provided.

[sjrd]

Such a change would have a significant impact on the Scala.js and Scala Native releases and version policy. I will talk about Scala.js, but the same applies to Scala Native, IIRC.

Scala.js builds its own version of the scala-library. Its build contains a scalalib project, which fetches the sources of scala-library and recompiles them for Scala.js. That is published as part of scalajs-library. That jar is versioned as artifact name scalajs-library_2.13, version 1.12.0 (for Scala.js 1.12.0). The sources used for that are the sources of scala-library.jar version X, where X is the current main Scala 2.13 version in the Scala.js build (currently, 2.13.10). The jar does not contain .class files (it cannot); it only contains .sjsir files. For separate compilation, the .class files in the original scala-library.jar are used, but for linking the .sjsir files are used.

When Scala 2.13.11 gets released, we cannot change scalajs-library_2.13 version 1.12.0. It will remain forever the version built from the 2.13.10 sources. We only publish a new compiler plugin scalajs-compiler_2.13.11 version 1.12.0.

With the current forward binary compat guarantee, this is not a problem. Users can immediately upgrade their scalaVersion to 2.13.11. The will use the new scalajs-compiler_2.13.11 compiler plugin, and the existing scalajs-library_2.13. Even though their code is compiled against scala-library v2.13.11, it can link against scalajs-library_2.13 v1.12.0, which was built from the sources of 2.13.10.

If we remove the forward binary compatibility guarantee, this model ceases to work. If someone updates their scalaVersion and their code now resolves to a new binary signature that appeared in 2.13.11, linking will fail. They would need to wait for a future version of Scala.js, built against 2.13.11, to be able to upgrade themselves to 2.13.11.

In practice, that will force us to do one of two things:

• Release a new version of Scala.js for every new minor version of 2.13.x, after upgrading internally to that new version. These upgrades are not trivial; they take several hours to complete, plus the CI time + review time, you have to account for 1 or 2 days. Then, 1 more full day to get a new release out. That's just the technical delays, not to mention the added stress for us to be ready to do these things in a timely manner at each release.
• Dramatically change our artifact decomposition and versioning scheme in a way that allows us to back-publish a scalajs-scalalib_2.13.1.jar for an existing Scala.js version. I am not quite sure how the logistics of that would work out. Especially since scalajs-scalalib would have to depend on scalajs-library and vice versa!

/cc @gzm0

[lrytz]

I see, thank you for the explanation.

Why is scalajs-library versioned according to scala-js? Could it be published as org.scala-js:scalajs-library_sjs1:2.13.10?

[gzm0]

Thank you for bringing this to my attention. My thoughts:

First thing that came to mind when I saw this: It feels like requiring the Scala std lib to maintain forward binary compatibility is too restrictive. I think we should try to work towards removing this requirement.

Why is scalajs-library versioned according to scala-js?

Because if it isn't it is impossible for us to publish a Scala.js only bug-fix to a library (we'd have to wait for a new Scala version).

The jar does not contain .class files (it cannot); it only contains .sjsir files. For separate compilation, the .class files in the original scala-library.jar are used, but for linking the .sjsir files are used.

This seems like the problem to me. I assume this is because of how build tools handle compilation? Or does the Scala compiler require the exact version (if yes, then maybe the cross versioning of scalalib was wrong all along?)

That's just the technical delays, not to mention the added stress for us to be ready to do these things in a timely manner at each release.

IIUC this would be the same scenario than during 2.13 milestone releases right? If so, this is not acceptable to me. It had way too much impact to my private life (I believe there are write-ups of this, I can dig them out if you need me to).

[lrytz]

Thanks @gzm0, happy to hear you're supportive in principle!

Because if it isn't it is impossible for us to publish a Scala.js only bug-fix to a library

Does that mean scalajs-library contains more than the .sjsir from compiling the sources of scala-library, i.e., code from other sources that you'd like to be able to fix? If so, could that be split up into a separate artifact?

Or is it about fixes in the scala-js compiler that would produce fixed .sjsir files for the sources of an existing Scala library release?

I see that for other libraries (I looked at scala-xml_sjs1_2.13), the jar contains both .class and .sjsir files. I guess for scalajs-library this needs to be different because the classfiles are already on the classpath through scala-library.jar.

I assume this is because of how build tools handle compilation? Or does the Scala compiler require the exact version?

I'm not sure what you mean exactly; libraries on the compilation classpath (including the Scala library) need to contain classfiles, which contain "pickles" / serialized Scala signatures. Until now, the Scala library on the compiliation classpath always matches the Scala version of the running compiler, but this SIP would change that, and this will work fine (i.e., a 2.13.10 compiler can run with a 2.13.11 library on the compilation classpath).

[sjrd]

Does that mean scalajs-library contains more than the .sjsir from compiling the sources of scala-library, i.e., code from other sources that you'd like to be able to fix? If so, could that be split up into a separate artifact?

Currently, scalajs-library.jar contains things coming from 2 sources:

• The .sjsir files for the scala-library (without .class files)
• The .class files and .sjsir files of the Scala.js-specific standard library (containing notably the things in the scala.scalajs.js package, such as js.Any)

It would be possible to split the scala-library in a separate artifact, for example called scalajs-scalalib.jar. In fact, we already did a similar split in Scala.js 1.11.0 for scalajs-javalib.jar. The difficulty here is that (unlike for scalajs-javalib), scalajs-library and scalajs-scalalib would have to mutually depend on each other. I am not entirely sure what that looks like in Maven artifacts, but it must be doable somehow.

Or is it about fixes in the scala-js compiler that would produce fixed .sjsir files for the sources of an existing Scala library release?

For the scalajs-scalalib part of the above, there are three places from which fixes can come:

• The upstream scala-library sources, obviously.
• Compiler improvements. And in particular, new versions of the IR with cleanups or performance improvements.
• Source file overrides. The Scala.js core repo contains overrides from some files of the scalalib, either for correctness or performance reasons. See https://github.com/scala-js/scala-js/tree/main/scalalib

Because of the latter two, I think we should keep versioning scalajs-scalalib with the Scala.js version number. Also, it's built from the Scala.js core build, and having diverging version numbers within one repo is always a bit of a hassle.

Therefore, the Scala version part of it must be in the artifact name. Something like scalajs-scalalib_2.13.10 version 1.13.0.

If we do that, we will have to tell the tools that, whenever scala-library.jar version X is on the classpath, and the project is a Scala.js project (determined for example by the presence of scalajs-library.jar version Y), we must also put scalajs-library_X.jar version Y on the classpath.

I see that for other libraries (I looked at scala-xml_sjs1_2.13), the jar contains both .class and .sjsir files. I guess for scalajs-library this needs to be different because the classfiles are already on the classpath through scala-library.jar.

Exactly. It is basically impossible to convince tools not to put scala-library.jar on the classpath without them panicking, so we must live with the fact that it's there, and provide sjsir files without class files in a separate artifact.

[lrytz]

Thank you for all the details. So the options are:

1. version scalajs-library only to the Scala version

• the scala-js specific parts of the library cannot be patched in a new scala-js release if the Scala version didn't change

1.a. split up the library, version only the part corresponding to the Scala library to the Scala version

• could patch the scala-js specific library additions in a new scala-js release (but not the scala.js specific overrides of the Scala library)
• requires a circular dependency (some of the patched standard library sources depend on the scala.js additions)

2. version scalajs-library to both the Scala and scala.js version

• requires back-publishing for new Scala releases, potentially for multiple scala.js releases

3. don't change the scalajs-library versioning

• leads to linkage errors
• a new scala.js release is needed for each Scala release

I think options 2 and 3 are not good because of the pressure on the scala.js maintainers.

1a seems the most reasonable solution to me. Maybe the circular dependency can be avoided by including a subset of the scala.js specific additions into the artifact corresponding to the Scala library?

I don't see any other / better alternatives either.

[gzm0]

Rather than requiring the compiler to work with newer standard libraries, have you considered to require downstream projects to be compiled with an equal or newer Scala version than dependencies? We are using this approach in Scala.js for quite a while.

Pros:

• Not require forward compilation classpath compatibility in the compiler
• Not lead to a different scala stdlib than scalaVersion

Cons:

• Library maintainers might not want to upgrade the Scala version since it would force it onto downstream projects (OTOH, this proposal would do this as well for the stdlib, just silently)

[lrytz]

Thanks! I have considered this option. In case there are effectively difficulties making older compilers work with a more recent Scala library, that would certainly be a solution. But I don't think this is going to cause problems.

[gzm0]

FWIW, I have attempted to formalize all of this a bit, but it ended up being very verbose. So I'm not sure it is really useful. In any case, here's the gist: https://gist.github.com/gzm0/d6ec17f8a8fe0d8e4989b1aa75818bd4

[sjrd]

Could we do something like this:

• Separate the scalalib in an artifact scalajs-scalalib.
• Clean its IR like we do for the javalib, so that it contains no reference to scala.scalajs.* (but of course, unlike the javalib, it can refer to scala.*)
• Version it as $scalaV+$sjsV, for example 2.13.11+1.13.1
• scalajs-scalalib would depend on scalajs-javalib, and scalajs-library would depend on the other two.
• The sbt plugin sbt-scalajs would add a dependency on scalajs-scalalib $scalaV+sjsV in addition to scalajs-library $sjsV

This has the following properties:

• No circular dependency.
• scalajs-scalalib 2.13.15+1.13.0 would be prioritized by Ivy resolution over 2.13.10+1.15.0. It may not contain the latest Scala.js-specific optimizations, but it will provide the required binary API for a transitive library that uses Scala 2.13.15.
• When a new Scala version comes out, say 2.13.14, we can back-publish scalajs-scalalib 2.13.14-1.13.0, so that users of the existing Scala.js 1.13.0 can upgrade to Scala 2.13.14 and receive the new binary API.

[lrytz]

Thank you @sjrd, that sounds good to me.

For the other reviewers of this porposal (@gabro, @Kordyjan), I think you don't need to understand the details how Scala.js and Scala Native would need to be adapted. Let's assume will be solution for that (having a solution is of course a precondition for this proposal).

[sjrd]

WiP for Scala.js: scala-js/scala-js#4787

[Kordyjan]

For a long time, I have supported treating the stdlib as any other dependency. However, I wasn't aware of all the complications for the non-JVM targets. This is why I have waited to express my opinion.

The proposed solution for scala.js seems reasonable.
cc @WojciechMazur for the scala native part of the story.

[WojciechMazur]

In the case of Scala Native the dependency tree looks somehow different.
For Scala.js we have the following dependency order: javalib -> scalalib -> library, where the library is the top artifact with dependencies on the other two, and javalib depends on sources of library

In Scala Native it looks a bit different, starting from the bottom of the dependency list:

• nativelib - contains SN specific methods and runtime defitions, eg. the definition of j.l.Object, implementation of primitive arrays. This corresponds to scalajs-library, but does not have any dependencies.
• Layer of native bindings including binding to C, POSIX, and Windows functions
• Javalib - depends on nativelib + bindings
• Scalalib (top artifact exposed to the users, depends on javalib, versioned by SN version)

In Scala Native, we currently don't try to remove references from dependencies in the upper layers. Eg. we try to limit the usage of the Scala standard library in the javalib, but there is no IR cleaner to make sure that this project is completely independent. Instead, it is cross-compiled with each binary Scala version.
As far as I understand this proposal, we should be able to adopt to it without any additional changes, besides changing the versioning scheme.

Version it as $scalaV+$sjsV, for example 2.13.11+1.13.1

Because Dotty's scala3-library is not published in Dotty, but separately in Scala Native project (similarly as it is done currently for Scala 2), we want to introduce similar versioning for our scala3lib (it depends on scalalib produced for Scala 2.13 and excludes all lower layer's of dependencies, replacing them with artifacts compiled with Scala3). It would be required to target Dotty minor versions in a safe manner.

[julienrf]

Note that build tools could provide warnings to say e.g. “scalaVersion has been promoted to 2.13.11 (instead of 2.13.10 as per the scalaVersion key) because it depends on the library foo, which itself depends on Scala 2.13.11.”

[julienrf]

Dear @sjrd, @Kordyjan, and @gabro, in our last SIP meeting we concluded that you would need more information to assess whether this proposal should be accepted or rejected. Do you mind elaborating here on which additional information would allow you to assess the proposal?

[sjrd]

We need to experiment, see what technical solutions are really needed. I'll try and make some progress on scala-js/scala-js#4787 soon-ish.

Someone (tm) should investigate the required changes in sbt.

[julienrf]

Isn’t that part of the “experimental” phase?

[sjrd]

Yes? No? Perhaps. 🤷‍♂️
As far as I'm concerned we could vote on the idea at the next meeting.

[lrytz]

Someone (tm) should investigate the required changes in sbt.

@dwijnand agreed to take a look

[Kordyjan]

Bazel is becoming increasingly popular among companies using scala, so I think it would be great to know if there is no hidden risk there.

@tanishiking: Will you be able to leave a short comments about how bazel is handling stdlib now and do you see any potential problems if we change our compatibility guarantees?

[tanishiking]

Basically, bazel (rules_scala) treats the Scala library in the same way as other dependencies.
However, the behavior around transitive dependencies depends on how the dependencies are downloaded and the dependency_mode in rules_scala.

How Bazel downloads external deps

Repositories / toolchain

Several libraries like scala-library, scalatest, and com.thesamet.scalapb are hardcoded to rules_scala. For these dependencies, resolving transitive dependencies is the responsibility of rules_scala maintainers (and users if they override the artifacts).

Those hard-coded dependencies are automatically added to the classpath. For example, when we define a test target with scala_test(...), Bazel will add scala_library and scalatest to the classpath
(We can override the dependencies from the user side https://github.com/bazelbuild/rules_scala/blob/11a2dd5f3d6b40542df791a043711b9f524077a6/docs/scala_toolchain.md)

http_jar / maven_jar

https://bazel.build/rules/lib/repo/http#http_jar
These are the Bazel embedded rules for downloading deps, but I don't think this is a "standard" way to download dependencies.
If we download deps in this way, the transitive deps won't be included https://bazel.build/docs/external#transitive-dependencies

rules_jvm_external / Bazel deps

Both tools resolve transitive dependencies, rules_jvm_external should have a same dependency resolution behavior as sbt since it uses coursier under the hood.

dependency_mode

rules_scala has a feature called dependency_mode. This mode controls how far Bazel will put transitive dependencies into the classpath, if we set dependency_mode="direct".

dependency_mode = "direct" - only include direct dependencies during compiliation; that is, those in the deps attribute

Therefore, even if the dependent library has a scala-library in its dependency, Bazel won't add its transitive deps (scala-library) to the classpath.

---

That being said, it's possible to run into a linkage error in Bazel in the following cases

• dependencies in rules_scala's repositories (such as scalatest)
• if the projects' third-party deps are managed using http_jar (which I don't think is a famous option)
• if we set dependency_mode="direct"

If stdlibs drop the forward compatibilities,

• rules_scala maintainers have to be careful not to update, for example, scalatest, to have an incompatible version of stdlib
• If people manage the dependencies using http_jar or dependency_mode="direct", users have to be careful not to update dependencies that have an incompatible version of stdlib.

Otherwise, scalalib from resolved transitive dependency will be included into the classpath (same as Gradle).

(FYI @eed3si9n since Eugene would be the best person to talk about dependency resolution in Bazel)

---

Also, FYI, I have a small setup for Bazel + Scala + rules_jvm_external: https://github.com/tanishiking/bazel-tutorial-scala/tree/main/02_scala_maven you can play around with.

[eed3si9n]

There were several premises that motivated this behavior of sbt, which we can list and reevaluate.

1. In general, sbt assumes that users knows what they are doing enough to specify the compiling Scala version. This is encoded as scalaVersion key.
2. sbt assumes that the scala-compiler version (scalaVersion) must match the scala-library version for Scala 2.x.
   1. Before sbt introduced the enforcement, I feel like there were some complaints from the Scala team that they can't reliably predict the scala-library.jar that would be on the classpath during compilation.
   2. As the ultimate variant of the alignment use case, sbt allows users to override scalaOrganization to fork scala-compiler and scala-library together to introduce additional feature to Scala 2.x. See Typelevel Scala and Override scala organization and version transitively at the Ivy level sbt/sbt#2634.

[eed3si9n]

I feel like the relationship between the sbt's alignment behavior and Scala library's compatibility requirement is a bit overstated here.

• The policy of compatibility predates the introduction of alignment in 2016.
• Scala 3 dropped forward compatibility.

Pre-2016, I think Scala 2.x kept forward compatibility, I think because not keeping the forward compatibility means coloring every library with an effective minimum Scala version beyond 0th patch i.e. 2.11.0, 2.12.0, 2.13.0 etc. So if someone wanted to use Maven or just download a bunch of JARs manually and use Ant or something they could use any patch version.

[eed3si9n]

Note prior to sbt 1.3.0 (2019) sbt used Apache Ivy, which emulates Maven's nearest-wins dependency resolution semantics, so it may not have resolved to the latest scala-version found in the transitive graph.

[eed3si9n]

Similar to Scala 3 minor versions 3.1.x, 3.2.x etc, I feel like Scala 2.13.x patch series could drop forward compatibility without modifying sbt's scala-library alignment behavior. To aid the user, we can fail the build with eviction error when greater Scala 2.13 versions is found, similar to what happens with cats-effect 2.x vs 3.x.

[eed3si9n]

Not that I am an expert on rules_scala, I can share a bit of perspective as I've worked with large-scale codebases using Bazel.

For companies using Bazel, Scala 2.13.x dropping forward compatibility likely would be net-positive or neutral. Yes, it might require bumping Scala 2.13 patch versions every now and then, but on Bazel figuring out the dependency version puzzle is already required for literally every other library dependencies anyway, and typically you have a team overseeing the health of monorepo, so if there's a breakage that could be caught at compile-time, Bazel will catch it.

Re dependency_mode, what I am about to reveal might be shocking for non-Bazel users, so beware. In an unhealthy monorepo, it's not uncommon to have 1000 JAR files, if not multiple thousands of dependencies, in part because every directory is a target in Bazel, kitchen sink targets, hand-written copy-pasta etc. It's so many JAR files that compilers or JVM simply going through them is a performance hit. There are multiple ways Bazel, plugins (rules), and users try to mitigate this issue. One is called ijar, which makes up fake JAR file without impl body. Another is this idea sometimes called "strict_deps" to present the compiler with only the direct dependencies. "dependency_mode" in rules_scala is a variant of that. This is really nice because instead of 1000 JARs, we can present compilers and JVM with 20 JARs just to get through the compilation. (As you can imagine Scala presents all sorts of challenges trying to load type info etc) This is a long-winded way of saying requiring "transitive" would be considered a major regression.

dependency_mode

rules_scala has a feature called dependency_mode. This mode controls how far Bazel will put transitive dependencies into the classpath, if we set dependency_mode="direct".

dependency_mode = "direct" - only include direct dependencies during compiliation; that is, those in the deps attribute

Therefore, even if the dependent library has a scala-library in its dependency, Bazel won't add its transitive deps (scala-library) to the classpath.

For the actual dependency resolution, it's difficult to make general statement about Bazel because each company has its own solution for how 3rdparty/jvm graph is resolved. Some people use johnynek/bazel-deps, Twitter had twitter/bazel-multiversion, but one thing that's common is that a single Scala 2.13.x version is selected for the entire monorepo both in terms of scala-compiler and scala-library, so we don't get into a situation where target foo/core:core is built using scala-library 2.13.10, but bar/core:core is built using scala-library 2.13.20. So my guess is that at the timing someone bumps up some 3rdparty dependency version (e.g. cats-effect), they would need to override the Scala library artifacts used by rules_scala, if not Scala version.

[gabro]

I don't have much to add on the technical side; the people already involved are more knowledgeable than I am on the details.

From a SIP process standpoint: I like the proposal, and I haven't seen any strong objection against it in principle.
That said, I see a few open points and different tradeoffs to settle, so I don't think this is ready to be voted on at the next SIP meeting.

In my opinion, the proposal would need to answer the open points raised recently and probably include a more specific idea of how Bazel support would look like.

Once we have that, I think it's fine to vote to promote it to the experimental phase, even though there will be gaps to fill via further experimentation.

[Kordyjan]

I also recommend moving this proposal to the experimental stage, where we can gather more information about potential implementation problems.

[julienrf]

I am merging this PR since the proposal has been voted on and accepted on February 17th. During the experimentation phase, it will still be possible to amend the content of the SIP by creating a new PR.

[anatoliykmetyuk]

As mentioned by @sjrd during the latest SIP meeting, implementing this is on the Scala Center's roadmap for Q4 of 2023.

[sjrd]

FTR, support in Scala.js for this SIP is under review at scala-js/scala-js#4913.

[sjrd]

Update I forgot to mention here: the Scala.js support was shipped in Scala.js 1.15.0.