java 6 publishing no longer possible

[ashawley]

In June 2018, Maven Central updated their servers to only support newer SSL/TLS protocols, so a Java 6 build will be unable to publish.

Moving builds to Java 7 or 8 could have unintended consequences, since there may be changes in the Java class library between versions, but what else can be done?

See Lukas Rytz comment in this issue where it was originally discovered:

scala/scala-collection-compat#77

[SethTisue]

I see at scala/scala-parser-combinators#159 that it isn't only publishing that's affected, it's also even just retrieving artifacts over https

[ashawley]

Is it possible to do the build in Java 6, but then do a separate publish step in Java 8 and skip rebuilding the Jar on the second call?

$ env JAVA_HOME=$(/usr/libexec/java_home -v 1.6) sbt ++2.11.12 test package
$ env JAVA_HOME=$(/usr/libexec/java_home -v 1.8) sbt ++2.11.12 publish

I don't know enough sbt to know what is and isn't possible.

[lrytz]

That's not straightforward, unfortunately, because sbt is very careful about invalidating previously compiled artifacts if some dependency changes (for example a jar on the classpath, like the java standard library). Also, Seth points out that the Java 6 build doesn't even manage to fetch artifacts (if they are not in the cache), so we could not build on 6 reliably. I'm really not sure how we can keep building, or even testing on 6.

[ashawley]

Are there any tricks in sbt to dismantle those dependencies between tasks?

I've seen there is a skip syntax:

set skip in packageBin := true

And then there's a syntax for unhooking things

set packageBin in publish := {}

Is there also a way to fake the classpath is the same?

[SethTisue]

perhaps some add-on is available for Java 6 that addresses the problem?

[lrytz]

I've tried skip, but it only worked for in Compile, nothing else, in sbt 0.13.x. Possibly that's better in 1.x.

[ashawley]

It looks like Travis uses OpenJDK 1.6u41, released on 2012. By comparison, Oracle released u201 of their JDK in April 2018.

Ubuntu tries to backport security fixes periodically. They have done that up to u121 which is July-2016.

https://launchpad.net/ubuntu/+source/openjdk-6

Coincidentally, there were some TLS protocol changes in u121. However, I haven't found any evidence that the JDK-8133817 change was backported or not.

http://www.oracle.com/technetwork/java/javase/documentation/overview-156328.html#R160_121

TLS v1.2 support now available

TLS v1.2 is now a TLS protocol option with the release of JDK 6u121. By default, TLS v1.0 will remain the default enabled protocol on client sockets.

As an example, both the TLS v1.1 and TLS v1.2 protocols can be enabled for use on SSL/TLS connections via SSLSocket/SSLEngine/SSLServerSocket APIs:

sslSocket.setEnabledProtocols(new String[] { "TLSv1.1", "TLSv1.2"}); 

or by setting up and using a TLS v1.2 based SSLContext :

SSLContext ctx = SSLContext.getInstance("TLSv1.2"); 

or by using the SSLParameters API:

sslParameters.setProtocols(new String[] {"TLSv1.1", "TLSv1.2"}); 

The new jdk.tls.client.protocols System Property may also be used to control the protocols in use for a TLS connection (JDK-8151183).
One may launch their application with this property. E.g. java -Djdk.tls.client.protocols="TLSv1.2" will enable only TLS v1.2 on client SSLSockets.

Note that protocol versions specified via the new jdk.tls.client.protocols property will suppress any value set via the jdk.tls.client.enableSSLv2Hello property. SSLv2Hello can be passed to the jdk.tls.client.protocols value if necessary.

[lrytz]

Aha, interesting! Any idea where we could find OpenJDK 6 builds? The Oracle JDK downloads are way behind, I guess they make the newer ones only available to subscribers. I looked at https://github.com/shyiko/jabba/blob/master/index.json#L85-L90, they are not there..

[ashawley]

In a blog post on this issue, there's a fourth option, that suggests using HTTP.

The last entry at the bottom of the FAQ suggests that HTTP will always exist one of the hosts.

I cannot implement the above required changes in my environment -- what are my options?

We don't recommend this, but http://repo1.maven.org is, and will continue to be supported in the long term.

Is there a way to have sbt use the HTTP protocol for Maven?

[ashawley]

I tried using -Dsbt.repository.secure=false, but it seems that not all the repositories are parameterized with this setting. In particular, it seems typesafe-ivy-releases and sbt-ivy-snapshots are hard-coded to use https:

  ==== Maven Central: tried
    http://repo1.maven.org/maven2/org/scala-sbt/sbt/0.13.17/sbt-0.13.17.pom
    -- artifact org.scala-sbt#sbt;0.13.17!sbt.jar:
    http://repo1.maven.org/maven2/org/scala-sbt/sbt/0.13.17/sbt-0.13.17.jar
  ==== typesafe-ivy-releases: tried
    https://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt/0.13.17/ivys/ivy.xml
  ==== sbt-ivy-snapshots: tried
    https://repo.scala-sbt.org/scalasbt/ivy-snapshots/org.scala-sbt/sbt/0.13.17/ivys/ivy.xml

[SethTisue]

isn't it only Maven Central that's now rejecting the old-style https connections?

[ashawley]

It seems to be a different error from Maven Central

:::: ERRORS
  Server access Error: Connection reset url=https://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt/0.13.17/ivys/ivy.xml
  Server access Error: Connection reset url=https://repo.scala-sbt.org/scalasbt/ivy-snapshots/org.scala-sbt/sbt/0.13.17/ivys/ivy.xml

[SethTisue]

I see at scala/scala-xml#247 you've found a way forward here 👍

looks like we should copy this for the other module repos

[ashawley]

I'm not sure how publishing works to bintray, but if it's HTTPS, there will be trouble.

[SethTisue]

now that we know (from scala/scala-xml#247) that we can make artifact retrieval work on JDK 6, I suggest the following plan:

• include JDK 6 in the build matrix, but only for building and testing, not publishing
• actually do the publishing on JDK 8 instead, to avoid any problems with https

how's that sound?

[ashawley]

I dug up an Oracle u121 Java 6 and ran it from CentOS Linux and found it was able connect to Maven servers over HTTPS if I enabled the TLS 1.2 option with jdk.tls.client.protocols.

[ashawley]

actually do the publishing on JDK 8 instead

What's the trick to doing this from sbt and in Travis? It's not clear to me.

[ashawley]

https://access.redhat.com/solutions/2893541

Does OpenJDK 6 support TLS 1.2 ?
Can you please help answer if TLS v1.2 is supported in Open JDK 1.6 ?
If so, in which exact update version of Open JDK 1.6 is it available from ?

The answer isn't shown, but I predict it is to use openjdk 7.

[SethTisue]

scala-collection-compat took a different approach to this: scala/scala-collection-compat#83

not sure if it's the best way forward. all the discussions on this (here, in scala-xml, in scala-parser-combinators, and this scala-collection-compat thing I just found) have gotten rather bewildering

[jrudolph]

Does OpenJDK 6 support TLS 1.2 ?

According to the OpenJDK sources at http://hg.openjdk.java.net/jdk6/jdk6/jdk/file/706392d93e02/src/share/classes/sun/security/ssl/ProtocolVersion.java#l84 it seems TLS1.2 support was never backported from Oracle JDK 6 to OpenJDK 6.

The elephant in the room here is why Java 6 support would still be relevant at all. Just downloading and installing a somewhat recent Java 6 JDK is quite hard. Oracle doesn't provide Java 6 downloads any more, Ubuntu doesn't ship any Java 6 binaries since several years. Azul still seems to provide somewhat recent ones at https://www.azul.com/downloads/zulu/zulu-linux/.

[ashawley]

Sorry this wasn't clear, but Java 6 is the build target for Scala 2.11, so that's why Java 6 is relevant. So the subtext in the discussion is about supporting Scala 2.11, and not really about Java 6.

Oracle is still supporting Java 6 on their proprietary channel. They are planning one more security fix this year, and technically will continue to support it for a few more years. Indeed, most community projects have dropped it, but there are probably Java 6 projects running somewhere and I'm sure there are a few Scala 2.11 projects still in the wild (we know of Apache Spark!).

[jrudolph]

Sorry this wasn't clear, but Java 6 is the build target for Scala 2.11, so that's why Java 6 is relevant. So the subtext in the discussion is about supporting Scala 2.11, and not really about Java 6.

Ah, so you mean, that because Scala 2.11 is still active and guarantees Java 6 support, the scala modules that were split off from the main project must also provide the same guarantees?

Because, if you look elsewhere, other libraries dropped Java 6 support a while ago, while still providing artifacts for Scala 2.11 that probably mostly still work with Java 6 because Scala 2.11 still targets Java 6. The only thing that could happen is that you introduce references to newer JDK APIs (or accidentally compile java files without setting the target).

I have no doubts that some people are still using Java 6 but that doesn't mean that they still have to be supported by community projects. Right now it seems that Java 6 support introduces serious friction into the whole process of going forward. That it's hard to get hold of a somewhat recent, free Java 6 implementation (and these are insecure enough that they cannot access modern https sites) is a pretty strong signal for me that using Java 6 should be discouraged by now and active support be dropped.

Yes, it's a somewhat sudden cut. But there are good reasons to do it now. People depending on Java 6 can still use the old artifacts. And, even when building with Java 8, the artifacts built with Scala 2.11 would probably still be usable with Java 6.

[JLLeitschuh]

Regardless of how you fix this, all releases made with dependencies loaded over HTTP need to be treated as suspect and need to be rebuilt to ensure that they weren't maliciously compromised. You can do so by comparing the checksums of artifact built safely and the one that was published unsafely.

[ashawley]

Yeah, if openjdk6 won't be updated, and Travis won't use Oracle JDK6, and we can't publish some other way, then the only other option is publishing JDK8 artifacts for Scala 2.11.

For Scala XML, I'm probably going to just bid sayonara to the 2.11 build.

[SethTisue]

it's pretty hard to contrive a scenario where we've built-and-tested on JDK 6 so we know that works, we publish from JDK 8, but then something somehow breaks on JDK 6. I can contrive one if I try (e.g. if JDK 8 added something to stdlib, causing the meaning of some code to change because on JDK 6 we had an extension method of the same name, and it manages to compile and pass testing on both JDKs, yet still not be 100% right).... but it just seems farfetched

[SethTisue]

hmm, what about building on JDK 6, but then switching to JDK 8 for the artifact-uploading part only? could be an appealing path if it were easy, but I suspect it's not easy, because once you switch JDKs and fire up sbt again, sbt will notice that the classpath is different and will want to re-build before uploading

[JLLeitschuh]

hmm, what about building on JDK 6, but then switching to JDK 8 for the artifact-uploading part only?

This leaves that build open to having malicious remote code execution loaded into the OS for the Java 6 phase that is then able to maliciously compromise the release made running Java 8.
Allowing any untrusted code to be loaded and run before a release artifact is produced is a security vulnerability.

[ashawley]

Yeah, it wouldn't be hard to retrieve the artifacts first with JDK8 so that you had some level of security, before making the JDK6 build. In hindsight, we could have done that.

The irony is most of the artifacts are already cached by Travis, or at least they advertise they should be.

[JLLeitschuh]

@ashawley You need to delete that Travis cache, those artifacts cached on that were downloaded insecurely may be compromised.

[ashawley]

I read up on Travis caching, and I was incorrect, it creates a new and separate cache for every build configuration, and that includes the branch or tag name.

[ashawley]

it's pretty hard to contrive a scenario where we've built-and-tested on JDK 6 so we know that works, we publish from JDK 8, but then something somehow breaks on JDK 6.

This isn't my expertise, but besides the source compatibility issue sneaking in that you mention, wouldn't building 2.11 from JDK 8 produce "unsupported class file version" for people running 2.11 from Java 6?

[JLLeitschuh]

Does the Scala compiler support compiling to target older versions of the JVM? I know that javac does by allowing you to pass flags.

[SethTisue]

wouldn't building 2.11 from JDK 8 produce "unsupported class file version" for people running 2.11 from Java 6?

no. the 2.11 compiler produces Java 6 friendly bytecode by default, regardless of what JDK you're on. the only gotcha here is accidental use of Java 7/8 only APIs

Does the Scala compiler support compiling to target older versions of the JVM?

2.11 doesn't have -release

(and, it's not directly relevant here, but to fully answer your question: 2.12 does have -release like javac, but it doesn't go back any further than 8. this could become relevant in some future where we want to start building on 11 or later for users on 8. I think that scenario is some years off)

[ashawley]

Ah, ok. It seems like we should just use JDK8 for 2.11, then. Sorry, I was so ignorant about this. It's 2019, and I forgot about all these compatibility details. Maybe I never really knew it in any detail, either.

[SethTisue]

no worries. it's complicated

[JLLeitschuh]

Any updates on this?

Just a heads up. I do plan on going public about this and other similar incidents in other projects on June 10th, 2019.

[ashawley]

I'm going to drop Scala 2.11 and JDK6 for scala-xml. Please take a look at scala/scala-xml#290.

[SethTisue]

the only other remaining affected repo I could find is scala-continuations, where I've just submitted scala/scala-continuations#47

[SethTisue]

that PR and the scala-xml one are now merged, so I think we can put this to rest.

thanks @JLLeitschuh for keeping after us about it :-)

[JLLeitschuh]

@SethTisue Can you follow up with the Lightbend security team about this impacting other Scala/Lightbend projects?

Also, is there any plan to audit old releases? If there isn't a plan to audit all previous releases, they should a CVE issued for them.

Here's a similar CVE for Eclipse hawkBit that was requested by Wayne Beaton of the Eclipse Security Team due to my discovering this vulnerability in several of the Eclipse repositories.
They chose not to audit previous releases, so a CVE for all previous releases was issued.
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-10240

[SethTisue]

I've passed that along to my team lead (Adriaan) and to [email protected]

[JLLeitschuh]

@SethTisue Any followup on the CVE? Do I need to reach out directly or should I just file for it myself?

[raboof]

I don't think the CVE database is intended for these kinds of issues.

You correctly identified an attack vector through which the build process could have been compromised. That is a very valuable finding, for which we are grateful. If there are any remaining issues we are eager to fix them as well.

However, it only means artifacts that have been published in the past may have been compromised. I don't think the CVE database is meant for recording that artifacts may have been compromised, if there is no direct reason to assume they actually have.

I understand https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-10240 provides some precedent for issuing CVE's for artifacts that may have been compromised without any particular indication that they have. Perhaps we should seek clarification from the CVE Board on this question? Or are you aware of any existing statements around this?

[JLLeitschuh]

Hi @raboof,
I asked them directly and never got a response.
Here are the CVE numbers that have since been successfully filed with respect to this same vulnerability:

• https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-10248
• https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-11404
• https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-11405
• https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-10240

There are more to come (see my recent research):

• https://github.com/issues?utf8=%E2%9C%93&q=author%3AJLLeitschuh+archived%3Afalse+sort%3Aupdated-desc+%5BSECURITY%5D+

[JLLeitschuh]

This may help on an audit if you want to perform one.
https://reproducible-builds.org/tools/