JPMS module names, super-package, and package renaming

[JohnLCaron]

It would be ideal to have JPMS module names follow the "super-package" naming convention, meaning that all code in the module lives under a single package, and no other module has code in that package or subpackage.

But we have legacy code where thats not true. In the list below,

• leave out packages expected to go away
• asterisks show modules that shouldnt be a big deal to change

array
ucar.array

bufr*
ucar.nc2.iosp.bufr

cdm-core
thredds.client
thredds.featurecollection
thredds.filesystem
thredds.inventory
ucar.nc2
ucar.unidata.geoloc
ucar.unidata.io
ucar.unidata.util

cdm-s3*
thredds.filesystem
thredds.inventory
ucar.unidata.io.s3

gcdm*
ucar.gcdm

grib*
ucar.nc2.grib

jj2000*
ucar.jpeg

netcdf4*
ucar.nc2.ffi
ucar.nc2.jni

opendap*
opendap
ucar.nc2.dods

uibase*
ucar.ui
ucar.util

uicdm*
thredds.logs
thredds.ui
ucar.nc2.ui
ucar.nc2.util

udunits*
ucar.units

If you believe the asterisks, then the problem is mostly in the cdm-core module.

The ability to rename is mostly tied up with backwards compatibility. It doesnt seem viable to have both old and new packages in the same release(?) see Make breaking transitions easy

The choice of names is tied into long term maintenance and ownership. Some options suggested are:

• ucar.xxxx
• edu.ucar.unidata.cdm.xxxx
• unidata.cdm.xxxx
• org.unique.cdm.xxxx

[lesserwhirls]

It looks like the bold face markup didn't show up in your post, but I think I have a pretty ok idea on where the idea of one superpackage per module works.

What would really be helpful is multi module jars. I think it was hinted at in Stephen Colebourne's blogpost Java SE 9 - JPMS module naming, as he has an example of it about half-way down the page, but it looks like it was deferred to a future release. But for now, one module per jar artifact, and I think that's why it's so dang tempting to try to use a convention to name the module based on the artifact (which is also argued against in the same blog post).

That said, I think we at least have to look at things from the point of view of one module per artifact since that is a hard limitation at the moment, which means one superpackage per artifact (e.g. gradle subproject), which is what you pretty much did above (talking to myself outloud a bit here).

If you believe the asterisks, then the problem is mostly in the cdm-core module.

I think that's right. The public API is contained completely in cdm-core, and all other artifacts extend runtime capabilities, but do not impact the public API. So for the sake of discussion, we can take those off the table for the moment.

Package-wise, the public API in version 6 looks like this:

Package	Description
thredds.client.catalog	THREDDS Catalog reading.
thredds.client.catalog.builder	THREDDS Catalog builders.
ucar.array	Multidimensional arrays stored in memory.
ucar.nc2	The public API to the Java-NetCDF library.
ucar.nc2.constants	Static final and enumerated constants used throughout the CDM library.
ucar.nc2.dataset	An extension to the netCDF API which provides support for coordinate systems, scale/offset/missing data, and NcML.
ucar.nc2.dataset.spi	Service Provider interfaces for NetcdfDatasets.
ucar.nc2.grid	Public interface for scientific feature type grid.
ucar.nc2.iosp	General classes used by I/O Service Providers.
ucar.nc2.time	Calendar Date handling.
ucar.nc2.units	Utility classes for scientific units.
ucar.nc2.util	Miscellaneous utility classes.
ucar.nc2.write	Classes to create and update Netcdf 3/4, and copy and dump utilities for all cdm files.
ucar.unidata.geoloc	Support library for georeferencing coordinate systems and transformations.
ucar.unidata.io	Classes for reading data from local and remote files.
ucar.unidata.io.spi	Service Provider interfaces for IO.

and these are all currently contained within the cdm-core jar. Thinking in terms of superpackages, I believe we practically have four:

• thredds.client.catalog
• ucar.array
• ucar.nc2
• ucar.unidata

(thredds and ucar are really the superpackages, but not at all practical).

It doesnt seem viable to have both old and new packages in the same release(?) see Make breaking transitions easy.

For those following along, see Page 7 of the hymnal. I agree that it does not seem viable to maintain both old and new packages in the same release if the packages are moved, BUT I think we could use some gradle magic to effectively do that while only maintaining the new packages. This would require:

1. a good mapping between old and new packages in the public API
2. no changes to the public API that only apply to one but not the other

Thoughts?

In terms of naming, as you mention, long term maintenance and ownership is important. Unidata is committed to being stewards of the codebase for as far as we can see into the future, but I like to think of ownership as more of a community based thing. If Unidata were to disappear (no signals to think we will, but you never know the future of NSF funded endeavors), I have no idea what would happen internally at UCAR. Would another UCAR group pick up the stewardship of the library? I don't believe so. In that scenario, my best guess is the library would get forked and maintained as a non-organization backed library (or, at least, that's what I would try to do if no other group at UCAR jumped at the chance to fund maintenance of or value add to (like hosting docs, mailing lists, support, etc.) the project.

All of that said, I'd add the following two options to the list as the superpackage, and thus potential module name, to use in cdm-core:

org.cdm.core
org.thredds.cdm.core

Packages in other gradle subprojects, like grib, bufr, could have their packages moved to org.cdm.<> (e.g org.cdm.grib, org.cdm.bufr, etc.) since they are not part of the public API.

[ethanrd]

Hi @JohnLCaron & @lesserwhirls - Sorry if I stumble over some of this. I'm a bit fuzzy on the constraints surrounding it all.

Both of you include some reverse-DNS looking names as an option for something (I think for module names and for possible renaming of packages that are not part of the public API). Only one of the options listed is an actual reverse-DNS name this project has any control over (John's edu.ucar.unidata.cdm suggestion). Is there a reason not to start ensuring all appropriate module/package names, where possible, start with "edu.ucar." or "edu.ucar.unidata."?

[lesserwhirls]

The background on this is related to what it takes to support the Java Platform Module System. At the moment, a jar file can contain only one module, and the module name should be very carefully chosen as it is what is used to determine any clashes on the module path. Once you pick a module name, it becomes very difficult for everyone involved to change it, so it's something you want to "get right", so to say.

Stephen Colebourne's blogpost Java SE 9 - JPMS module naming make a good argument that the module name should be the superpackage of the code that makes up the module. Since a jar can only have one module, that would mean the code contained within the jar artifact would need to be under a single super package (although Stephen also argues against conflating modules and artifacts, but again, that's currently where things stand). I totally agree with his assessment, and that means we'd need to think about renaming packages (even those in the public API), because right now we do not have anything close to the concept of super packages.

If we consider the current public API, which is totally contained within cdm-core.jar, there is not a single superpackage (I think there are, practically, four). Not only that, other jars that we produce have similar superpackages, making ownership of any one part of the codebase unclear (which module would own ucar.nc2 or ucar.nc2.iosp, for example?). Also, with modules, a package can only appear on the module path from one source - if a package shows up on the module path from more than one source, the java will not compile or run the project (unlike the situation with the classpath). Therefore, it's very convenient and useful to have, for example, everything under org.cdm.core contained within one subproject of the codebase, and at least right now, that means living in one jar artifact. No other subprojects or artifacts would include code that has org.cdm.core as part of their package name.

That's the gist of it, I think.

We've been talking offline about possibilities of what module names / package renames might look like. The current list of ideas for top level package names (super-superpackage names?, under which everything within netCDF-Java would live, or the common part of all module names) is:

• ucar (almost where we stand now)
• edu.ucar
• edu.ucar.unidata
• edu.ucar.unidata.cdm
• unidata.cdm
• org.cdm
• org.<something unique>.cdm
• org.thredds.cdm

I like the idea of reverse-DNS names, but that's only one way to the end goal of being unique. As John said, the top level package name should reflect the ideas of long term maintenance and ownership. I would say that long term maintenance is something Unidata is committed to, but only as long as we successfully obtain our next five year budget cycle with NFS (as opposed to something like a Google, Microsoft, etc., which have a much longer assumed time horizon). In my mind, going one level up with ucar or edu.ucar does not make sense, as I doubt any other group within the organization would take up stewardship of the library if Unidata was no longer a thing, making it a purely community run project (and a purely community run project should not be publishing java modules using the ucar namespace). That leads to the idea of making a strategic choice, for the long term stability of the module names at a minimum, to at least consider going outside of the ucar namespace all together. While changing packages names would be a hard break, the choice by a project to use java modules is in someways a hard break itself, and changing import statements, while annoying, is the least of the troublesome issues users of netCDF-Java might encounter when making their code modular.

In such a landscape (no mention of ucar or unidata in the package names of the library), it becomes more clear that Unidata is the steward of the codebase, but the project itself belongs to the community. This is currently how our python projects operate (they could include ucar or unidata in their package name structure, but do not), and feels like the right move overall. It's in no way meant to be a signal that Unidata is backing away from support of the library or anything like that.

Sorry for being longwinded, but hopefully that gives a bit more of at least my thought process here.

[ethanrd]

Thanks @lesserwhirls for this explanation and for the off-line discussion the other day.

[JohnLCaron]

cdm is way too common, and cdm.org is taken.

netcdfjava.org? the domain name is available...

[lesserwhirls]

thredds.org isn't taken yet, and could be used in the TDS as well as Rosetta (org.thredds.cdm, org.thredds.tds, org.thredds.rosetta)

[jonathanschilling]

Hi, I would like to revive this discussion. @uhoefel and me are trying to get the Nujan pure Java HDF5/NetCDF writer modernized, i.e., compliant with JPMS. Here, we require cdm-core to also be a Java module. Are there any plans/news/ways to contribute for this?