Upgrade jsonschema to 3.0.1

[eslavich]

Addresses #530

In its current state, this PR isn't compatible with jsonschema < 3, due to some (relatively minor) changes to validators.create(...). It would be straightforward to support both, but I thought I'd keep it clean unless I hear otherwise.

I installed this and ran the unit tests on jwst, gwcs, and astropy. With the changes in spacetelescope/jwst#3705, they all pass.

[stsci-bot]

Hi there @eslavich 👋 - thanks for the pull request! I'm just a friendly 🤖 that checks for issues related to the changelog. I help make sure that this pull request is milestoned and labeled correctly. This is mainly intended for the maintainers, so if you are not a maintainer you can ignore this, and a maintainer will let you know if any action is required on your part 😃.

I noticed the following issue with this pull request:

• Changelog entry not present (or pull request number missing) and neither the Affects-dev nor the no-changelog-entry-needed label are set

Would it be possible to fix this? Thanks!

If there are any issues with this message, please report them here.

[codecov]

Codecov Report

Merging #684 into master will decrease coverage by 0.05%.
The diff coverage is 66.66%.

@@            Coverage Diff             @@
##           master     #684      +/-   ##
==========================================
- Coverage   93.45%   93.39%   -0.06%     
==========================================
  Files          39       39              
  Lines        4341     4347       +6     
==========================================
+ Hits         4057     4060       +3     
- Misses        284      287       +3

Impacted Files	Coverage Δ
asdf/schema.py	93.15% <66.66%> (-1.01%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 178194c...fbca2d3. Read the comment docs.

[jdavies-st]

This looks good!

Is it possible to have asdf work with jsonschema 2.6.0 and jsonschema 3.0.1? There is some dependency version checking code elsewhere in asdf if that is what is required. This would make the transition to the next version of asdf much easier for other packages that currently depend on asdf.

[nden]

LGTM

[nden]

Needs a change log.

[jdavies-st]

This looks good.

Both jsonschema 3.0.1 and jsonschema 2.3.0 are being tested in the TravisCI matrix. The 2.3.0 build is here

https://travis-ci.org/spacetelescope/asdf/jobs/549785630

I think that is not reflected in codecov. That's fine by me.

[pllim]

I don't understand why this did not solve my problem trying to load this JWST image: https://stsci.box.com/s/hwrc5reqygmmv2rl3yvvz90l7ryjac6h

setuptools thinks the dev asdf (getting the correct dev version is another headache, see #690) is compatible with jsonschema 3.0.1 that is installed. But then I tried to read the file above:

from astropy.io import fits
from asdf.fits_embed import AsdfInFits
filename = 'jw1069001001_01203_00002_nrca1_level2_cal.fits'
pf = fits.open(filename)
asdf_f =  AsdfInFits.open(pf)

I get this:

---------------------------------------------------------------------------
VersionConflict                           Traceback (most recent call last)
<ipython-input-8-10be42d48b7d> in <module>
----> 1 asdf_f =  AsdfInFits.open(pf)

~\Miniconda3\envs\py37\lib\site-packages\asdf\fits_embed.py in open(cls, fd, uri, validate_checksums, extensions, ignore_version_mismat
ch, ignore_unrecognized_tag, strict_extension_check, ignore_missing_extensions)
    215                        ignore_unrecognized_tag=ignore_unrecognized_tag,
    216                        strict_extension_check=strict_extension_check,
--> 217                        ignore_missing_extensions=ignore_missing_extensions)
    218
    219     @classmethod

~\Miniconda3\envs\py37\lib\site-packages\asdf\fits_embed.py in _open_impl(cls, fd, uri, validate_checksums, extensions, ignore_version_
mismatch, ignore_unrecognized_tag, strict_extension_check, _extension_metadata, ignore_missing_extensions)
    239         self = cls(hdulist, uri=uri, extensions=extensions,
    240                    ignore_version_mismatch=ignore_version_mismatch,
--> 241                    ignore_unrecognized_tag=ignore_unrecognized_tag)
    242         if _extension_metadata is not None:
    243             self._extension_metadata = _extension_metadata

~\Miniconda3\envs\py37\lib\site-packages\asdf\fits_embed.py in __init__(self, hdulist, tree, **kwargs)
    150         if hdulist is None:
    151             hdulist = fits.HDUList()
--> 152         super(AsdfInFits, self).__init__(tree=tree, **kwargs)
    153         self._blocks = _EmbeddedBlockManager(hdulist, self)
    154         self._hdulist = hdulist

~\Miniconda3\envs\py37\lib\site-packages\asdf\asdf.py in __init__(self, tree, uri, extensions, version, ignore_version_mismatch, ignore
_unrecognized_tag, ignore_implicit_conversion, copy_arrays, lazy_load, custom_schema, _readonly)
    120         self._extension_metadata = {}
    121
--> 122         self._process_extensions(extensions)
    123         self._ignore_version_mismatch = ignore_version_mismatch
    124         self._ignore_unrecognized_tag = ignore_unrecognized_tag

~\Miniconda3\envs\py37\lib\site-packages\asdf\asdf.py in _process_extensions(self, extensions)
    204     def _process_extensions(self, extensions):
    205         if extensions is None or extensions == []:
--> 206             self._extensions = default_extensions.extension_list
    207             self._extension_metadata = default_extensions.package_metadata
    208             return

~\Miniconda3\envs\py37\lib\site-packages\asdf\extension.py in extension_list(self)
    231     def extension_list(self):
    232         if self._extension_list is None:
--> 233             self._extension_list = AsdfExtensionList(self.extensions)
    234
    235         return self._extension_list

~\Miniconda3\envs\py37\lib\site-packages\asdf\extension.py in extensions(self)
    224                 self._extensions.append(BuiltinExtension())
    225
--> 226             self._load_installed_extensions()
    227
    228         return self._extensions

~\Miniconda3\envs\py37\lib\site-packages\asdf\extension.py in _load_installed_extensions(self, group)
    195             with warnings.catch_warnings(record=True) as w:
    196                 warnings.simplefilter('always', category=AsdfDeprecationWarning)
--> 197                 ext = entry_point.load()
    198             if not issubclass(ext, AsdfExtension):
    199                 warnings.warn("Found entry point {}, from {} but it is not a "

~\Miniconda3\envs\py37\lib\site-packages\pkg_resources\__init__.py in load(self, require, *args, **kwargs)
   2431             )
   2432         if require:
-> 2433             self.require(*args, **kwargs)
   2434         return self.resolve()
   2435

~\Miniconda3\envs\py37\lib\site-packages\pkg_resources\__init__.py in require(self, env, installer)
   2454         # requirements for that extra are purely optional and skip over them.
   2455         reqs = self.dist.requires(self.extras)
-> 2456         items = working_set.resolve(reqs, env, installer, extras=self.extras)
   2457         list(map(working_set.add, items))
   2458

~\Miniconda3\envs\py37\lib\site-packages\pkg_resources\__init__.py in resolve(self, requirements, env, installer, replace_conflicting,
extras)
    789                 # Oops, the "best" so far conflicts with a dependency
    790                 dependent_req = required_by[req]
--> 791                 raise VersionConflict(dist, req).with_context(dependent_req)
    792
    793             # push the new requirements onto the stack

VersionConflict: (jsonschema 3.0.1 (c:\users\pl\miniconda3\envs\py37\lib\site-packages), Requirement.parse('jsonschema<=2.6,>=2.3'))

What is going on?

[jdavies-st]

The released version of asdf doesn't have this in it yet. We should do a release soon.

[pllim]

@jdavies-st , I am using dev version of asdf.

[jdavies-st]

Excellent and you're still seeing this issue in the dev version?

[pllim]

Yes, it appears to pass the install check but something crashes when I read the file in the Box link above. You can try to reproduce it yourself, and if it works for you, then at least I know it is just me.

[jdavies-st]

I suspect that because asdf uses an extension from the jwst package via entry_points (under-the-hood) to load the JWST file, because jwst currently has a requirement that jsonschema<=2.6, that is causing the issue.

[pllim]

In that case... Is there a plan for jwst to unpin jsonschema?

[jdavies-st]

Yes, there is a plan and PR ^

But I'm not sure we want to do this before a new release of asdf is done.

[cdeil]

I ran into this jsonschema version issue also.
Log: https://gist.github.com/cdeil/89f1e87e2b1f86b24e92cbae675e40d5

I think most (all?) asdf users will run into this issue at the moment?

Using asdf master resolved it, but it requires quite a dance and the commands aren't given here
https://asdf.readthedocs.io/en/latest/asdf/install.html#building-from-source
so it's only something advanced users will manage to do.

git clone https://github.com/spacetelescope/asdf
cd asdf
pip install .  # error
git submodule init
git submodule update
pip install .

Is it possible to release a new stable asdf version with this fix soon?
Or is something else holding back a new release?

[nden]

@cdeil Yes, we will have a release out early next week.

Your gist example initializes a Gaussian model and this reminds me the Gaussian doesn't have a schema yet. We need to add schemas for all models but it'd be good to know what models you need to serialize and add those schemas first. Even better if someone else can add them :).

[cdeil]

Yes, we will have a release out early next week.

Great!

We need to add schemas for all models but it'd be good to know what models you need to serialize and add those schemas first.

In Gammapy, we currently have our own models, not using Astropy models yet. E.g.:

• https://docs.gammapy.org/0.13/api/gammapy.spectrum.models.SpectralGaussian.html
• https://docs.gammapy.org/0.13/api/gammapy.image.models.SkyGaussian.html

And now we are adding config files and serialisation, my comment was triggered by gammapy/gammapy#2314 (comment)

I don't know if we can / will switch to use astropy.modeling and ASDF, it's an option we'll investigate.

So for now, from our side, there's no need to add or prioritise any Astropy models.

cc @QRemy and @adonath who will work on model serialisation in Gammapy.

If you have time to make a PR for Gaussian model for Astropy, that might be a good opportunity to learn how it works and to make a contribution.

@nden - Adding a model means writing code like this?
https://github.com/astropy/astropy/blob/ebc0611a929a93e56494251765d3a3c253905039/astropy/io/misc/asdf/tags/transform/basic.py#L106-L118
Or in addition writing a YAML file with a schema?

[nden]

@cdeil Yes, the tag is implemented in a class defining the two methods to_tree_transform and from_tree_transform. In addition a schema is needed.
We should also soon have a "checklist" in the documentation of how to add a schema and a tag.

[cdeil]

Where is the schema e.g. for ConstantType? I see there's schemas in astropy/io/misc/asdf/data/schemas/astropy.org/astropy, but I couldn't find any for models?

We should also soon have a "checklist" in the documentation of how to add a schema and a tag.

@nden - Great!

[perrygreenfield]

They are in spacetelescope/asdf-standard. Yes, we are discussing whether this kind of separation makes sense. It does if there were other language implementations, but until then, it's more awkward to manage.

[cdeil]

Ah, it's here: https://github.com/spacetelescope/asdf-standard/blob/master/schemas/stsci.edu/asdf/transform/constant-1.3.0.yaml

I see, yes, that's a difficult decision then where to maintain the schemas.