Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check datasets shape and dtype #16

Merged
merged 5 commits into from
Oct 31, 2022
Merged

Conversation

will-moore
Copy link
Member

@will-moore will-moore commented Oct 21, 2022

This PR validates a few things that aren't checked by the schema validation:

  • 1 - All dataset.shapes are the same length (number of dimensions is the same for all)
  • 2 - All dataset.shapes are the same length as the length of multiscale.axes.
  • 3 - Checks dimension_separator is / when version is 0.2-0.4
  • 4 - All dtypes are the same (see Require same datatype for all arrays in multiscales:datasets ngff#154)
  • 5 - Fixes handling of invalid version (previously got stuck with 'loading...'

To test:

NB: This currently asserts that the dtypes are all equal for v0.4 (so this PR can be tested). Need to revert to testing v0.5 and above BEFORE merging!

For other tests above, we really need some invalid sample data available... Might try to make some...

cc @constantinpape

@netlify
Copy link

netlify bot commented Oct 21, 2022

Deploy Preview for ome-ngff-validator ready!

Name Link
🔨 Latest commit 58094dd
🔍 Latest deploy log https://app.netlify.com/sites/ome-ngff-validator/deploys/6356cd12c609250008c85bf6
😎 Deploy Preview https://deploy-preview-16--ome-ngff-validator.netlify.app/
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@joshmoore
Copy link
Member

Nice, @will-moore! I got some "Error loading" messages from the URLs above.

Also: it might be worth considering how we maintain multiple validators for these non-json-schema constraints. Would defining "rule numbers" for each of them be feasible/useful?

@will-moore
Copy link
Member Author

I got some "Error loading" messages from the URLs above.

As expected! The "Error loading" is what you get if you use an invalid version, since we use the version to construct the schema URL. We don't try to maintain a list of all valid versions in this tool. I guess we could try to handle that 404 with a more custom message.

Other errors for the other test URL are also expected (violations of rules 1, 2, 3 & 4.

@joshmoore
Copy link
Member

For me, "error loading" meant something like "the data is not accessible". 👍 for "incompatible version: ${version}", etc.

@will-moore
Copy link
Member Author

will-moore commented Oct 24, 2022

@joshmoore: better?

Sorry, wrong image - this is the right one...
Screenshot 2022-10-24 at 18 36 33

@joshmoore
Copy link
Member

I like it! Thanks.

One last thing: did the file for "4: dtype mismatch" get deleted?

@will-moore
Copy link
Member Author

Hmmm - not sure why that got deleted. But here's another with the same dtype issue:
https://deploy-preview-16--ome-ngff-validator.netlify.app/?source=https://minio-dev.openmicroscopy.org/idr/v0.4/idr0077/9836832_z.zarr

Copy link
Member

@joshmoore joshmoore left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beautiful. Merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants