Rendering settings specification

[dominikl]

Over the past few weeks we assembled a draft for a "rendering settings" specification (current spec: https://ngff.openmicroscopy.org/latest/#omero-md ); taking into account how OMERO stores rendering information, how other image formats handle it, and what information various clients / viewers need for rendering image data.

Here's an example how this could look like:

"renderingDefinition": {
   "@id": "renderingDef1"
   "groups": [
        {
            "views": ["view1", "view2"],
            "name": "Cycle1",
            "visible": true
        },
        {
            "views": ["view3"],
            "name": "Cycle2",
            "visible": true
        }
    ],
    "views": [
        {
            "@id": "view1",
            "dimension": [null, 0, null, null, null]  // (t,c,z,y,x)
            "visible": true,
            "mapping": {
               "family": "linear",
            }
            "color": {
                "format": "name",
                "type": "lut",
                "value": "brgbcmyw.lut"
            },
            "deviceSpaceMapping": {
                "reverseIntensity": false
            },
            "label": "GFP",
            "window": {
                "end": 188.89093017578125,
                "max": 188.89093017578125,
                "min": 0.007875712588429451,
                "start": 0.007875712588429451
            }
        },
        {
            "@id": "view2",
             "dimension": [null, 1, null, null, null],  // (t,c,z,y,x)
            "visible": false,
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "00FF00"
            },
            "deviceSpaceMapping": {
                "reverseIntensity": false
            },
            "mapping": {
               "family": "polynomial", // part of enumeration 
               "coefficient": 1.5,
               "noiseReduction": false,
            }
            "label": "DAPI",
            "window": {
                "end": 1542.5789794921875,
                "max": 1542.5789794921875,
                "min": 0.0,
                "start": 0.0
            }
        },
        {
            "@id": "view3",
            "dimension": [null, 2, null, null, null]  // (t,c,z,y,x)
            "visible": true,
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "FFFFFF"
            },
            "deviceSpaceMapping": {
                "reverseIntensity": false
            },
            "mapping": {
               "family": "linear",
            }
            "label": "OOOO",
            "window": {
                "end": 1555.0537109375,
                "max": 1555.0537109375,
                "min": 0.0,
                "start": 0.0
            }
        }
    ],
    // e.g. shape is (t,c,z,y,x)
    "defaultIndices": [5, null, "10:20", null, null],   // default view: t=5 and range 10-20 on z axis 
    "planarView": [3, 4], // we want to view y,x plane by default
    "zoom": 50 // percentage starting from highest resolution
}

RGB example

"renderingDefinition": {
   "@id": "renderingDef1"
   "groups": [
        {
            "views": ["view1", "view2", "view3"],
            "name": "The name you want to use",
            "visible": true,
            "metadata": {"rgb": true}
        }
    ],
    "views": [
        {
            "@id": "view1",
            "dimension": [null, 0, null, null, null]  // (t,c,z,y,x)
            "visible": true,
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "FF0000"
            },
            "label": "Red",
        },
        {
            "@id": "view2",
            "dimension": [null, 2, null, null, null]  // (t,c,z,y,x)
            "visible": true,
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "00FF00"
            },
            "label": "Green",
        },
        {
            "@id": "view3",
            "dimension": [null, 1, null, null, null]  // (t,c,z,y,x)
            "visible": true,
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "0000FF"
            },
            "label": "Blue",
        },
    ],
}

Comments, suggestions, etc. welcome!

(Similar issues #36 and #23)

concept like "defaultIndices" and dimension should allow the introduction of new dimension e.g. angle.

The concept of group can be used for example to "combine" channel like RGB, group view for some imaging modalities e.g. CyCIF

renderingDefinition is not mandatory. Currently no fields in renderingDefinition are mandatory.

An image could have multiple renderingDefinition per image

For color: see https://en.wikipedia.org/wiki/List_of_color_palettes and https://colorcet.holoviz.org/

• Support for one format only e.g. hex
  List of family: linear, logarithmic, exponential, polynomial

[tischi]

Looks interesting 😃 !

Some questions:

1. what does that do?

"dimension": [null, 1, null, null, null], // (t,c,z,y,x)

2. what does that mean?

            "window": {
                "end": 1555.0537109375,
                "max": 1555.0537109375,
                "min": 0.0,
                "start": 0.0
            }

3. what is the point of "@id"? Is there a special meaning for the "@" symbol in JSON?

[will-moore]

"dimension": [null, 1, null, null, null], // (t,c,z,y,x) means that the rendering setting applies to the C index of 1 and across all x, y, z, t dimensions. The idea here is that you might want the rendering setting to apply only to a sub-set of the image (eg. time) or when we support additional axes (e.g. rotation angle) you might want the setting to apply only to some rotation angles.

The window attributes refer to the pixel range of the image (min is value of darkest pixel, max is value of brightest pixel and the rendering settings (typically within that range). start -> end is the rendering range. start - a pixel of this value is black end - a pixel of this value is saturated.

@id is a JSON-ld concept, although I don't think this is full JSON-ld, since we'd need a @type too.

[tischi]

another note: contrastLimits is a wording that is been used in this context. maybe more intuitive than window (which made me initially think of a crop)?

maybe then a bit more explicit: valueRangeMin, contrastMin, valueRangeMax, contrastMax.

Computing the valueRangeMin and valueRangeMax might be a bit expensive for a big data set, but when writing out the image one anyway has to loop through all pixels so it is nice to store those values.

[jburel]

see also https://github.com/saalfeldlab/render/

[will-moore]

@jburel I read a bit of the saalfeldlab/render, but it seems that's all about transforming raw pixel data (affine transforms etc) and not about rendering to rgb?

[jburel]

I have not looked at it. I put the link as a reference to see if something needs to be reviewed/checked.

[constantinpape]

Just a quick comment: I don't think https://github.com/saalfeldlab/render/ is related here. It is about applying coordinateTransformations like affine transformations on very large dataset, e.g. for registration of EM data.

[tischi]

Just realizing that one of the main points of OME.Zarr is that you can write the voxel values in a distributed manner, thus one may not know valueRangeMin/Max. Is it critical for us to know them for rendering? I know it is very convenient, but due to potentially distributed nature of the voxel data generation it seems to me that we should not have them in the spec.

@dominikl What do you think?

[d-v-b]

Just realizing that one of the main points of OME.Zarr is that you can write the voxel values in a distributed manner, thus one may not know valueRangeMin/Max.

I don't see the connection here -- computing the min / max of a big distributed image dataset is not much harder than writing it to disk.

[tischi]

I don't see the connection here -- computing the min / max of a big distributed image dataset is not much harder than writing it to disk.

It is not hard, but it takes time. I wonder thus whether the spec should require that this MUST be done. What do you think?

[will-moore]

I don't think this should be MUST since it may be a burden to calculate.
Even in OMERO where the DB expects this, there is an option to skip the min/max calculation on import because it can be expensive.
In the current NGFF spec, we have this window under the omero metadata and it's useful when rendering in clients, since you can set initial rendering settings based on the known pixel values instead of the pixel-type alone (which will often give you a black image).
E.g. in vizarr, Trevor recently added the ability to read the lowest resolution data to get min/max values if they weren't provided (hms-dbmi/vizarr#137) to avoid the problem of "black image".
So I think there is good reason to include this metadata in the Spec, but make it optional.

contrastLimits is certainly clearer than window. How about we split it up like:

pixelIntensity: {
    "min": 0,
    "max": 1234
},
contrastLimits: {
    "start": 50,
    "end": 1000
}

The idea with start/end rather than min/max for contrastLimits is that these are just the user-preferred current settings and a user may choose to view the image with values outside this range. But I'm OK with min/max if preferred?

[tischi]

Cool! I am very happy with optional pixelIntensity (as said I acknowledge that it is very handy to have those values), however I do not find the name very intuitive. What about valueLimits? start and end is fine for me for the contrastLimits.

[jburel]

One point to note regarding the min/max. It is usually a global value i.e. max of all maxima. If it is not set (e.g. too expensive to calculate), this might default to only the min/max on the plane currently viewed.
The suggested valueLimits will be fine. If not calculated do we want to set them by default to the min/max pixels range depending on the pixels type so a validation can be performed.

[tischi]

If not calculated do we want to set them by default to the min/max pixels range depending on the pixels type so a validation can be performed.

I was wondering about this, but for floating point data types one gets, for my taste, pretty non-sensical stuff there, or you think that's OK?

[will-moore]

I think that if you don't know the valueLimits then they shouldn't be set. Otherwise a client won't know if the valueLimits are the actual pixelValues or just the defaults? If they're missing, the client can choose to calculate or use the full range of the pixel-type etc.

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/next-call-on-next-gen-bioimaging-data-tools-2022-01-27/60885/11

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/querying-and-modifying-omeros-rendering-luts/63885/9

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/intermission-ome-ngff-0-4-1-bioformats2raw-0-5-0-et-al/72214/1

[aliaksei-chareshneu]

Dear all,

I would like to ask you if you could explain the meaning of channel properties in “omero” metadata (https://ngff.openmicroscopy.org/latest/#omero-md):
specifically, “coefficient”,“family”, “inverted”, and “window”.

Thank you for any input,
Best regards,
Aliaksei

[will-moore]

coefficient is a gamma setting (non-linear mapping between pixel intensity and rendered intensity), and the family is the type of gamma curve. There are a small number of types that are supported by the OMERO server (where this spec comes from) but the NGFF spec doesn't define what's allowed here.

Ifinverted is true then we map 0 pixel intensity -> maximum brightness and vice versa.
The window values for each channel are: min/max: pixel intensity range. start/end: current rendering settings:

• min: darkest pixel value in the image
• max: brightest pixel value in the image
• start: pixel intensity that renders as "black" or zero intensity
• end: pixel intensity that renders as "white" or max intensity

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/community-call-metadata-in-ome-ngff/77570/16