J2K in HEIF specification

[farindk]

I moved over this discussion topic from (#915) @palemieux to here.

@dukesook @farindk I would appreciate your feedback (before December 15) on the proposed revision of the J2K in HEIF specification. This revision allows the presence of zero or more Colour Specification boxes, as defined in Rec. ITU-T T.801 | ISO/IEC 15444-2, in the JPEG 2000 header item. If multiple Colour Specification boxes are present, the decoder can choose the one it believes is most appropriate.

[dukesook]

Initially, it was unclear to me as to how J2K in HEIF relates to HTJ2K in HEIF since 15444-16 is silent on the matter. I erroneously assumed that HTJ2K images would have their own item type.

Of course, we don't want to add redundant information, but I think some clarification would be very helpful.
For example:

• All JPEG 2000 codestreams, including HTJ2K codestreams, are stored the same way in HEIF. The identification of J2K vs HTJ2K depends on the CAP Marker Segment in the codestream and is not explicit at the HEIF level. This is because the HT block coder can be mixed with other JPEG 2000 features.

[palemieux]

HT is J2K, so I do not think it is a good idea to handle it differently.

We could expose the CAP and RSIZ markers in HEIF, but that opens the door for the values to be inconsistent between the codestream and the HEIF metadata.

How big of a challenge is it to support reading CAP from the codestream? Any major blockers/obstacles?

[bradh]

I have implemented this. It feels like a layering violation, but we already needed it for SIZ, so no big deal.

It took me a while to understand that there is no such thing as a HT-J2K codestream. There is only a J2K codestream that might use -15 in some tileparts... Possibly the marketing for HT confused the story.

[palemieux]

I have implemented this. It feels like a layering violation, but we already needed it for SIZ, so no big deal.

What do you mean by "this"? :)

[bradh]

By "this", I mean CAP parsing from the codestream main header.

Specifically, see #1110

This is infrastructure so libheif can tell if it needs a HT-capable decoder. That part isn't implemented yet though.

[palemieux]

Thanks for the quick reply and the details.

[farindk]

6.2 "The image_width and image_height fields of the 'ispe' item property shall be equal to width and height of the decoded image."

This is already clear from the definition of ispe. If you want to repeat it here, you could take the chance to make it more explicit that it is the size of the decoded JPEG2000 image prior to any transformation properties applied from the HEIF container.

"The coded image item shall be associated with exactly one 'colr' item property"

This contradicts the definition of colr in HEIF, which says: "at most one". I also would advocate to leave this open such that it will be possible to add further types of content (IR, multi-spectral, radio, ...) through other ways to describe the content later.
In particular, I do not see how this should work together with the specification of colour in the J2K header (Section 6.3.6) that can hold colour-spaces that cannot be expressed in colr.
My interpretation would be like this: if there is a colr box, it must match the definition in the J2K colour spec. If there is no colr box, the J2K colour spec should be taken.

[palemieux]

This is already clear from the definition of ispe. If you want to repeat it here, you could take the chance to make it more explicit that it is the size of the decoded JPEG2000 image prior to any transformation properties applied from the HEIF container.

This is probably a leftover of some earlier iteration. I plan to suggest that the statement is removed as redundant.

[palemieux]

I also would advocate to leave this open such that it will be possible to add further types of content (IR, multi-spectral, radio, ...) through other ways to describe the content later.

Agree. I plan to suggest removing the constraints.

My interpretation would be like this: if there is a colr box, it must match the definition in the J2K colour spec. If there is no colr box, the J2K colour spec should be taken.

I am not sure we need to/should mandate that the colr box, if one is present, must match a J2KColour box in j2kH:

• it opens the door for conflicting information
• it means that the spec must provide a mapping between colr and J2KColour boxes

Instead, could we state that:

• if a colr box with colour_type == nclx is present, then no J2KColour box with METH = 5 shall be present
• if a colr box with colour_type == rICC is present, then no J2KColour box with METH = 2 shall be present
• if a colr box with colour_type == prof is present, then it supplements any J2KColour box with METH = 3
• for compatibility with previous editions of this document, a colr box should be present

We could add that:

• if both a colr box with colour_type == nclx and a J2KColour box with METH = 5 are present, then the J2KColour box takes precedence
• if both a colr box with colour_type == rICC and a J2KColour box with METH = 2 are present, then the J2KColour box takes precedence

[palemieux]

Actually we would say:

• at most one colr box with colour_type == nclx or one J2KColour box with METH = 5 shall be present
• at most one colr box with colour_type == rICC or one J2KColour box with METH = 2 shall be present

[palemieux]

All JPEG 2000 codestreams, including HTJ2K codestreams, are stored the same way in HEIF. The identification of J2K vs HTJ2K depends on the CAP Marker Segment in the codestream and is not explicit at the HEIF level. This is because the HT block coder can be mixed with other JPEG 2000 features.

Will add.

[palemieux]

@farindk @dukesook Any interest in exposing the CAP and RSIZ data in the hej2k media type as optional parameters? Example:

image/hej2k;cap=000200000026;rsiz=4000

[palemieux]

Please review the revised draft of the proposed revision to the mapping of J2K into HEIF/ISOBMFF: https://ds.jpeg.org/documents/jpeg2000/wg1n100805-103-ICQ-CD_15444-16_3ED.pdf

This addresses issues identified in the earlier draft; adds support for most of JPX's color signaling boxes; and adds an explicit mapping to ISOBMFF video tracks.