Divina profile with synced narration.

[m-abs]

At Nota (the Danish Library and Expertise Center for people with print disabilities) we are moving from DAISY 2.02 to EPUB and RWPM.

We use an in-house DAISY 2.02 format* for comic books with narration, but if possible we'd like to use a format others can use when we migrate to EPUB and RWPM in 2021.

The Divina profile for RWPM looks promising but lacks a few features for our needs, the main one being synced narration (the other being transitions and panning - but that is for another topic).

I've been looking into Region-Based Navigation used by EPUB 3.3 for comics, which looks very promising for distributing our comics as EPUB.

But that still leaves the RWPM Divina, should we just add a sync-narration-json like we would for an ordinary audio+text book? Or should we do something else?

• Demo here: https://m.nota.dk/embedded/43210?guest=true if it doesn't play the first time, open the link again.

[HadrienGardeur]

That's a very interesting use-case.

My recommandation would be to add this feature on top of the guided role in a comics. This means that for each "step" in the guided navigation, you could provide the equivalent audio segment using alternate. If we re-use the example from the DiViNa profile, this could look like that:

"guided": [
  {
    "href": "http://example.org/page1.jpeg",
    "title": "Page 1"
  },
  {
    "href": "http://example.org/page1.jpeg#xywh=0,0,300,200",
    "title": "Panel 1",
    "alternate": [
      {
        "href": "http://example.org/track#t=,23",
        "type": "audio/mpeg" 
      }
    ]
  },
  {
    "href": "http://example.org/page1.jpeg#xywh=300,200,310,200",
    "title": "Panel 2",
    "alternate": [
      {
        "href": "http://example.org/track#t=23,50",
        "type": "audio/mpeg" 
      }
    ]
  }
]

Unlike with text, there's no need for an additional SMIL-like structure for comics. Since we can already easily reference an area in a page, it feels natural to enhance it without using an external ressource.

For EPUB 3.x, although region based navigation remains an official satellite spec, it's not implemented anywhere. We'll need to decide how we plan on addressing this use case in Readium Mobile/Desktop but it seems unlikely that we'll support region-based navigation given the lack of files available.

One way this could be handled would be through an RWPM enhanced EPUB 3.3 file. In an OPF, we can use a link to an alternate resource which would be a DiViNa in this case:

<link rel="alternate" 
      href="manifest.json" 
      media-type="application/divina+json" />

Your EPUB would continue to work as-is with any EPUB FXL reading system, but this could trigger a specific behavior in apps that support DiViNa as well.

[m-abs]

Is there a better name than alternative? The narration audio isn't alternative, it is complimentary to the image.

[HadrienGardeur]

We could create a dedicated attribute for that, but I'm not frankly in favor of this approach. In some situations, an app might strictly fallback to the audio narration since the visual one won't be available.

Rather than thinking in terms of name, we should think in terms of semantics and in this case I believe that the semantics are appropriate.

[m-abs]

@HadrienGardeur
I like that suggestion, perhaps something like this:

{
  "narration": [
    {
      "role": "body",
      "href": "page1.jpeg",
      "title": "Page 1",
      "narration": [
        {
          "role": "panel",
          "text": "#xywh=0,0,300,200",
          "audio": "track.mp3#t=,23"
        },
        {
          "role": "panel",
          "text": "#xywh=300,200,310,200",
          "audio": "track.mp3#t=23,50"
        }
      ]
    }
  ]
}

Inspired by syncronized narration.json

[danielweck]

Note that in Thorium, Divina rendering/playback (i.e. images, videos, animated transitions, canvas rasterizing, etc.) is currently completely "outsourced" to this library: https://github.com/readium/divina-player-js

In other words, synchronized audio narration would have to be implemented in this library, which is maintained separately from the r2-navigator-js component that handles EPUB3 Media Overlays playback.

[m-abs]

If I have time over the Christmas vacation I'll see if I can write a quick example.

[HadrienGardeur]

I'm not in favor of that approach, I think it would be better to have a "visual navigator" that would be shared between multiple formats, including CBR/CBZ instead of a dedicated DiViNa library.

We could also default to this visual navigator for FXL EPUB files that are basically a list of bitmap images.

[anymus]

For the record, the divina-player-js library mentioned above (which I am currently working on) can go beyond reading Divinas: the module parsing Divinas is only invoked at the very start of the reading/playing process, and then an inner format is used during the actual read/play - meaning we could add other modules to parse other kinds of format, like CBR/CBZ for instance.

The ability to play audio is on the roadmap; I can target an implementation in February. The only issue to tackle first is how to describe the content in the manifest - I'll go with Hadrien's guided+alternate approach by default if no other consensus emerges.

(As for transition and panning, which is another topic as you said, what are your needs exactly? Transitions do exist in Divina.)

[m-abs]

@anymus

(As for transition and panning, which is another topic as you said, what are your needs exactly? Transitions do exist in Divina.)

I started another discussion about that in #76. There is a small demo of our current format+content-viewer.

[m-abs]

Besides an audio narration, we'd be very interested in TTS. So we need a way to provide the text per frame, bubble, headlines etc. along with language and other TTS related properties.

[HadrienGardeur]

This goes back to #21

Each Link Object already allows a title but if we added a description as well, it could contain text that a TTS engine could leverage.

[m-abs]

TTS should probably be a separate discussion. There is more to it than just providing the text to be read by the TTS engine, e.g. volume, Pronunciation Lexicon, voice family etc.

It would also be nice to have for the EPUB profile or maybe a separate profile from the Audiobook profile.

[HadrienGardeur]

@m-abs I'm ressuscitating an old discussion, but what exactly did you had in mind for TTS then? SSML rather than just plain text?

[m-abs]

@HadrienGardeur

SSML rather than just plain text?
Yes, SSML (or something else) rather than plain text.

My (loose) thinking was that comic books with TTS could be much better:

• if angry, happy or sad speech bubbles could read as such.
• emphasising parts of speech bubble (bold text etc.)
• Define what a character sounds like, with characteristics such as gender, age, how fast they speak, if they are timid etc.

I of course understand, that how the TTS sounds varies a lot based on the system and we could probably never get it to where a human narration would be.

[HadrienGardeur]

@m-abs we're looking into the same direction then.

if angry, happy or sad speech bubbles could read as such.

I'd like to support different types of bubbles to convey that information: readium/guided-navigation#1 (comment)

emphasising parts of speech bubble (bold text etc.)

I'm struggling with that part right now. SSML could be a good fit but it's poorly supported by on-device TTS and even cloud TTS API only offer a subset of SSML. There's also a trend where newer (and better sounding) voices built with ML offer no support at all for SSML.

Chopping up an utterance into smaller utterances is risky as well, since it can get in the way of the prosody and degrade the overall listening experience.

Define what a character sounds like, with characteristics such as gender, age, how fast they speak, if they are timid etc.

This could be handled by SSML or CSS Speech but it feels like a better fit for structured information in the JSON. Gender and age should be fine, but emotions/styles vary a lot between TTS engines and voices, which makes it hard to provide a consistent list of expressions.