[RFC] New feature: Images in labels

[alexshalamov]

Images inside text-field

This RFC explores possibility of embedding images within a text defined by symbol layer's text-field property.

Style interface

Images within a text could be added via new image expression that takes one mandatory argument that must evaluate to string, for example ["image", "car-15"].

Within 'format' expression, optional image options object may be provided after 'image' expression. Image options properties could be:

• height - sets height of an image
• width - sets width of an image
• scale - proportional scaling
• color - applicable for SDF icons
• sources - array of sources from where image would be tried to be fetched, like <img src=[url1, url2...]> (spritesheet, base url, etc). Example:
  ["spritesheet://icons", "https://www.mapbox.com/images/"]

Support for aforementioned image option properties may be added in the future if found useful.

Example syntax

The new image expression can be used as a section within text-field’s format expression, for instance:

"icon-image": "philadelphia-septa",
"text-field": ["format", 
                    "London ",
                    ["image", "gb-national-rail.london-tfl-rail.london-underground"],
                    " and Taipei ",
                    ["image", "taipei-metro"], { "useful": "image option" } 
                    " metro\n",
                    "SF ", {"text-color": "blue"},
                    ["image", "san-francisco-bart"],
                    " bart"
]

The new image expression could be represented by Image object ('coercable' to string if needed). The Image object would contain image id and image options. If needed, "icon-image" property could be converted to accept "image" expression, so that the text-field and icon-image properties would benefit from new functionality.

"icon-image": ["image", "philadelphia-septa"]

Rendering result

[ryanhamley]

If needed, "icon-image" property could be converted to accept "image" expression, so that the text-field and icon-image properties would benefit from new functionality.

Just wanted to note that this is what we're proposing for default images in #8052 (comment)

[asheemmamoowala]

@alexshalamov this is a really neat way to expand on the formatted text property to provide richer text composing and layout!

The timing of this couldn't be better, because there is alignment on adding an "image" expression operator in #8052 as well.

sources - array of sources from where image would be tried to be fetched, like <img src=[url1, url2...]> (spritesheet, base url, etc). Example:
["spritesheet://icons", "https://www.mapbox.com/images/"]

This has been discussed in the past and is a big change that affects not just the expression and style-spec, but also studio and offline functionality in gl-native.

Third argument is an options object whose properties may be [...] The options argument would be optional. Support for aforementioned object properties may be added in the future if found useful.

Is the options argument needed for the images to be usable inside text fields? In the absence of options, would the images in text-fields always be scaled to fit within the line height?

When the text field is laid out vertically (either using text-writing-mode: "vertical" or symbol-placement: "line", will the image be laid out horizontally or vertically as well? It's not clear to me what the sensible default would be for this.

cc @vakila @ansis @mapbox/studio @mapbox/map-design-team

[alexshalamov]

@asheemmamoowala thanks for feedback! Few comments.

This has been discussed in the past and is a big change that affects not just the expression and style-spec, but also studio and offline functionality in gl-native.

All properties for an options object could be added later if we find any of them useful for map designers. Initially, we could provide 'placeholder' in parser code. Parser would expect optional 'image options' object that in the future could contain new properties, therefore, we could guarantee style parsing backward / forward compatibility.

Is the options argument needed for the images to be usable inside text fields?

I would say yes, I would expect image with same options to produce similar rendering result.

In the absence of options, would the images in text-fields always be scaled to fit within the line height?

Most rich-text editors would change line metrics if image with height that is larger than the line height is embedded. Another option is to proportionally scale image to the line height and if designers want image to be larger, scale option may be used in that case.

When the text field is laid out vertically

I would say that image should be in it's natural orientation, upright, in any writing mode. I will add note to myself to research if there are use-cases for rotated images in vertical writing modes.

[asheemmamoowala]

The image operator options will really only be useful in the usage inside a formatted text-field. If used with icon-image or other image properties, it would be confusing in conjunction with the other layout properties, for ex: icon-size.

Would it work to put the image formatter options into the formatter object, similar to how text-color and text-scale are implemented for text sections?

"text-field": ["format", 
                    "London ",
                    ["image", "london-underground"], {"icon-color": "green", "icon-width":24},
                    " and Taipei ",
                    ["image", "taipei-metro"],
                    " metro\n",
                    "SF ", {"text-color": "blue"},
                    ["image", "san-francisco-bart"],
                    " bart"
]

[alexshalamov]

@asheemmamoowala Good point. Let's move options out of expression, so that it can be reused in 'icon-image'. I will update proposal accordingly.

Still, I would keep one optional 'object' (3rd argument) for an 'image' expression. Would be easy to add common options in the future, for example 'sources'.

[asheemmamoowala]

@mapbox/studio @mapbox/map-design-team - any thoughts on this proposal?

[chloekraw]

cc: mobile engineers in case you have any feedback -- we're planning to start implementation soon! @mapbox/maps-ios @mapbox/maps-android @1ec5

[tobrun]

This is going to be great @alexshalamov, really like the api design.

[ansis]

This looks great!

I agree that sources/width/height/color could be left out in the first version and considered separately so that they also work with icon-image.

In order to avoid reshaping the text on every frame the icon would need to scale together with the text. This means that text-size would affect the icon. How does this work? Let's say we have "text-size": ["step", ["zoom"], 16, 5, 24] and a 10px image. How big would the image be rendered at zoom 4? at zoom 5?

Most rich-text editors would change line metrics if image with height that is larger than the line height is embedded.

Sounds good to me

[samanpwbb]

I reviewed this on behalf of the Studio team.

First of all, can you verify that the example has some errors? Here's what I would expect a valid example to look like (always require an options object, no arbitrary option keys):

"icon-image": "philadelphia-septa",
"text-field": ["format", 
                    "London ",
                    {},
                    ["image", "gb-national-rail.london-tfl-rail.london-underground"],
                    {},
                    " and Taipei ",
                    {},
                    ["image", "taipei-metro"], 
                    { "image-scale": 1.2 },
                    " metro\n",
                    {}
                    "SF ", 
                    {"text-color": "blue"},
                    ["image", "san-francisco-bart"],
                    {},
                    " bart",
                    {}
]

Feedback:

• Overall, the proposal looks great. I expect Studio's UI to be able to adapt to this extension to format well, but I do have one big question (see the first question below).
• Would love to separate the proposal for sources from the main proposal here – Studio will likely never be able to support such an option due to it's CSP, for example.
• Unless we have vector icons, I strongly suggest we do not scale inlined icons based on text size. Otherwise, icons will not render crisply. User should provide icons with the appropriate size. I do think scaling line height to match image height is the correct move 👍.
• I question the value of height and width options – a user would not typically want to distort the proportions of their icon, so I don't think we need these options.

Additional questions:

• In order for an entry in format to register as an image, does the outer expression need to be an image expression, or can an expression that resolves to image be used? What happens in a case like this, where a decision expression either resolves to an image operator or not an image operator?:

["format", 
    "London ",
    {},
    ["match", ["get", "building_type"], ["image", "taipei-metro"], "some more text"]
    { "font-scale": 1.2}
]

[1ec5]

This is a powerful feature and I’m excited to see it coming together. 🎉

So far all the examples I’ve seen demonstrate manually hard-coding images inside text-field. But as I alluded to in mapbox/mapbox-gl-native#15666 (comment), it’s only a matter of time before developers would want the ability to systematically insert images into labels based on the contents of a feature property. The find-and-replace expression operator proposed in #4100 would enable the developer to substitute a placeholder token in a string with an image – probably a more palatable approach than making the name feature property more structured in the vector tiles.

Whereas a standalone icon may be decorative when paired with a text label, an icon embedded within a label would tend to carry semantic meaning. Currently, the iOS map SDK and gl-accessibility plugin integrate with screen readers to read the name feature property aloud. Ideally they would instead read the evaluated value of text-field, but if text-field contains non-decorative images, those images could need an alternative text representation.

I would say that image should be in it's natural orientation, upright, in any writing mode. I will add note to myself to research if there are use-cases for rotated images in vertical writing modes.

I wonder if this feature could be a solution for proper display of concurrent highway route shields. The existing icon symbol options are inadequate for concurrent shields, because conventionally they’re line-placed, offset from each other along the line, and always oriented upright with respect to the map:

Unfortunately, it isn’t currently possible to achieve the desired effect using symbol-placement: line, icon/text-rotation-alignment: viewport, and icon/text-offset. The Mapbox Streets source has long worked around symbol layer limitations by combining multiple shields into one, baking two numbers into a single shield, but this approach has a number of aesthetic and practical downsides.

The formatted-images-line rendering test in #8904 accomplishes the desired line placement and offset. The two missing pieces would be a way to:

• Overlay a text component atop an image component within the label, for example to overlay “74” atop to get
• Ensure that the shields and text remain upright with respect to the viewport, regardless of the line segment’s direction or the map’s rotation

Even if it isn’t the right time to support concurrent shields via this feature, I’m glad this feature will give us an easier way to put emoji-oids inside labels that doesn’t require overhauling the text rendering system. 😅

[chloekraw]

@alexshalamov asked on the PR:

One important design decision is that the size of images in text-field is controlled by the text-size. Should it be?

One suggestion:

Unless we have vector icons, I strongly suggest we do not scale inlined icons based on text size. Otherwise, icons will not render crisply. User should provide icons with the appropriate size.

I'm of the opinion that the image size has to scale with text-size, or else the label would look weird, considering the zoom-dependent expression @ansis asked about. In this case:

• Could you write a comparable expression in text-field to provide the appropriate size image?
• Even if so, could you then do this with a missing image?

Pretty sure that, if you wanted to use a zoom-dependent expression for text-size and a missing image in text-field, you would need the image size to scale with text-size in order for the image to be able to be the appropriate size for the text.

The other use cases that come to mind are when a user needs the size of text on the map to change based on 1) accessibility settings or 2) differing aspect ratios and screen sizes. A user might deploy a single map simultaneously on small and large screens of varying sizes, e.g., in cars.

I'm not sure to what extent we support these use cases today or what users do in these situations, but I think that we shouldn't break those implementations, i.e., we shouldn't make a decision that would force those users to start managing multiple styles if they used this feature and wanted the inlined images to scale with the text.

So, I'm not so sure about lack of vector icon support forcing fixed image sizes in labels, even though I do think crisply-rendered images are important.

[tristen]

One important design decision is that the size of images in text-field is controlled by the text-size. Should it be?

I think images should not scale in relation to text size. Resolution should be maintained and a designer would upload a scaled version to match larger text if needed.

Ideally they would instead read the evaluated value of text-field, but if text-field contains non-decorative images, those images could need an alternative text representation.

@1ec5 Thanks for bringing up the accessibility plugin. Should an image field support some sort of aria-label attribute for non-decorative images?

'text-field': ['format', 
  'London',
  {},
  ['image', 'gb-national-rail.london-tfl-rail.london-underground'],
  {
    'aria-label': 'GB National rail, London TfL rail, London underground'
  }
]

[alexshalamov]

@samanpwbb @tristen Thanks for feedback. At the moment I'm porting image operator to native and integrating it with format expression. After that I can try rewriting bits that scale image size by text-size property value.

For data-driven and constant sizes it should be rather simple to implement. For zoom-dependent, I'm not sure what to do at the moment.

[kkaefer]

This merged 🎉