Map all glyphs to the private use area and duplicate the first glyph. #9340

brendandahl · 2018-01-04T22:49:47Z

This was all spurred from a discussion with Jonathan Kew about our issues glyphs when the are mapped to certain unicode values. In an ideal world we'd have a canvas API to do something like drawGlyphID(), but he agreed that for now it's best to always use the private use area.

...

There have been lots of problems with trying to map glyphs to their unicode
values. It's more reliable to just use the private use areas so the browser's
font renderer doesn't mess with the glyphs.

Using the private use area for all glyphs did highlight other issues that this
patch also had to fix:

small private use area - Previously, only the BMP private use area was used
which can't map many glyphs. Now, the (much bigger) PUP 16 area can also be
used.
glyph zero not shown - Browsers will not use the glyph from a font if it is
glyph id = 0. This issue was less prevalent when we mapped to unicode values
since the fallback font would be used. However, when using the private use
area, the glyph would not be drawn at all. This is illustrated in one of the
current test cases (issue Letter ä not showing in pdf #8234) where there's an "ä" glyph at position
zero. The PDF looked like it rendered correctly, but it was actually not
using the glyph from the font. To properly show the first glyph it is always
duplicated and appended to the glyphs and the maps are adjusted.
supplementary characters - The private use area PUP 16 is 4 bytes, so
String.fromCodePoint must be used where we previously used
String.fromCharCode. This is actually an issue that should have been fixed
regardless of this patch.

Fixes:

bug 1425312
issue PDF rendered using pdf.js shows square like garbage #9279
issue Font rendering broken due to overfull PUA #8795
issue Google Noto Serif CJK JP font is not rendered properly in Mozilla firefox #9278
issue Letter ä not showing in pdf #8234 This is already in our test suite, but the glyph is now the correct ‘ä’ (compare to acrobat)

brendandahl · 2018-01-04T22:50:06Z

/botio test

pdfjsbot · 2018-01-04T22:50:07Z

From: Bot.io (Windows)

Received

Command cmd_test from @brendandahl received. Current queue size: 0

Live output at: http://54.215.176.217:8877/2ad0196f95f80db/output.txt

pdfjsbot · 2018-01-04T22:50:08Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @brendandahl received. Current queue size: 0

Live output at: http://54.67.70.0:8877/c4e1589c97d6f77/output.txt

pdfjsbot · 2018-01-04T23:01:52Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.67.70.0:8877/c4e1589c97d6f77/output.txt

Total script time: 11.72 mins

Font tests: Passed
Unit tests: Passed
Regression tests: FAILED

Image differences available at: http://54.67.70.0:8877/c4e1589c97d6f77/reftest-analyzer.html#web=eq.log

pdfjsbot · 2018-01-04T23:15:16Z

From: Bot.io (Windows)

Failed

Full output at http://54.215.176.217:8877/2ad0196f95f80db/output.txt

Total script time: 25.14 mins

Font tests: Passed
Unit tests: Passed
Regression tests: FAILED

Image differences available at: http://54.215.176.217:8877/2ad0196f95f80db/reftest-analyzer.html#web=eq.log

brendandahl · 2018-01-04T23:23:02Z

Looks like I have some issues on Linux (besides slight glyph movement).

Snuffleupagus · 2018-01-05T12:02:13Z

Looks like I have some issues on Linux (besides slight glyph movement).

You're probably already aware of it, but even on Windows there appears to be some issues still. Note that in both issue1049 and bug1354114 the previously undefined glyphs are now replaced by actual glyphs, but unfortunately not the correct ones (so it's better, but still not quite right).

Snuffleupagus · 2018-01-05T12:39:45Z

src/core/fonts.js

-var PRIVATE_USE_OFFSET_END = 0xF8FF;
-var SKIP_PRIVATE_USE_RANGE_F000_TO_F01F = false;
+ // Unicode Private Use Areas:
+var PRIVATE_USE_AREAS = [


Nit: Use const instead here.

Snuffleupagus · 2018-01-05T12:45:23Z

src/core/cff_parser.js

+  CFF.prototype = {
+    duplicateFirstGlyph: function CFF_duplicateFirstGlyph() {
+      // Browsers will not display a glyph at position 0. Typically glyph 0 is
+      // notdef, but a number of fonts put a valid glyph so it must be


Grammar nit: Shouldn't it perhaps be a "there" after "... valid glyph", i.e
// notdef, but a number of fonts put a valid glyph there so it must be

Snuffleupagus · 2018-01-05T12:47:21Z

src/core/cff_parser.js

+            0, // nRanges place holder
+            (start >> 8) & 0xFF,
+            start & 0xFF,
+            lastFD];


Nit: For improved readability, the ]; should be moved to the next line instead.

Snuffleupagus · 2018-01-05T13:24:45Z

In addition to the issues/bugs listed in #9340 (comment), the following ones also appear to be fixed:

Bug 1484598
Issue Missing glyphs in Source Han Sans font poster #5377 (improved, but still not entirely fixed; there's one missing glyph at the start now)
Issue Source Han Sans not displaying correctly at specific weight #5476
Issue Cigna Explanations Of Benefits render as gibberish #6397
Issue Artefacts #7787
Issue Figure shows garbage inside (e001 shows up) #8680
Issue Find safe spaces for our glyphs #9101 (since we're putting everything in the PUA, this one ought to be obsolete now)
Issue Font is broken □□ #9568
Issue issue with the last character arabic #10051

Snuffleupagus · 2018-01-05T16:08:00Z

test/pdfs/bug1425312.pdf.link

@@ -0,0 +1 @@
+https://www.unicode.org/charts/PDF/U11A50.pdf


How about linking to the Internet Archive here, to avoid the file becoming unavailable, i.e. using the following link instead https://web.archive.org/web/20171116124714/https://www.unicode.org/charts/PDF/U11A50.pdf.

Snuffleupagus · 2018-01-05T16:09:10Z

test/test_manifest.json

@@ -3327,6 +3339,12 @@
      "rounds": 1,
      "type": "eq"
    },
+    {  "id": "bug1425312",


This entry needs "link": true, added, to fix the following from the test logs:

WARNING: Unable to open file for reading "Error: ENOENT: no such file or directory, open 'pdfs/bug1425312.pdf'". Unable to verify the checksum for the files that are used for testing. Please re-download the files, or adjust the MD5 checksum in the manifest for the files listed above.

Also, it'd probably suffice to only test the second page here.

brendandahl · 2018-01-05T23:13:43Z

The above changes don't fix everything, just the windows issues and nits. I'm still looking into the linux issues.

brendandahl · 2018-01-12T00:19:12Z

For reference, the code that causes glyph 0 to not be mapped/drawn:
https://searchfox.org/mozilla-central/rev/7fb999d1d39418fd331284fab909df076b967ac6/gfx/thebes/gfxFontUtils.cpp#255

and bug
https://bugzilla.mozilla.org/show_bug.cgi?id=831548

brendandahl · 2018-01-12T21:24:11Z

Small update and some troubles:

Issue #2881 was failing on linux because the platform was rejecting the font (OTS was happy). After downloading and building a debug build of freetype, I used ftlint to find the font was being rejected because the number of charstrings was greater than the number of charset values. I then wrote a function to write out an updated cff charset (which fixed the issue). However, after this, endchar.pdf started failing. This is because it uses the enchchar\seac operator which requires the charset to be set to a predefined charset, whereas we were now writing out a custom format 0 charset.

So far I don't have a great plan to handle fixing both issues. Some possible ideas though:

Don't duplicate glyph 0, but instead use our js font renderer if we're trying to draw glyph 0. Cons: a few glyphs may look different relative to others.
Move glyph 0 to it's own font. Cons: lots of work to move a glyph and all it supporting stuff and we'd have to add another path to change the font in show text.
Merge endchar/seac into a single glyph. Not entirely sure if this is possible, but it may be the cleanest situation.

I'm open to ideas though @Snuffleupagus @yurydelendik

brendandahl · 2018-07-11T00:22:52Z

/botio test

pdfjsbot · 2018-07-11T00:22:53Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @brendandahl received. Current queue size: 0

Live output at: http://54.67.70.0:8877/5543dc2d2ea2b83/output.txt

pdfjsbot · 2018-07-11T00:22:53Z

From: Bot.io (Windows)

Received

Command cmd_test from @brendandahl received. Current queue size: 0

Live output at: http://54.215.176.217:8877/13600c876438903/output.txt

pdfjsbot · 2018-07-11T00:43:30Z

From: Bot.io (Windows)

Failed

Full output at http://54.215.176.217:8877/13600c876438903/output.txt

Total script time: 20.61 mins

Font tests: Passed
Unit tests: Passed
Regression tests: FAILED

Image differences available at: http://54.215.176.217:8877/13600c876438903/reftest-analyzer.html#web=eq.log

pdfjsbot · 2018-07-11T00:44:15Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.67.70.0:8877/5543dc2d2ea2b83/output.txt

Total script time: 21.36 mins

Font tests: Passed
Unit tests: Passed
Regression tests: FAILED

Image differences available at: http://54.67.70.0:8877/5543dc2d2ea2b83/reftest-analyzer.html#web=eq.log

brendandahl · 2018-07-11T00:46:43Z

Try that again with the correct changes...
/botio test

pdfjsbot · 2018-07-11T00:46:43Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @brendandahl received. Current queue size: 0

Live output at: http://54.67.70.0:8877/e2303cce49bb20b/output.txt

pdfjsbot · 2018-07-11T00:46:43Z

From: Bot.io (Windows)

Received

Command cmd_test from @brendandahl received. Current queue size: 0

Live output at: http://54.215.176.217:8877/464643c59b1b377/output.txt

pdfjsbot · 2018-07-11T01:00:16Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.67.70.0:8877/e2303cce49bb20b/output.txt

Total script time: 13.53 mins

Font tests: Passed
Unit tests: FAILED
Regression tests: FAILED

Image differences available at: http://54.67.70.0:8877/e2303cce49bb20b/reftest-analyzer.html#web=eq.log

pdfjsbot · 2018-07-11T01:12:26Z

From: Bot.io (Windows)

Failed

Full output at http://54.215.176.217:8877/464643c59b1b377/output.txt

Total script time: 25.70 mins

Font tests: Passed
Unit tests: FAILED
Regression tests: FAILED

Image differences available at: http://54.215.176.217:8877/464643c59b1b377/reftest-analyzer.html#web=eq.log

timvandermeij · 2018-09-08T15:51:26Z

Nice work!

Snuffleupagus added the font-conversion label Jan 5, 2018

Snuffleupagus reviewed Jan 5, 2018

View reviewed changes

brendandahl force-pushed the private-use branch from ddd7ddc to db4c76c Compare July 11, 2018 00:22

brendandahl force-pushed the private-use branch from db4c76c to 95baca8 Compare July 11, 2018 00:40

brendandahl force-pushed the private-use branch from 95baca8 to 228159e Compare July 11, 2018 01:31

timvandermeij merged commit 66422eb into mozilla:master Sep 8, 2018

Snuffleupagus mentioned this pull request Sep 10, 2018

PDF is not rendering properly. Getting console warning Failed to load font "g_d0_f21". #9747

Closed

brendandahl mentioned this pull request Sep 19, 2018

Use the Font Loading API in MOZCENTRAL builds, and GENERIC builds for Firefox version 63 and above (issue 9945, bug 1088663) #9982

Merged

This was referenced Nov 1, 2018

How to get a font used on the page? #10204

Closed

Odd Unicode characters instead of real letters are now used to render texts #10205

Closed

This was referenced Feb 1, 2019

Big product/sum signs looks funny #9602

Closed

Fallback to the built-in font renderer when font loading fails #10520

Closed

Snuffleupagus mentioned this pull request Feb 11, 2019

[api-minor] Fallback to the built-in font renderer when font loading fails #10539

Merged

Snuffleupagus mentioned this pull request Feb 19, 2019

Text in SVG pages is not aligned to unicode codepoints #9507

Closed

Snuffleupagus mentioned this pull request Mar 10, 2019

Text does not render on Mac OS X Chrome/Safari (Includes Reproduction) #5535

Closed

Snuffleupagus mentioned this pull request Mar 30, 2019

Incorrect conversion of LATIN SMALL LIGATURE FI #10697

Closed

Snuffleupagus mentioned this pull request Jul 3, 2020

[PDF to SVG, Text/Glyphs] How to have the reel text (remapped) into the exported svg? #12053

Closed

Snuffleupagus mentioned this pull request Feb 19, 2021

Unable to access PDF fonts #13002

Closed

Snuffleupagus mentioned this pull request May 1, 2021

Multiple spaces in PDF being squashed to single space in iOS during SVG conversion #8935

Closed

Snuffleupagus mentioned this pull request Dec 1, 2021

Unable to render font family #14330

Closed

calixteman mentioned this pull request Jul 7, 2022

Saving a PDF (Print>Save as PDF) turns it unsearchable #14277

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Map all glyphs to the private use area and duplicate the first glyph. #9340

Map all glyphs to the private use area and duplicate the first glyph. #9340

brendandahl commented Jan 4, 2018

brendandahl commented Jan 4, 2018

pdfjsbot commented Jan 4, 2018

pdfjsbot commented Jan 4, 2018

pdfjsbot commented Jan 4, 2018

pdfjsbot commented Jan 4, 2018

brendandahl commented Jan 4, 2018

Snuffleupagus commented Jan 5, 2018 •

edited

Loading

Snuffleupagus Jan 5, 2018

Snuffleupagus Jan 5, 2018 •

edited

Loading

Snuffleupagus Jan 5, 2018

Snuffleupagus commented Jan 5, 2018 •

edited

Loading

Snuffleupagus Jan 5, 2018

Snuffleupagus Jan 5, 2018 •

edited

Loading

brendandahl commented Jan 5, 2018

brendandahl commented Jan 12, 2018 •

edited

Loading

brendandahl commented Jan 12, 2018

brendandahl commented Jul 11, 2018

pdfjsbot commented Jul 11, 2018

pdfjsbot commented Jul 11, 2018

pdfjsbot commented Jul 11, 2018

pdfjsbot commented Jul 11, 2018

brendandahl commented Jul 11, 2018

pdfjsbot commented Jul 11, 2018

pdfjsbot commented Jul 11, 2018

pdfjsbot commented Jul 11, 2018

pdfjsbot commented Jul 11, 2018

timvandermeij commented Sep 8, 2018

		@@ -0,0 +1 @@
		https://www.unicode.org/charts/PDF/U11A50.pdf

Map all glyphs to the private use area and duplicate the first glyph. #9340

Map all glyphs to the private use area and duplicate the first glyph. #9340

Conversation

brendandahl commented Jan 4, 2018

brendandahl commented Jan 4, 2018

pdfjsbot commented Jan 4, 2018

From: Bot.io (Windows)

Received

pdfjsbot commented Jan 4, 2018

From: Bot.io (Linux m4)

Received

pdfjsbot commented Jan 4, 2018

From: Bot.io (Linux m4)

Failed

pdfjsbot commented Jan 4, 2018

From: Bot.io (Windows)

Failed

brendandahl commented Jan 4, 2018

Snuffleupagus commented Jan 5, 2018 • edited Loading

Snuffleupagus Jan 5, 2018

Choose a reason for hiding this comment

Snuffleupagus Jan 5, 2018 • edited Loading

Choose a reason for hiding this comment

Snuffleupagus Jan 5, 2018

Choose a reason for hiding this comment

Snuffleupagus commented Jan 5, 2018 • edited Loading

Snuffleupagus Jan 5, 2018

Choose a reason for hiding this comment

Snuffleupagus Jan 5, 2018 • edited Loading

Choose a reason for hiding this comment

brendandahl commented Jan 5, 2018

brendandahl commented Jan 12, 2018 • edited Loading

brendandahl commented Jan 12, 2018

brendandahl commented Jul 11, 2018

pdfjsbot commented Jul 11, 2018

From: Bot.io (Linux m4)

Received

pdfjsbot commented Jul 11, 2018

From: Bot.io (Windows)

Received

pdfjsbot commented Jul 11, 2018

From: Bot.io (Windows)

Failed

pdfjsbot commented Jul 11, 2018

From: Bot.io (Linux m4)

Failed

brendandahl commented Jul 11, 2018

pdfjsbot commented Jul 11, 2018

From: Bot.io (Linux m4)

Received

pdfjsbot commented Jul 11, 2018

From: Bot.io (Windows)

Received

pdfjsbot commented Jul 11, 2018

From: Bot.io (Linux m4)

Failed

pdfjsbot commented Jul 11, 2018

From: Bot.io (Windows)

Failed

timvandermeij commented Sep 8, 2018

Snuffleupagus commented Jan 5, 2018 •

edited

Loading

Snuffleupagus Jan 5, 2018 •

edited

Loading

Snuffleupagus commented Jan 5, 2018 •

edited

Loading

Snuffleupagus Jan 5, 2018 •

edited

Loading

brendandahl commented Jan 12, 2018 •

edited

Loading