ERROR: InvalidCharacterError: The string to be encoded contains characters outside of the Latin1 range.

[ShixiangWang]

Bug description

$ quarto render posts/2024-11-11-random/index.qmd 
[1/2] posts/2024-11-11-random/index.qmd
[2/2] index.qmd
ERROR: InvalidCharacterError: The string to be encoded contains characters outside of the Latin1 range.

Stack trace:
    at btoa (ext:deno_web/05_base64.js:52:13)
    at Object.utilities.metadataAttrs (file:///opt/quarto/bin/quarto.js:98020:34)
    at eval (eval at <anonymous> (file:///opt/quarto/bin/quarto.js:3925:16), <anonymous>:51:29)
    at renderEjs (file:///opt/quarto/bin/quarto.js:90595:18)
    at include (file:///opt/quarto/bin/quarto.js:90606:28)
    at partial (eval at <anonymous> (file:///opt/quarto/bin/quarto.js:3925:16), <anonymous>:9:40)
    at eval (eval at <anonymous> (file:///opt/quarto/bin/quarto.js:3925:16), <anonymous>:13:2)
    at renderEjs (file:///opt/quarto/bin/quarto.js:90595:18)
    at templateMarkdownHandler (file:///opt/quarto/bin/quarto.js:97842:28)
    at markdownHandler (file:///opt/quarto/bin/quarto.js:98800:24)

Steps to reproduce

No response

Expected behavior

No response

Actual behavior

No response

Your environment

• OS: Ubuntu 22.04

Quarto check output

$ quarto check
Quarto 1.6.33
[✓] Checking environment information...
      Quarto cache location: /home/wsx/.cache/quarto
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.4.0: OK
      Dart Sass version 1.70.0: OK
      Deno version 1.46.3: OK
      Typst version 0.11.0: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 1.6.33
      Path: /opt/quarto/bin

[✓] Checking tools....................OK
      TinyTeX: (not installed)
      Chromium: (not installed)

[✓] Checking LaTeX....................OK
      Using: Installation From Path
      Path: /usr/bin
      Version: 2021

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.12.7 (Conda)
      Path: /home/wsx/miniconda3/bin/python
      Jupyter: (None)

      Jupyter is not available in this Python installation.
      Install with conda install jupyter

[✓] Checking R installation...........OK
      Version: 4.4.1
      Path: /usr/local/lib/R
      LibPaths:
        - /home/wsx/R/x86_64-pc-linux-gnu-library/4.4
        - /usr/local/lib/R/library
      knitr: 1.48
      rmarkdown: 2.28

[✓] Checking Knitr engine render......OK

[mcanouil]

We need you to fill the issue template entirely, especially the part about providing a small reproducible example.
Without it, it's very unlikely we will be able to help you.

---

You can share a self-contained "working" (reproducible) Quarto document using the following syntax, i.e., using more backticks than you have in your document (usually four ````).
See https://quarto.org/bug-reports.html#small-is-beautiful-aim-for-a-single-document-with-10-lines.

If you have multiple files (and if it is absolutely required to have multiple files), please share as a Git repository.

R	Python
````qmd --- title: "Reproducible Quarto Document" format: html engine: knitr --- This is a reproducible Quarto document. {{< lipsum 1 >}} ```{r} x <- c(1, 2, 3, 4, 5) y <- c(1, 4, 9, 16, 25) plot(x, y) ``` {#fig-placeholder} {{< lipsum 1 >}} The end after @fig-placeholder. ````	````qmd --- title: "Reproducible Quarto Document" format: html engine: jupyter --- This is a reproducible Quarto document. {{< lipsum 1 >}} ```{python} import matplotlib.pyplot as plt x = [1, 2, 3, 4, 5] y = [1, 4, 9, 16, 25] plt.plot(x, y) plt.show() ``` {#fig-placeholder} {{< lipsum 1 >}} The end after @fig-placeholder. ````

Additionally and if not already given, please share the output of quarto check within a code blocks (i.e., using three backticks ```txt), see https://quarto.org/bug-reports.html#check.

[ShixiangWang]

For the simple example, it works.

[ShixiangWang]

I downloaded the latest version v1.5.57 from GitHub, intead of from the website. The issue resolved.

[mcanouil]

@ShixiangWang that's not at all the same version.
Please provide a reproducible example that uses 1.6.33.

[cderv]

@ShixiangWang If you could at least share what the content of your project looks like.

It is probably related to encoding. Error is thrown in

 at btoa (ext:deno_web/05_base64.js:52:13)

which we use to encode things around and avoid some escaping issue. For example in Quarto 1.6, we did solve something regarding listing categories. Do you use some categories and listing with specific characters ?

I believe btoa() may not be working on the providing string because it may be used with UTF-8 input, and outside of the supported ASCII range... 🤔 From what I can read at https://developer.mozilla.org/en-US/docs/Web/API/Window/btoa

This is all just assumption. Can you confirm anything related to this in your project ?

Thank you !

[cderv]

This is all just assumption. Can you confirm anything related to this in your project ?

Ok I confirmed it myself. Thanks for reporting !

1. quarto create project blog test
2. Tweak posts\welcome\index.qmd to switch categorie using UTF8 characters

   categories: ['Euro (€)']
   

3. quarto render

❯ quarto render
[1/4] about.qmd
[2/4] index.qmd
ERROR: InvalidCharacterError: The string to be encoded contains characters outside of the Latin1 range.

Stack trace:
    at btoa (ext:deno_web/05_base64.js:52:13)
    at Object.utilities.metadataAttrs (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/project/types/website/listing/website-listing-template.ts:458:28)
    at eval (eval at <anonymous> (https://cdn.skypack.dev/-/lodash@v4.17.21-K6GEbP02mWFnLA45zAmi/dist=es2019,mode=imports/unoptimized/template.js:58:12), <anonymous>:51:29)
    at renderEjs (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/core/ejs.ts:39:16)
    at include (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/core/ejs.ts:64:18)
    at partial (eval at <anonymous> (https://cdn.skypack.dev/-/lodash@v4.17.21-K6GEbP02mWFnLA45zAmi/dist=es2019,mode=imports/unoptimized/template.js:58:12), <anonymous>:9:40)
    at eval (eval at <anonymous> (https://cdn.skypack.dev/-/lodash@v4.17.21-K6GEbP02mWFnLA45zAmi/dist=es2019,mode=imports/unoptimized/template.js:58:12), <anonymous>:13:2)
    at renderEjs (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/core/ejs.ts:39:16)
    at templateMarkdownHandler (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/project/types/website/listing/website-listing-template.ts:176:26)
    at markdownHandler (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/project/types/website/listing/website-listing.ts:362:14)

So categories can't be UTF-8 characters anymore - this is a regression compared to Quarto 1.5 as it renders ok.

It follows

• listings - encode category names as base64 to avoid escaping issues #11177

[ShixiangWang]

@cderv Thanks for your kind reply, I don't know and understand the technical details, so I did not provide a reproducible case (my repo is https://github.com/ShixiangWang/ShixiangWang.github.io). I am happy to see that you have generated an example.

[cderv]

Thanks for providing the repo ! That is really helpful, and confirm what I found. You have UTF-8 character in your categories.
https://github.com/ShixiangWang/ShixiangWang.github.io/blob/7cb51d9b2c1007e7345841c1129a8632d8873926/posts/2024-01-22-immune/index.qmd#L5

I'll try something and test also on your repo. Thanks !

[cscheid]

@cderv I think "all" we need to do here is first convert the utf-8 stream to a byte sequence, then encode that. It shouldn't be too hard to do it correctly based on the diffs in #11177.

[cderv]

Yes I have a fix for this part already, but I am trying to get #11177 really fixing the apostrophe in category, as in fact it is not working. Took me a while to get to the bottom of it, but a +smart extension problem again.

I did not push the branch yet, but I am on it. I'll share before merging as I used decodeURIComponent and encodeURIComponent not utf-8 stream so you may want change it

[cscheid]

Yes I have a fix for this part already, but I am trying to get #11177 really fixing the apostrophe in category, as in fact it is not working. Took me a while to get to the bottom of it, but a +smart extension problem again.

@cderv Huh, I'm surprised - we have a test for that in the PR: https://github.com/quarto-dev/quarto-cli/blob/ef1648deebe7ae5b5eaa232301f26331ae61df32/tests/docs/smoke-all/2024/10/23/issue-10829/posts/post-with-code/index.qmd

[cderv]

Yes but it is not testing our JS logic quarto-listing.js. The problem is that clicking on the category in a post won't activate the category on the listing page. I am adding Playwright test for this.