Wrong unicode char reporting.

[lunny]

[wolfogre]

It is a Chinese unicode ’, my typo.

[silverwind]

I agree, this detection is still far too agressive. It should only warn on actual suspicious stuff, like RTL characters and such, not "uncommon" punctations.

[wxiaoguang]

Also agree that "the detection is too aggressive"

At least, I think we can make the detection skip the Markdown/HTML/XML/etc types. That detection only makes sense for source code files.

[silverwind]

I would prefer a solution that adds a config option ui.CHARACTER_HIGHLIGHT=bidi,ambiguous that triggers on certain character classes:

1. bidi: Nefarious BIDI characters like RTL Override character. I think this list on Wikipedia is accurate.
2. ambiguous: Also highlight ambiguous characters like non-standard whitespace and puncation.

The default should be bidi to cover CVE-2021-42574, but admins could also set it to empty string to disable the checking completely.

[silverwind]

At least, I think we can make the detection skip the Markdown/HTML/XML/etc types. That detection only makes sense for source code files.

No, don't skip any files. BIDI exploits can also be in the filename. From the linked page:

when using it in the file name my-text.'U+202E'cod.exe, the file name is actually displayed as my-text.exe.doc

[wxiaoguang]

I think it worries too much.

https://github.com/wxiaoguang/playground/wiki/test-202e

$ xxd playground.wiki/test-202e.md
00000000: 6162 63e2 80ae 7879 7a                   abc...xyz

[silverwind]

GitHub only does it for code. I guess that is where it matters the most. So we could do two options with these defaults:

[ui]
CHARACTER_HIGHLIGHT=bidi
CHARACTER_HIGHLIGHT_INCLUDE_MARKUP=false

[wxiaoguang]

GitHub only does it for code.

Yup, for "source code" only. For markdown in code repo:

GitHub only does it for "source code", not for rendered contents.

---

Update: it seems that the "display invisible char" button on the github page is broken .... at least it doesn't work for me 😂

[silverwind]

Yeah, that's what I meant, source code, e.g. stuff that renders in a monospace font 😉.

[jirutka]

It is a Chinese unicode ’, my typo.

No, it’s not; U+2019 (’) is Right Single Quotation Mark. This is the true (typographic) single quotation mark, unlike U+0027 Apostrophe (').

This got my attention because of this nonsensual and stupid PR in one of my repositories replacing correct characters with their plainpoor ASCII alternatives. Please remove this ignorant “feature”.

[techknowlogick]

@jirutka there is a a recently merged config option to disable this highlighting. As for removing it (or disabling by default), due to it being a claimed security issue it is required to have, otherwise we risk having a CVE opened that'll apply to all versions (and ones in the future). If that happens then that prevents many people from running the software (there are many places that can't run software with known CVEs regardless of context, sometimes this is legislated by law too).

[wxiaoguang]

You could set this option: [ui].AMBIGUOUS_UNICODE_DETECTION=false (Gitea >= 1.21.3)