incorrect cursor position for bold math text

[minrk]

Bold math text, such as 𝐚 (MATHEMATICAL BOLD SMALL A (U+1D41A), takes up two characters in cursor calculations, but is only one character.

Can be verified by inputting 𝐚 into any CodeMirror demo and calling editor.getCursor(). Appears to be related to #1813, possibly just another codepoint range to be added somewhere.

Reported downstream at JuliaLang/IJulia.jl#541 , jupyter/jupyter_client#259

[marijnh]

Is the problem that such a character 'counts' as two characters in string offsets? If so, that's intentional -- JavaScript (unfortunately) represents strings as UTF-16, so 'astral plane' characers like this take up two string indices, and short of writing our own string representation, there's not much JavaScript libraries can do about that.

[stevengj]

Yes, 𝐚 is in an astral plane (U+01d41a), and it appears that CodeMirror is using 16-byte UTF-16 code units (not Unicode characters) for cursor calculations, since this is how JavaScript indexes strings. Unrelated to #1813 (combining characters, not surrogate pairs).

Probably Jupyter needs to convert this to/from Unicode character counts itself (so that it has an encoding-independent index).

[minrk]

Thanks, CodeMirror's behavior matching underlying js details makes sense. Adding the workaround in Jupyter: jupyter/notebook#2509

[oeddyo]

@minrk this literally saved my life. Thanks!