UFT-8 multibyte characters are broken when using the "Pull" function on your Remote Sketchbook

[BlackBrix]

Describe the problem

When you do a "Pull" on a file in your Remote Sketchbook that contains UTF-8 multibyte characters, these characters are not displayed correct in Arduino IDE.

To reproduce

1. Create a sketch in Arduino Web Editor that contains UFT-8 multibyte characters.
   Here's my shared one as an example: https://create.arduino.cc/editor/Dirk67/d86881f0-faf3-48cd-8097-ce9915bb1ca0/preview .
2. Sync Remote Sketchbook in Arduino IDE.
3. Pull the sketch you created in step 1.
4. Open this sketch in Arduino IDE
5. Compare the display of the UFT-8 multibyte characters within Arduino IDE to the characters as they are displayed in "Arduino Web Editor".

Expected behavior

UFT-8 multibyte characters, should always display correctly.

Arduino IDE version

2.0.0-beta.9

Operating system

Windows

Operating system version

10

Additional context

Additional reports:

• Emoji not displaying properly when pulling sketch from cloud arduino editor #1874
• Minor bug: Special characters (in comments) not handled correct #1509

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the latest nightly build
• My report contains all necessary details

[kittaakos]

Quality bug report 🙏 Thank you!

This issue is related to #634, at least on the code level, and will be fixed soon.

After the fix, IDE2 can show all the characters for a sketch pulled from the cloud:

---

Does IDE2 want to highlight the invisible, ambiguous, and non-basic ASCII characters in the editor (leave as is) or hide them as IDE2 did for the output (#1375)

Currently, it's like this:

449.mp4

What's your opinion on this, @per1234? Thank you!

[per1234]

What's your opinion on this, @per1234?

I think it is a very useful feature. It is common to see forum posts from people asking for help with cryptic compilation errors like:

C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:3:16: error: stray '\342' in program
   Serial.print(“don’t use smart quotes in code”);
                ^
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:3:17: error: stray '\200' in program
   Serial.print(“don’t use smart quotes in code”);
                 ^
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:3:18: error: stray '\234' in program
   Serial.print(“don’t use smart quotes in code”);
                  ^
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:3:22: error: stray '\342' in program
   Serial.print(“don’t use smart quotes in code”);
                      ^
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:3:23: error: stray '\200' in program
   Serial.print(“don’t use smart quotes in code”);
                       ^
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:3:24: error: stray '\231' in program
   Serial.print(“don’t use smart quotes in code”);
                        ^
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:3:51: error: stray '\342' in program
   Serial.print(“don’t use smart quotes in code”);
                                                   ^
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:3:52: error: stray '\200' in program
   Serial.print(“don’t use smart quotes in code”);
                                                    ^
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:3:53: error: stray '\235' in program
   Serial.print(“don’t use smart quotes in code”);
                                                     ^
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:4:18: error: stray '\342' in program
   Serial.println(‘!’);
                  ^
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:4:19: error: stray '\200' in program
   Serial.println(‘!’);
                   ^
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:4:20: error: stray '\230' in program
   Serial.println(‘!’);
                    ^
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:4:22: error: stray '\342' in program
   Serial.println(‘!’);
                      ^
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:4:23: error: stray '\200' in program
   Serial.println(‘!’);
                       ^
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:4:24: error: stray '\231' in program
   Serial.println(‘!’);
                        ^
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino: In function 'void setup()':
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:3:19: error: 'don' was not declared in this scope
   Serial.print(“don’t use smart quotes in code”);
                   ^~~
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:3:19: note: suggested alternative: 'tone'
   Serial.print(“don’t use smart quotes in code”);
                   ^~~
                   tone
C:\Users\per\Documents\Arduino\sketch_feb21a\sketch_feb21a.ino:4:25: error: expected primary-expression before ')' token
   Serial.println(‘!’);
                         ^

This is caused by website software or word processors prettying up code text by doing things like replacing correct straight quotes with "smart quotes".

So I would leave the feature enabled, but we can consider refining the default "Unicode Highlight" settings if we see indications that specific aspects of the current settings are producing a poor UX. Unfortunately I see that the most common culprit: the curved double quote is not even highlighted with the current defaults.