Roslyn failed to build itself under system codepage 936

[pawchen]

I was building Roslyn with cibuild.cmd, this error happened when the bootstrapped compiler compiles a test project

My system locale is Simplified Chinese/CodePage936/GB2312, from cmd invoke cibuild.cmd failed to build the following file:

src\EditorFeatures\VisualBasicTest\Diagnostics\GenerateType\GenerateTypeTests.vb(695,114)
...
src\EditorFeatures\VisualBasicTest\Diagnostics\GenerateType\GenerateTypeTests.vb(724,6)

When I used NotePad++ to open this file, I saw

Then I changed encoding to UTF-8 Without BOM

When I use some Charset from European the content looks correct.

[Pilchie]

Tagging @jaredpar who has looked at some encoding issues before.

[jaredpar]

Thanks for reporting. This is a duplicate of #4264. This has already been fixed and will be included in update 1.

In the mean time you can work around this by adding the following entry int your vbproj file:

<CodePage>936</CodePage>

This will override the encoding detection logic to prefer this encoding instead of the standard fall back mechanisms.

[pawchen]

@jaredpar No, I don't think this is duplicate. I was compiling Roslyn. The cibuild.cmd bootstrap itself then use the new compiler to compile Roslyn, if it's fixed then this won't happen.

The issue you referred was about building files with encoding X under system code page X, while this issue is about building files with encoding X under system code page Y. Unfortunately there's no reasonable fix for a compiler if the files were from external project but to fix the external project source file, but fortunately & interestingly the files are from Roslyn itself, so the fix would be fix that "external" project source file which is from the Roslyn itself.

Converting the file encoding into UTF-8 is the correct way, which is what my PR #5415 is doing.

[dpoeschl]

Reopening on @DiryBoy's behalf for now and removing Resolution-Duplicate, as per #5424.

@jaredpar If you still believe this to be a duplicate and find me reopening this to be annoying, please chime in on the #5424 discussion. 😄

[jaredpar]

@DiryBoy @dpoeschl ah okay. I misread the original bug report.

[pawchen]

@jaredpar I didn't explain the issue well earlier, apology for that.