[Bug report] Chinese characters are not handled correctly in Rterm for Windows
Thanks for the update. I believe I've fixed a part of the problem you
have reported, the crash while entering Chinese characters to the
console (e.g. via Pinyin, the error message about invalid multibyte
character in mbcs_get_next). The fix is in R-devel 74693 - Windows
function ReadConsoleInputA no longer works with multibyte characters (it
is not documented, probably a Windows bug, according to reports online
this problem exists since Windows 8, but I only reproduced/tested in
Windows 10). Could you please verify the crash is no longer happening on
your system?
Re the other problem, Chinese characters not being displayed. I found
this is caused by R calling setlocale(LC_CTYPE, *). Setting this to
"Chinese" and variants (code page 936) causes the problem, but running
in the "C" locale as per default works fine. This is easily reproduced
by an external program below - when setlocale() is called, the Chinese
character disappears from the output. A workaround is to run R with
environment variable LC_CTYPE=C. Could you please verify the printed
characters are ok with this setting? Would you have an explanation for
this behavior? It seems a bit odd - why would the CRT remove characters
valid in the console code page, when both the console code page and the
"setlocale" code page are 936.
Thanks
Tomas
??? #include <stdio.h>
??? #include <locale.h>
??? int main(int argc, char **argv) {
??????? //if (!setlocale(LC_CTYPE, "Chinese")) fprintf(stderr,
"setlocale failed\n");
??????? int chars[] = { 67, 196, 227, 68 };
??????? for(int i = 0; i < 4; i++) fputc(chars[i], stdout);
??????? fprintf(stdout, "\n");
??????? return 0;
??? }
On 04/28/2018 04:53 PM, Azure wrote:
Hi Tomas, Sorry for the delayed response. I have tested the problem on the latest R-devel build (2018-04-27 r74651), and it still exists. RGui is always fine with Chinese characters, but some IDEs rely on the CLI version of R (e.g. Visual Studio Code with R plugin).
Your example print("ABC\u4f60\u597dDEF") is printing two Chinese characters, right?
Yes. U+4F60, U+597D or C4E3, BAC3 in CP936.
Could you reproduce the problem with printing just one of the characters, say print("ABC\u4f60DEF") ?
Yes. The console output is pasted in [ https://paste.ubuntu.com/p/TYgZWhdgXK/ ] (to avoid gibberish in e-mail). The Active Code Page is 936 before and after running Rterm.
As a sanity check - does this display the correct characters in RGui?
Yes.
If you take the sequence of the "fputc" commands you captured by the debugger, and create a trivial console application to just run them - would the characters display correctly in the same terminal from which you run R.exe?
Yes. I created an Win32 Console Application in VS [ https://paste.ubuntu.com/p/h3NFV6nQvs/ ], and all the characters were displayed correctly in two ways. The WriteConsoleA variant uses the current console CP settings, and it should behave like fputc. I guess the Rterm uses its own console I/O mechanism so the 2nd parameter of fputc is not stdout's handle. (I tried to read the source but unable to figure out how it works). The crash in mbcs_get_next, which is also mentioned in the previous post, may be related to this mechanism. If you need further information, please let me know. Thanks, i at azurefx.name Tomas Kalibera <tomas.kalibera at gmail.com> 2018/4/5 22:42
Thank you for the report and initial debugging. I am not sure what is going wrong, we may have to rely on your help to debug this (I do not have a system to reproduce on). A user-targeted advice would be to use RGui (Rgui.exe). Does the problem also exist in R-devel? https://cran.r-project.org/bin/windows/base/rdevel.html Your example print("ABC\u4f60\u597dDEF") is printing two Chinese characters, right? The first one is C4E3 in CP936 (4F60 in Unicode) and the second one is BAC3 in CP936 (597D in Unicode)? Could you reproduce the problem with printing just one of the characters, say print("ABC\u4f60DEF") ? As a sanity check - does this display the correct characters in RGui? It should, and does on my system, as RGui uses Unicode internally. By correct I mean the characters shown e.g. here https://msdn.microsoft.com/en-us/library/cc194923.aspx https://msdn.microsoft.com/en-us/library/cc194920.aspx What is the output of "chcp" in the terminal, before you run R.exe? It may be different from what Sys.getlocale() gives in R. If you take the sequence of the "fputc" commands you captured by the debugger, and create a trivial console application to just run them - would the characters display correctly in the same terminal from which you run R.exe? Thanks Tomas