R 3.5.3 and 3.6.0 alpha Windows bug: UTF-8 characters in code are simplified to wrong ones
On 4/10/19 6:32 PM, Jeroen Ooms wrote:
On Wed, Apr 10, 2019 at 5:45 PM Duncan Murdoch <murdoch.duncan at gmail.com> wrote:
On 10/04/2019 10:29 a.m., Yihui Xie wrote:
Since it is "technically easy" to disable the best fit conversion and the best fit is rarely good, how about providing an option for code/package authors to disable it? I'm asking because this is one of the most painful issues in packages that may need to source() code containing UTF-8 characters that are not representable in the Windows native encoding. Examples include knitr/rmarkdown and shiny. Basically users won't be able to knit documents or run Shiny apps correctly when the code contains characters that cannot be represented in the native encoding.
Wouldn't things be worse with it disabled than currently? I'd expect the line containing the "?" to end up as NA instead of converting to "r".
I don't think it would be worse, because in this case R would not implicitly convert strings to (best fit) latin1 on Windows, but instead keep the (correct) string in its UTF-8 encoding. The NA only appears if the user explicitly forces a conversion to latin1, which is not the problem here I think. The original problem that I can reproduce in RGui is that if you enter "?" in RGui, R opportunistically converts this to latin1, because it can. However if you enter text which can definitely not be represented in latin1, R encodes the string correctly in UTF-8 form.
Rgui is a "Windows Unicode" application (uses UTF16-LE) but it needs to convert the input to native encoding before passing it to R, which is based on locales. However, that string is passed by R to the parser, which Rgui takes advantage of and converts non-representable characters to their \uxxxx escapes which are understood by the parser. Using this trick, Unicode characters can get to the parser from Rgui (but of course then still in risk of conversion later when the program runs). Rgui only escapes characters that cannot be represented, unfortunately, the standard C99 API for that implemented on Windows does the best fit. This could be fixed in Rgui by calling a special Windows API function and could be done, but with the mentioned risk that it would break existing uses that capture the existing behavior. This is the only place I know of where removing best fit would lead to correct representation of UTF-8 characters. Other places will give NA, some other escapes, code will fail to parse (e.g. "incomplete string", one can get that easily with source()). Tomas