gsub, utf-8 replacements and the C-locale

Hi all,

I'd like to discuss a infelicity/possible bug with gsub.  Take the
following function:

f <- function(x) {
 gsub("\u{A0}", " ", gsub(" ", "\u{A0}", x))
}

As you might expect, in utf-8 locales it is idempotent:

Sys.setlocale("LC_ALL", "UTF-8")
f("x y")
# [1] "x y"

But in the C locale it is not:

Sys.setlocale("LC_ALL", "C")
f("x y")
# [1] "x\302\240y"

This seems weird to me. (And caused a bug in a package because I
didn't realise some windows users have a non-utf8 locale)

I'm not sure what the correct resolution is.  Should the encoding of the output of gsub be utf-8 if either the input or replacement is utf-8?

It is if the input is UTF-8 but only then - that is what is causing the asymmetry. Part of the problem is that you cannot declare 7-bit string as UTF-8 (even though it is valid) so you can't work around it by forcing the encoding.

 In non-utf-8 locales should the encoding of "\u{A0}" be bytes?

No, because the whole point of the encoding is to define the content. "\ua0" defines one unicode character whereas "\302\240" defines two bytes with unknown meaning. The meaning of UTF-8 encoded strings is still valid in non-UTF-8 locales and the reason why your can work with UTF-8 strings in R irrespective of the locale (very useful thing).

I would suggest to handle the special case of 7-bit input and UTF-8 replacement such that it results in UTF-8 output (as opposed to bytes output with happens now). The relevant code is somewhat convoluted (and more so in R-devel) so I'm not volunteering to do it, though.

Just to make things more clear - the current result (in C locale):

gsub(" ","\ua0", "foo bar")

[1] "foo\302\240bar"

Possibly desired result:

gsub(" ","\ua0", "foo bar")

[1] "foo<U+00A0>bar"

Cheers,
Simon