accented vowels

Mon, Aug 15, 2011 4:48 PM

Le 2011-08-15 ? 19:06, Duncan Murdoch a ?crit :

On 11-08-15 2:42 PM, Denis Chabot wrote:

Hi,

I usually do not give second thought to accented vowels and R handles everything fine thanks to UTF8 being used in my R scripts. But today I have a problem. Accented vowels do not behave properly when they were imported into R using list.files.

Maybe this is because  OS X (I'm using 10.6.8) still uses MacRoman for file names, though visually the names seem to have been read correctly into R.

An example is better than words:

sessionInfo()
R version 2.13.1 (2011-07-08)
Platform: x86_64-apple-darwin9.8.0/x86_64 (64-bit)

locale:
[1] fr_CA.UTF-8/fr_CA.UTF-8/C/C/fr_CA.UTF-8/fr_CA.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base


This does not cause problem:
a = c("1_MO2 crevettes po2crit.Rda", "1_MO2 soles S?te sda.Rda", "1_MO2 turbots po2crit.Rda"); a
[1] "1_MO2 crevettes po2crit.Rda" "1_MO2 soles S?te sda.Rda"    "1_MO2 turbots po2crit.Rda"

a2 = gsub(" S?te", "S", a); a2
[1] "1_MO2 crevettes po2crit.Rda" "1_MO2 solesS sda.Rda"        "1_MO2 turbots po2crit.Rda"


but if instead of creating the vector within the R script, I read it as a series of file names, the substitution does not work. I am sorry that I cannot make this a reproducible example as it requires the 3 files to exist on your computer, but you could create 3 dummy files having the same names in the directory of your choice.

don = file.path("donn?es/")
b = list.files(path = don, pattern = "1_MO2"); b
[1] "1_MO2 crevettes po2crit.Rda" "1_MO2 soles S?te sda.Rda"     "1_MO2 turbots po2crit.Rda"

b2 = gsub(" S?te", "S",  b); b2
[1] "1_MO2 crevettes po2crit.Rda" "1_MO2 soles S?te sda.Rda"     "1_MO2 turbots po2crit.Rda"

I am puzzled and also "stuck". For now I'll modify the file name, but I need to be able to handle such names at some point.

Any advice?


Possibly your system really is using MacRoman or some other local encoding; in that case, iconv(x, "", "UTF-8") should convert from the local encoding to UTF-8.

I think declaring everything to be UTF8 may be sufficient.  When I use list.files(), I see the encoding listed as "unknown", but

x <- list.files()
Encoding(x) <- "UTF-8"

works.  However, the iconv() method should be safer.

Duncan Murdoch

Hi Duncan, 

iconv() confirmed what I suspected: there was no problem with the encoding of the result of list.files, and if there had been one, the "?" would not have looked like a "?". Therefore, I got nonsense when treating this "?" as MacRoman to be converted into UTF-8:

iconv(b, from="MacRoman", to="UTF-8")
[1] "1_MO2 crevettes po2crit.Rda" "1_MO2 soles Se??te sda.Rda"  "1_MO2 turbots po2crit.Rda"  

It is not clear however that R considered b to be UTF=8:
Encoding(b)
[1] "unknown" "unknown" "unknown"

so I followed your suggestion:

Encoding(b) <- "UTF-8"
Encoding(b)
[1] "unknown" "UTF-8"   "unknown"

but gsub still did not work:
b2 = gsub(" S?te", "S",  b); b2  
[1] "1_MO2 crevettes po2crit.Rda" "1_MO2 soles S?te sda.Rda"     "1_MO2 turbots po2crit.Rda"  

I do not know why gsub worked with example "a" but not "b" in the example shown in my original message. Strange and frustrating.

Denis

accented vowels

Thread (7 messages)