Writing Unicode Text into Text File from R (in Windows)

Dear R Helpers,

See the Code:

a <- intToUtf8(1777)
show(a)
zz <- file(description="test.txt",open="w",encoding="UTF-8")
cat(a, file = zz)
close(zz)

in a Unicode aware environment (such as RGui console or RStudio Console)
you will see this as output:

[1] "??"


but the character is not written correctly in the file test.txt (which is
encoded in UTF-8 without BOM) :

<U+06F1>

The problem seems to be this: R changes text to the locale of system (for
me this is Arabic Windows (Codepage 1256) that does not have a relevant
code for U+06F1, then changes it back to UTF-8 and writes it into file.
What do I miss here?
  How can I write a Unicode string into a text file correctly?

Majid Einian,
Economics Researcher, Monetary and Banking Research Institute, Central Bank
of Islamic Republic of Iran, Tehran, IRAN
and
PhD Candidate in "Economics", Graduate School of Management and
Economics, Sharif University of Technology, Tehran, IRAN

	[[alternative HTML version deleted]]

On 14-02-04 5:49 AM, Majid Einian wrote:

Dear R Helpers,

See the Code:

a <- intToUtf8(1777)
show(a)
zz <- file(description="test.txt",open="w",encoding="UTF-8")
cat(a, file = zz)
close(zz)

in a Unicode aware environment (such as RGui console or RStudio Console)
you will see this as output:

[1] "?"


but the character is not written correctly in the file test.txt (which is
encoded in UTF-8 without BOM) :

<U+06F1>

The problem seems to be this: R changes text to the locale of system (for
me this is Arabic Windows (Codepage 1256) that does not have a relevant
code for U+06F1, then changes it back to UTF-8 and writes it into file.
What do I miss here?
  How can I write a Unicode string into a text file correctly?

There are a lot of places in R where it converts strings to the local encoding, perhaps too many. On the other hand, maybe Windows should be offering UTF-8 locales by now.

I haven't tested in your locale, but I believe writeLines() to a connection declared to be in a UTF-8 encoding will maintain the encoding.

 You can declare a file to be in encoding "UTF-8-BOM" if you want to ignore a BOM on input; I forget whether it will write one on output.  If it doesn't, you can always write one explicitly.

I was hoping to make some progress on this before R 3.1.0 so that more cases of writing strings to UTF-8 files would work, but time is running out.

Duncan Murdoch

Majid Einian,
Economics Researcher, Monetary and Banking Research Institute, Central Bank
of Islamic Republic of Iran, Tehran, IRAN
and
PhD Candidate in "Economics", Graduate School of Management and
Economics, Sharif University of Technology, Tehran, IRAN

        [[alternative HTML version deleted]]