Error in substring: invalid multibyte string
Thanks for the quick response Ivan. readLines with encoding='latin1' works for me (on Ubuntu). However I was more concerned with the inconsistency in results between substr and regexpr. I was expecting that if one of them errors because of an unknown encoding then the other should as well. Even better, if regexpr works, why shouldn't substr work as well? Incidentally the analogous stringi function stri_sub works fine in this case:
stringi::stri_sub("<I>Jens Oehlschl\xe4gel-Akiyoshi", 1, 100)
[1] "<I>Jens Oehlschl\xe4gel-Akiyoshi" But the stringi analog to nchar gives a similar warning:
stringi::stri_length("<I>Jens Oehlschl\xe4gel-Akiyoshi")
[1] NA
Warning message:
In stringi::stri_length("<I>Jens Oehlschl\xe4gel-Akiyoshi") :
invalid UTF-8 byte sequence detected; try calling stri_enc_toutf8()
On Sat, Jun 27, 2020 at 2:12 AM Ivan Krylov <krylov.r00t at gmail.com> wrote:
On Fri, 26 Jun 2020 15:57:06 -0700 Toby Hocking <tdhock5 at gmail.com> wrote:
invalid multibyte string at '<e4>gel-A<6b>iyoshi'
The server says that the text is UTF-8: curl -sI \ https://stat.ethz.ch/pipermail/r-devel/1999-November/author.html | \ grep Content-Type # Content-Type: text/html; charset=UTF-8 But it's not, at least not all of it. If you ask readLines to mark the text as Latin-1, you get Jens Oehlschl?gel-Akiyoshi without the mojibake and invalid multi-byte characters: x <- readLines( 'https://stat.ethz.ch/pipermail/r-devel/1999-November/author.html', encoding = 'latin1' )[28] substr(x, 1, 100) # [1] "<I>Jens Oehlschl?gel-Akiyoshi" The behaviour we observe when encoding = 'latin1' is not specified results from returned lines having "unknown" encoding. The substr() implementation tries to interpret such strings according to multi-byte C locale rules (using mbrtowc(3)). On my system (yours too, probably, if it's GNU/Linux or macOS), the multi-byte C locale encoding is UTF-8, and this Latin-1 string does not result in valid code points when decoded as UTF-8. -- Best regards, Ivan