Add support for Windows UTF-8 version of R

[SpecterShell]

An experimental build of R using UTF-8 as native encoding on Windows has been released recently, and here is the instruction: https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/howto.html

I'm on Windows 10 20H2 and has set $LANG to zh_CN.UTF-8. After setting the codepage to 65001(UTF-8), the encoding in Rterm successfully switches to UTF-8, but the encoding in radian remains unchanged.

In most of the time the terminal and the graph can still handle CJK characters properly. But sometimes some messy codes show, like the red text in the screenshot.

I thought this issue came from Python itself, so I set $PYTHONUTF8 to 1 to force Python to use UTF-8, and ran python -m radian instead of radian. However it doesn't work for radian. I also tried to directly change the encoding in radian, and then it crashed.

[randy3k]

These are the relevant lines for decoding and encoding: /~https://github.com/randy3k/rchitect/blob/cb6e77c860ef180bd7f47416c94185fb9db04370/rchitect/utils.py#L142

Is there any prebuilt binary? The steps seem a bit involved. It seems that there is.

[SpecterShell]

https://www.r-project.org/nosvn/winutf8/ucrt3/
And R-devel-win-80146-4517-4499.exe(or newer) is the installer of the prebuilt binary.

I just found that running demo(recursion) can also reproduce the issue while R is waiting for user's response. So there is no need to prepare building environment. Just install and test.

[randy3k]

Does something like Sys.setlocale(locale = "English_United States.utf8") fix it?

[randy3k]

I have to set this

to change the default locale of python applications.

But then, things are still not working well in radian:

r$> x = "床前明月光"

r$> print(x)
[1] "�����"

r$> Sys.getlocale()
[1] "LC_COLLATE=English_United States.utf8;LC_CTYPE=English_United States.utf8;LC_MONETARY=English_United States.utf8;LC_NUMERIC=C;LC_TIME=English_United States.utf8"

r$> l10n_info()
$MBCS
[1] TRUE

$`UTF-8`
[1] TRUE

$`Latin-1`
[1] FALSE

$codepage
[1] 65001

$system.codepage
[1] 65001

[randy3k]

randy3k/rchitect@7c0d72a should fix the garbled text issue. However, it requires checking the beta UTF-8 setting in Region settings.

[randy3k]

There is some discussion on enabling UTF-8 codepage on Windows, but it won't happen soon. So for now, the unicode setting in regional settings is the only way to enable UTF-8 on radian.

[SpecterShell]

In my opinion this issue occurs when radian calls R, system tends to use system-wide encoding settings instead of one set in the console, making R think that the current codepage is not UTF-8.
Setting the unicode setting may fix this problem by making all the applications use Unicode, but will make non-Unicode applications and even system corrupted.
Therefore, before there is a better way to solve this, I would use English as default in radian or rollback R to stable version.

[randy3k]

We have to wait until python supports Unicode codepage on Windows.

See https://groups.google.com/g/python-ideas/c/iPgyhq3_zyI/m/uGXrunPWAQAJ

[SpecterShell]

UTF-8 has become the default encoding now. 🤔
https://cran.r-project.org/bin/windows/base/NEWS.R-4.2.0.html

[SpecterShell]

The bare R term uses English by default utill the LANG is set. However, radian (or rchitect) always use system locale, resulting in garbled text.
A temporary workaround is to set LANG to en_US.UTF-8 at startup.

[randy3k]

I guess we'd need to force utf8 encoding for newer version of R.

[randy3k]

Hi @SpecterShell, would it work if you set the variable LANG using Sys.setenv(LANG = "en_US.UTF-8")?

[SpecterShell]

Works

[randy3k]

Then the solution for us is simpler, we could just set it if it is unset.

[randy3k]

@SpecterShell
Could you test if f016d46 fixes it.