Redirected stdout encoding on Windows 11 (PowerShell Core 7.4) is cp1252 and Unicode Replacement Character (\uFFFD) causes encoding error?

[JohnL4]

Ok, this is causing me to pull my hair out a bit, and I'm still working on it, but I thought I'd throw this out:

I'm running snowddl-singledb plan (after having run snowddl-convert) and getting a UnicodeEncodeError trying to encode a \uFFFD, but only when I'm redirecting the plan sql to a file. (It turns out there's a lot of these in our Snowflake COMMENTS, probably because there's been a lot of copy/paste from government (? I think) spec docs.) Works fine when just emitting the SQL to the console.

Hacking in an exception handler, I see stdout.encoding is actually "cp1252", and I can reproduce the error with stdout.reconfigure( encoding="cp1252") followed by print( "\ufffd").

I have no idea where this encoding is coming from; it's my understanding that the whole stack should be defaulting to utf-8 (Windows 11, PowerShell Core 7.4, Python 3.12). Is it setuptools, maybe, doing something wacky in the .exe? snowddl-convert didn't have any trouble writing the yaml, but it generated it with the actual escape sequence "\uFFFD" (as opposed to simply jamming the actual unicode character into the output).

2nd question (after the implied one above) is: how to fix this? I guess I could hack in a stdout.reconfigure() call somewhere early in the run, but is there a better way? Some config file to tweak? I barely know what I'm doing, hence the wacky questions.

[littleK0i]

You should be able to skip all comments during conversion by replacing these lines in converters:
/~https://github.com/littleK0i/SnowDDL/blob/master/snowddl/converter/table.py#L49C7-L49C67

with something like "comment": None,

You may re-apply better comments later on, when the rest of migration is done. I bet unusual UTF-8 chars may cause issues in other types of software as well, so getting rid of it might be wise.

[JohnL4]

Thanks. I may wind up doing that, depending on what headaches I wind up encountering next.

For now, it looks like setting the PYTHONIOENCODING environment variable to "utf-8" does the trick.

( https://peps.python.org/pep-0587/#environment-variables for anybody else crazy enough to walk this road.)