Why is sys.stdin.encoding under PyDev / Jython different from standalone Jython console? - jython

Question

I am trying to get python-gnupg to work in Jython under PyDev under Eclipse. There is a problem which seems to be caused by the sys.stdin.encoding, which is tested by python-gnupg at it's initialisation.
I found that the following script
import sys
print sys.stdin.encoding
outptus cp850 when I run it from Python standalone in a console and from Jython standalone in a console. When I run it in Python under PyDev, it outputs Cp1252, and when I run it in Jython under PyDev, I get None.
Looking at python-gnupg, both encodings cp850 and Cp1252 seem to be ok, but if it's None, it raises an exception:
File "C:\Python27\lib\site-packages\gnupg.py", line 487, in __init__
self._collect_output(p, result, stdin=p.stdin)
File "C:\Python27\lib\site-packages\gnupg.py", line 561, in _collect_output
stderr = codecs.getreader(self.encoding)(process.stderr)
File "C:\jython2.5.2\Lib\codecs.py", line 920, in getreader
return lookup(encoding).streamreader
at org.python.core.codecs.normalizestring(codecs.java:101)
at org.python.core.codecs.lookup(codecs.java:75)
at org.python.modules._codecs.lookup(_codecs.java:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
java.lang.NullPointerException: java.lang.NullPointerException
Looks like there are differences between PyDev and standalone encodings, as well as no (?) encoding for PyDev/Jython? I'd really like to use the convenient PyDev development environment, but how can I get the sys.stdin.encoding to be something sensible when developing a Jython program?
Update 1
Maybe the problem is related to one of the following Jython bug reports:
http://bugs.jython.org/issue1839
http://bugs.jython.org/issue1807 (see msg6717)
But then again, these seem to be independent from PyDev/Eclipse..?

Answer 1

It's actually a Jython issue:
In PySystemState.java, when setting up the encodings, Jython will skip setting any encoding if it's not a tty console:
private void initEncoding() {
String encoding = registry.getProperty(PYTHON_CONSOLE_ENCODING);
if (encoding == null) {
return;
}
for (PyFile stdStream : new PyFile[] {(PyFile)this.stdin, (PyFile)this.stdout,
(PyFile)this.stderr}) {
if (stdStream.isatty()) {
stdStream.encoding = encoding;
}
}
}
So, a workaround could be creating a java module to set that (as encoding is a public field of PyFile, but does not have a setter from the Jython side -- or you could use java reflection to set it)...
Maybe you could ask the Jython guys why this is made in the first place (I think the encoding could be set even if it was not a tty device, but I'm not sure which implications could it have and why it's done that way).
EDIT to summarise the results of the many comments below, between the developers of Jython and PyDev as well as the author of the original question (Philip Jenvey, Fabio Zadrozny and Christian Gelinek, resp.):
During the progress, a couple of test scripts were discussed and developed:
A Python unit test written by Philip to test different encodings set by minimal custom created sitecusomize.py scripts. Fabio thinks that the reason for it not actually setting the encoding is because sys.stdin.encoding is inherited from the parent process instead configured by the unit test script.
A Java test program written by Christian which sets up custom I/O streams for creating a Python subprocess, which was found by Fabio to be more similar to how PyDev sets up the subprocess.
As this didn't work as expected by Fabio, he remembered that PyDev also uses the PYTHONIOENCODING environment variable.
As was pointed out by Philip, the current (2.5) version of Jython doesn't support the PYTHONIOENCODING environment variable, which is used by PyDev (in combination with sitecusomize.py) to set the encoding:
You can log a bug for it, although PYTHONIOENCODING was only added in CPython 2.6, so Jython probably wouldn't support it until 2.7 (Jython's skipping 2.6).

Answer 2

I believe you should be able to explicitly encoding on the launch properties.
Run configurations -> Python run -> Xxx -> Common tab -> Encoding.
PyDev inherits some settings from global Eclipse settings. On Linux the encoding is sane UTF-8 everywhere, but some cranky operating systems cause their own issues..
Also you can probably detect encoding in your .py main module and have PyDev specific hack to fix problems related to your configuration.