I am calling tasks in celery with a RabbitMQ broker on an Ubuntu box, but just getting set up using Redis as the result backend. I can find task results, but they look like ""\x80\x02}q\x01(U\x06statusq\x02U\aSUCCESSq\x03U\ttracebackq\x04NU\x06resultq\x05}q\x06(X\x06\x00\x00\x00result}q\a(X\x06\x00\x00\x00statusK\x01X\r\x00\x00\x00total_resultsM\xf4\x01X\a\x00\x00\x00matches]q\b(}q\t(X\a\x00\x00\x00players]q\n(}q\x0b(X\a\x00\x00\x00hero_idK\x15X\n\x00\x00\x00account_idI4294967295\nX\x0b\x00\x00\x00player_slotK\x00u}q\x0c(X\a\x00\x00\x00hero_idK\x0cX\n\x00\x00\x00account_idI4294967295\nX\x0b\x00\x00\x00player_slotK\x01u}q\r(X\a\x00\x00\x00hero_idK\x1bX\n\x00\x00\x00account_i...."
My default celery encoding is ASCII, and Redis does not appear to have an encoding specified in its base conf.
utils.encoding.default_encoding()
'ascii'
How should I go about turning this text into something meaningful? I cannot tell how this is encoded on sight; any suggested decodings to try?
The result is pickled by default into a utf-8 string (see task serializers). You can inspect the payload manually with:
import pickle
s = "\x80\x02}q..."
obj = pickle.loads(s)
print obj
pickle is generally fine unless you are operating in a polyglot environment, and then JSON or msgpack are fine solutions.
Related
I am using Python's Paramiko library to SSH a remote machine and fetch some output from command-line. I see a lot of junk printing along with the actual output. How to get rid of this?
chan1.send("ls\n")
output = chan1.recv(1024).decode("utf-8")
print(output)
[u'Last login: Wed Oct 21 18:08:53 2015 from 172.16.200.77\r', u'\x1b[2J\x1b[1;1H[local]cli#BENU>enable', u'[local]cli#BENU#Configure',
I want to eliminate, [2J\x1b[1;1H and u from the output. They are junk.
It's not a junk. These are ANSI escape codes that are normally interpreted by a terminal client to pretty print the output.
If the server is correctly configured, you get these only, when you use an interactive terminal, in other words, if you requested a pseudo terminal for the session (what you should not, if you are automating the session).
The Paramiko automatically requests the pseudo terminal, if you used the SSHClient.invoke_shell, as that is supposed to be used for implementing an interactive terminal. See also How do I start a shell without terminal emulation in Python Paramiko?
If you automate an execution of remote commands, you better use the SSHClient.exec_command, which does not allocate the pseudo terminal by default (unless you override by the get_pty=True argument).
stdin, stdout, stderr = client.exec_command('ls')
See also What is the difference between exec_command and send with invoke_shell() on Paramiko?
Or as a workaround, see How can I remove the ANSI escape sequences from a string in python.
Though that's rather a hack and might not be sufficient. You might have other problems with the interactive terminal, not only the escape sequences.
You particularly are probably not interested in the "Last login" message and command-prompt (cli#BENU>) either. You do not get these with the exec_command.
If you need to use the "shell" channel due to some specific requirements or limitations of the server, note that it is technically possible to use the "shell" channel without the pseudo terminal. But Paramiko SSHClient.invoke_shell does not allow that. Instead, you can create the "shell" channel manually. See Can I call Channel.invoke_shell() without calling Channel.get_pty() beforehand, when NOT using Channel.exec_command().
And finally the u is not a part of the actual string value (note that it's outside the quotes). It's an indication that the string value is in the Unicode encoding. You want that!
This is actually not junk. The u before the string indicates that this is a unicode string. The \x1b[2J\x1b[1;1H is an escape sequence. I don't know exactly what it is supposed to do, but it appears to clear the screen when I print it out.
To see what I mean, try this code:
for string in output:
print string
I have an GUI application, which is quite big.I have it in Python 2.7. Since Python 2 is no longer being updated, I converted my application to Python 3.8 using 2to3 module. I am facing this problem and have no idea how to solve it. I referred some of the similar problems but did not get anywhere. I have the following error:
QtCore.qRegisterResourceData(0x01, qt_resource_struct, qt_resource_name, qt_resource_data)
TypeError: qRegisterResourceData(int, bytes, bytes, bytes): argument 2 has unexpected type 'str'
What should I do to get pass this issue?
Resource files on PyQt are actually python scripts with base64 encoded data.
When porting to newer systems (both python 3 and Qt5) requires proper updating of those files.
Generally, it can be done by calling again the pyrcc command (pyrcc5 or pyrcc5.exe if both Qt versions are installed), but they can be manually ported, considering the following aspects:
the import statement has obviously be modified to PyQt5;
all variables (qt_resource_data and qt_resource_name) are bytes literals and require the b'...' prefix;
from PyQt5 import QtCore
qt_resource_data = b"\
-- raw data --
"
qt_resource_name = b"\
-- raw data --
"
I have an Apache web server and I made a python script to run a command. Command that I'm running is launching a ROS launch file, that is working indefinitely. I would like to read output from the subprocess live and display it in the page. With my code so far I could only manage to make output to be printed after I terminate the process. I've tried all kinds of solutions from the web but none of them seem to work
command = "roslaunch package test.launch"
proc = subprocess.Popen(
command,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
env=env,
shell=True,
bufsize=1,
)
print "Content-type:text/html\r\n\r\n"
for line in iter(proc.stdout.readline, ''):
strLine = str(line).rstrip()
print(">>> " + strLine)
print("<br/>")
The problem is that the output of roslaunch is being buffered. subprocess is not the best tool for real-time output processing in such situation, but there is a perfect tool for just that task in Python: pexpect. The following snippet should do the trick:
import pexpect
command = "roslaunch package test.launch"
p = pexpect.spawn(command)
print "Content-type:text/html\r\n\r\n"
while not p.eof():
strLine = p.readline()
print(">>> " + strLine)
print("<br/>")
Andrzej Pronobis' answer above suffices for UNIX-based systems but the package pexpect does not work as effectively as one would expect for Windows in certain particular scenarios. Here, spawn() doesn't work for Windows as expected. We still can use it with some alterations that can be seen here in the official docs.
The better way here might be to use wexpect (official docs here). It caters to Windows alone.
I have a wikipedia dump compressed with bzip2 (downloaded from http://dumps.wikimedia.org/enwiki/), but I don't want to unpack it: I want to process it while decompressing on the fly.
I know that it's possible to do it in plain Java (see e.g. Java - Read BZ2 file and uncompress/parse on the fly), but I was wondering how do it in Apache Flink? What I probably need is something like https://github.com/whym/wikihadoop but for Flink, not Hadoop.
It is possible to read compressed files in the following formats in Apache Flink:
org.apache.hadoop.io.compress.BZip2Codec
org.apache.hadoop.io.compress.DefaultCodec
org.apache.hadoop.io.compress.DeflateCodec
org.apache.hadoop.io.compress.GzipCodec
org.apache.hadoop.io.compress.Lz4Codec
org.apache.hadoop.io.compress.SnappyCodec
As you can see from the package names, Flink does this using Hadoop's InputFormats.
This is an example for reading gz files using Flink's Scala API:
(You need at least Flink 0.8.1)
def main(args: Array[String]) {
val env = ExecutionEnvironment.getExecutionEnvironment
val job = new JobConf()
val hadoopInput = new TextInputFormat()
FileInputFormat.addInputPath(job, new Path("/home/robert/Downloads/cawiki-20140407-all-titles.gz"))
val lines = env.createHadoopInput(hadoopInput, classOf[LongWritable], classOf[Text], job)
lines.print
env.execute("Read gz files")
}
Apache Flink has only build-in support for .deflate files. Adding support for more compression codecs is easy to do, but hasn't been done yet.
Using HadoopInputFormats with Flink doesn't cause any performance loss. Flink has build-in serialization support for Hadoop's Writable types.
How do I read in a text file in python 3.3.3 and store it in a variable? I'm struggling with this unicode coming from python 2.x
Given this file:
utf-8: áèíöû
This works as you expect (IFF utf-8 is your default encoding):
with open('/tmp/unicode.txt') as f:
variable=f.read()
print(variable)
It is better to explicitly state your intensions if you are unsure what the default is by using a keyword argument to open:
with open('/tmp/unicode.txt', encoding='utf-8') as f:
variable=f.read()
The keyword encodings supported are in the codec module. (For Python 2, you need to use codecs open to open the file rather than Python 2's open BTW.)