Python 2.7.3 process.Popen() failures - process

I'm using python 2.7.3 on a Windows 7 64-bit machine currently (also developing in Linux 32 bit, Ubuntu 12.04) and am having odd difficulties getting python to communicate with the command prompt/terminal successfully. My code looks like this:
import subprocess, logging, platform, ctypes
class someClass (object):
def runTerminalCommand:
try:
terminalOrCmdLineCommand = self.toString()
process = subprocess.Popen(terminalOrCmdLineCommand, shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
output = process.stdout.readlines()
if len(output) < 1:
logging.exception("{} error : {}".format(self.cmd,process.stderr.readlines()))
raise ConfigError("cmd issue : {}".format(self.cmd))
return output
except ValueError as err:
raise err
except Exception as e:
logging.exception("Unexpected error : " + e.message)
raise ConfigError("unexpected error")
Now, I know that the self.toString() returned value will process correctly if I enter it manually, so I'm limiting this to an issue with how I'm sending it to the command line via the subprocess. I've read the documentation, and found that the subprocess.check_call() doesn't return anything if it encounters an error, so I'm using .Popen()
The exception I get is,
[date & time] ERROR: Unexpected error :
Traceback (most recent call last):
File "C:\[...]"
raise ConfigError("cmd issue : {}".format(self.cmd))
"ConfigError: cmd issue : [the list self.cmd printed out]"
What I am TRYING to do, is run a command, and read the input back. But I seem to be unable to automate the call I want to run. :(
Any thoughts? (please let me know if there are any needed details I left out)
Much appreciated, in advance.

The docs say:
Use communicate() rather than .stdin.write, .stdout.read or
.stderr.read to avoid deadlocks due to any of the other OS pipe
buffers filling up and blocking the child process.
You could use .communicate() as follows:
p = Popen(cmd, stdout=PIPE, stderr=PIPE)
stdout_data, stderr_data = p.communicate()
if p.returncode != 0:
raise RuntimeError("%r failed, status code %s stdout %r stderr %r" % (
cmd, p.returncode, stdout_data, stderr_data))
output_lines = stdout_data.splitlines() # you could also use `keepends=True`
See other methods to get subprocess output in Python.

Related

saleae logic 2 package in notebook

it is my first time with saleae.
I’ve installed it on my windows machine and launch a notebook. My problem
is that I can’t create an Saleae object. Here is my code
import saleae
from saleae import Saleae
s = Saleae()
I’m having this error message:
INFO:saleae.saleae:Could not connect to Logic software, attempting to launch it now
Output exceeds the size limit. Open the full output data in a text editor
ConnectionRefusedError Traceback (most recent call last) File ....\lib\site-packages\saleae\saleae.py:211, in Saleae.init(self, host, port, quiet, args) 210 self._s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) → 211 self._s.connect((host, port)) 212 except ConnectionRefusedError: ConnectionRefusedError: [WinError 10061] Aucune connexion n’a pu être établie car l’ordinateur cible l’a expressément refusée
how can I solve the issue ?
I found the solution by reverting back to Logic 1.2.40.

Airflow Xcom with SSHOperator

Im trying to get param from SSHOperator into Xcom and get it in python.
def decision_function(**context):
ti = context['ti']
output_ssh= ti.xcom_pull(task_ids='ssh_task')
print('ssh output is: {}'.format(output_ssh))
ls = SSHOperator(
task_id="ssh_task",
command= "ls",
ssh_hook = sshHook,
dag = mydag)
get_xcom = PythonOperator(
task_id='test',
python_callable=decision_function,
provide_context=True,
dag=mydag
ls >> get_xcom
I get the wrong result in the xcom_pull:
ssh output is: MTcySyBhaXJmbG93CTIuMUcgQXV0b0hpdHVtX05ld
Every thing is work fine with BashOperator but when I try to use the SSH, it not working currect.
I also try to change in the config the enable_xcom_pickling param, but still not working.
airflow: 2.1.4
linux
thank you.
The value of Xcom may be b64encoded depending on the value of enable_xcom_pickling as you can see in the source code
The issue you are facing has been discussed in PR which suggested to change functionality to be like other operators but the PR was not merged and you can see the reasons for it in the code review comments.
if you wish you can create custom version of the operator without the enable_xcom_pickling condition:
class MySSHOperator(SSHOperator):
def execute(self, context=None) -> Union[bytes, str]:
result: Union[bytes, str]
if self.command is None:
raise AirflowException("SSH operator error: SSH command not specified. Aborting.")
# Forcing get_pty to True if the command begins with "sudo".
self.get_pty = self.command.startswith('sudo') or self.get_pty
try:
with self.get_ssh_client() as ssh_client:
result = self.run_ssh_client_command(ssh_client, self.command)
except Exception as e:
raise AirflowException(f"SSH operator error: {str(e)}")
return result.decode('utf-8')

SQL CE connection with Python

Problem Statement: Extract data stored in the .sdf file to python.
System config: Win 10 Pro 64-bit, python 3.5.2-64 bit, adodbapi library, SQL CE 3.5
I am fairly new to programming and I have picked up Python as my first language to learn. Currently, I have hit a wall in process of connecting a SQL CE 3.5 .sdf file.
I have used the adodbapi library. I have searched the web extensively over the past week to find a solution to this problem and to make sure that my connection string is correct. I have tried multiple options/solutions provided on stack overflow and https://www.connectionstrings.com/microsoft-sqlserver-ce-oledb-3-5/.
Code:
import adodbapi
cons_str = "Provider=Microsoft.SQLSERVER.MOBILE.OLEDB.3.5;" \
"Data Source=D:\Work\Programming\Python\SQL_DataTransfer\LF.sdf;"\
"Persist Security Info=False;" \
"SSCE:Max Database Size=4091"
connection = adodbapi.connect(cons_str)
print(connection)
Error Message:
Traceback (most recent call last):
File "D:\Work\Programs\Python35.virtualenvs\sql_output\lib\site-packages\adodbapi\adodbapi.py", line 93, in make_COM_connecter
c = Dispatch('ADODB.Connection') #connect after CoIninialize v2.1.1 adamvan
NameError: name 'Dispatch' is not defined
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Work\Programs\Python35.virtualenvs\sql_output\lib\site-packages\adodbapi\adodbapi.py", line 112, in connect
co.connect(kwargs)
File "D:\Work\Programs\Python35.virtualenvs\sql_output\lib\site-packages\adodbapi\adodbapi.py", line 269, in connect
self.connector = connection_maker()
File "D:\Work\Programs\Python35.virtualenvs\sql_output\lib\site-packages\adodbapi\adodbapi.py", line 95, in make_COM_connecter
raise api.InterfaceError ("Windows COM Error: Dispatch('ADODB.Connection') failed.")
adodbapi.apibase.InterfaceError: Windows COM Error: Dispatch('ADODB.Connection') failed.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:/Work/Programming/Python/SQL_DataTransfer/SQL_CE_reportDB.py", line 8, in
connection = adodbapi.connect(cons_str)
File "D:\Work\Programs\Python35.virtualenvs\sql_output\lib\site-packages\adodbapi\adodbapi.py", line 116, in connect
raise api.OperationalError(e, message)
adodbapi.apibase.OperationalError: (InterfaceError("Windows COM Error: Dispatch('ADODB.Connection') failed.",), 'Error opening connection to "Provoider=Microsoft.SQLSERVER.MOBILE.OLEDB.3.5;Data Source=D:\Work\Programming\Python\SQL_DataTransfer\LF.sdf;Persist Security Info=False;SSCE:Max Database Size=4091"')
At this point any help is much appreciated.
Thank you,
Sincerely,
JD.
Looks like you have a typo:
Provoider => Provider
adodbapi version = '2.6.0.6' depends on pypiwin32 to be installed in your Python environment.
For adodbapi.py, from line 51:
if api.onIronPython:
from System import Activator, Type, DBNull, DateTime, Array, Byte
from System import Decimal as SystemDecimal
from clr import Reference
def Dispatch(dispatch):
type = Type.GetTypeFromProgID(dispatch)
return Activator.CreateInstance(type)
def getIndexedValue(obj,index):
return obj.Item[index]
else: # try pywin32
try:
import win32com.client
import pythoncom
import pywintypes
onWin32 = True
def Dispatch(dispatch):
return win32com.client.Dispatch(dispatch)
except ImportError:
import warnings
warnings.warn("pywin32 package (or IronPython) required for adodbapi.",ImportWarning)
def getIndexedValue(obj,index):
return obj(index)
In my situation, I traced the fact that the Dispatch function was not defined because an ImportError exception is generated at line 62, (import win32com.client) caught in the except block but for some reason the warning message was not displayed in my console.
Try:
pip install pypiwin32
and the ImportError exception described above should not be raised anymore.

Puzzling Python I/O error: [Errno 2] No such file or directory

I'm trying to grab an XML file from a server (using Python 3.2.3), but I keep getting this error that there's "no such file or directory". I'm sure the URL is correct, since it outputs the URL in the error message, and I can copy-n-paste it and load it in my browser. So I'm very puzzled how this could be happening. Here's my code:
import xml.etree.ElementTree as etree
class Blah(object):
def getXML(self,xmlurl):
tree = etree.parse(xmlurl)
return tree.getroot()
def pregameData(self,url):
try:
x = self.getXML('{0}linescore.xml'.format(url))
except IOError as err:
x = "I/O error: {0}".format(err)
return x
if __name__ == '__main__':
x = Blah()
l = ['http://gd2.mlb.com/components/game/mlb/year_2013/month_04/day_15/gid_2013_04_15_anamlb_minmlb_1/',
'http://gd2.mlb.com/components/game/mlb/year_2013/month_04/day_15/gid_2013_04_15_phimlb_cinmlb_1/',
'http://gd2.mlb.com/components/game/mlb/year_2013/month_04/day_15/gid_2013_04_15_slnmlb_pitmlb_1/'
]
for url in l:
pre = x.pregameData(url)
print(pre)
And it always returns this error:
I/O error: [Errno 2] No such file or directory: 'http://gd2.mlb.com/components/game/mlb/year_2013/month_04/day_15/gid_2013_04_15_anamlb_minmlb_1/linescore.xml'
I/O error: [Errno 2] No such file or directory: 'http://gd2.mlb.com/components/game/mlb/year_2013/month_04/day_15/gid_2013_04_15_phimlb_cinmlb_1/linescore.xml'
I/O error: [Errno 2] No such file or directory: 'http://gd2.mlb.com/components/game/mlb/year_2013/month_04/day_15/gid_2013_04_15_slnmlb_pitmlb_1/linescore.xml'
You can copy-n-paste those URL's and see the files do exist in those locations. I even copied the files & directories to localhost, and tried this as localhost in case the foreign server had some kind of block. It gave me the same errors, so that's not an issue. I wondered if Etree's parse() can't handle HTTP, but the documentation doesn't say anything about that, so I'm guessing that's not an issue either.
UPDATE: As suggested in the comments, I went with using open(), but it still returned the error. Importing & trying urllib.request.urlopen(url) returns an error that AttributeError: 'module' object has no attribute 'request'.
You're correct, xml.etree dosen't automatically download and parse urls, if you want to do that you'll need to download it yourself first (using urllib or requests...).
The documentation explicitly states that parse takes a filename or fileobject, if it would support an url i'm sure it would say so explicitly. lxml.etree.parse() for example does.

How do I set a backend for django-celery. I set CELERY_RESULT_BACKEND, but it is not recognized

I set CELERY_RESULT_BACKEND = "amqp" in celeryconfig.py
but I get:
>>> from tasks import add
>>> result = add.delay(3,5)
>>> result.ready()
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/djangoprojects/venv/local/lib/python2.7/site-packages/celery/result.py", line 105, in ready
return self.state in self.backend.READY_STATES
File "/djangoprojects/venv/local/lib/python2.7/site-packages/celery/result.py", line 184, in state
return self.backend.get_status(self.task_id)
File "/djangoprojects/venv/local/lib/python2.7/site-packages/celery/backends/base.py", line 414, in _is_disabled
raise NotImplementedError("No result backend configured. "
NotImplementedError: No result backend configured. Please see the documentation for more information.
I just went through this so I can shed some light on this. One might think for all of the great documentation stating some of this would have been a bit more obvious.
I'll assume you have both RabbitMQ up and functioning (it needs to be running), and that you have dj-celery installed.
Once you have that then all you need to do is to include this single line in your setting.py file.
BROKER_URL = "amqp://guest:guest#localhost:5672//"
Then you need to run syncdb and start this thing up using:
python manage.py celeryd -E -B --loglevel=info
The -E states that you want events captured and the -B states you want celerybeats running. The former enable you to actually see something in the admin window and the later allows you to schedule. Finally you need to ensure that you are actually going to capture the events and the status. So in another terminal run this:
./manage.py celerycam
And then finally your able to see the working example provided in the docs.. -- Again assuming you created the tasks.py that is says to.
>>> result = add.delay(4, 4)
>>> result.ready() # returns True if the task has finished processing.
False
>>> result.result # task is not ready, so no return value yet.
None
>>> result.get() # Waits until the task is done and returns the retval.
8
>>> result.result # direct access to result, doesn't re-raise errors.
8
>>> result.successful() # returns True if the task didn't end in failure.
True
Furthermore then you are able to view your status in the admin panel.
I hope this helps!! I would add one more thing which helped me. Watching the RabbitMQ Log file was key as it helped me identify that django-celery was actually talking to RabbitMQ.
Are you running django celery?
If so, you need to start a python shell in the context of django (or whatever the technical term is).
Type:
python manage.py shell
And try your commands from that shell
HI tried everything to work celery v3.1.25 with Django 1.8 version nothing worked..
Finally below line helped me ,feeling happy
app = Celery('documents',backend="celery.backends.amqp:AMQPBackend")
Setting backend="celery.backends.amqp:AMQPBackend" fixed my error.