I was using this script on python2 to launch an application and then immediately exit the python script without waiting for the child process to end. This was my code:
kwargs = {}
if platform.system() == 'Windows':
# from msdn [1]
CREATE_NEW_PROCESS_GROUP = 0x00000200 # note: could get it from subprocess
DETACHED_PROCESS = 0x00000008 # 0x8 | 0x200 == 0x208
kwargs.update(creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP)
kwargs.update(stderr=subprocess.PIPE)
elif sys.version_info < (3, 2): # assume posix
kwargs.update(preexec_fn=os.setsid)
else: # Python 3.2+ and Unix
kwargs.update(start_new_session=True)
subprocess.Popen([APP], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,**kwargs)
It works OK on python 2.7, but it doesn't start the expected 'APP' on python 3.7 without changes.
In order to make it work in python3, I found two independent workarounds:
Change stderr to this: stderr=subprocess.DEVNULL
or
Add a time.sleep(0.1) call after the Popen (before closing the script).
I assume this is not actually related to python, but to some event that needs to happen after the process gets opened before the python script can safely exit?
Any hints? I'd really like to know why it happens. Right now I simply added the sleep call.
Thank you
Related
This Python code shows how to call some process in Windows 10 and to send to it string commands, to read its string responses through stdin, stdout pipes of the process:
Python 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:37:50) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from subprocess import *
>>> p = Popen("c:/python38/python.exe", stdin=PIPE, stdout=PIPE)
>>> p.stdin.write(b"print(1+9)\n")
11
>>> p.communicate()
(b'10\r\n', None)
>>>
As you can see the python.exe process returned 10 as an answer to print(1+9). Now I want to do the same in Pharo (or Squeak): in Windows 10 OS - I suppose something similar, i.e. short, simple, understandable, really working.
I installed OSProcess, ProcessWrapper (they were missing in Pharo, also its strange that I got warning that they are not marked for Pharo 8.0 and were not checked to work in Pharo 8.0, but OK), and I tried ProcessWrapper, PipeableOSProcess (copy-pasted different snippets from the Web), etc - with zero success! The results were:
nothing happens, python.exe was not started
VM errors console was opened (white console in the bottom of the Pharo, which is controlled with F2 menu)
different exceptions
etc
Would somebody show me simple working example how to start a process and to to send it commands, read answers, then send again, and so on in some loop - I plan to have such communication in a detached thread and to use it as some service, because Pharo, Smalltalk in general is missing most bindings, so then I will use subprocess communication like in "good" old days...
I know how to call a command and to get its output:
out := LibC resultOfCommand: 'dir ', aDir.
but I am talking about another scenario: a communication with a running process interactively (for example, with SSH or similar like in the example above - python.exe).
PS. Maybe it's possible to do it with LibC #pipe:mode even?
Let me start with that the PipeableOsProcess is probably broken on Windows. I have tried it and it just opened a command line and nothing else (it does not freeze my Pharo 8). The whole OSProcess does not work correctly in my eyes.
So I took a shot at LibC which is supposed to not work with Windows.
I’m a module defining access to standard LibC. I’m available under Linux and OSX, but not under Windows for obvious reasons :)
Next is to say that Python's Windows support is probably much better than Pharo's.
The solution, which is more like a workaround using files, is to use LibC and #runCommand: (I tried to come up with a similar example as you had shown above):
| count command result outputFile errorFile |
count := 9+1. "The counting"
command := 'echo ', count asString. "command run at the command line"
outputFile := 'output'. "a file into which the output is redirected"
errorFile := 'error'. "a file where the error output is redirected "
result := LibC runCommand: command, "run the command "
' >', outputFile, "redirect the output to output file"
' 2>', errorFile.
"reading back the value from output file"
outputFile asFileReference contents lines.
"reading back the value from the error file - which is empty in this case"
errorFile asFileReference contents lines.
My current architecture is that at the start of my Snakefile I have a long running function somefunc which helps decide the "input" to rule all. I realized when I was running the workflow with slurm that somefunc is being executed by each job. Is there some variable I can access that defines whether the code is a submitted job or whether it is the main process:
if not snakemake.submitted_job:
config['layout'] = somefunc()
...
A solution which I don't really recommend is to make somefunc write the list of inputs to a tmp file so that slurm jobs will read this tmp file rather than reconstructing the list from scratch. The tmp file is created by whatever job is executed first so the long-running part is done only once.
At the end of the workflow delete the tmp file so that later executions will start fresh with new input.
Here's a sketch:
def somefunc():
try:
all_output = open('tmp.txt').readlines()
all_output = [x.strip() for x in all_output]
print('List of input files read from tmp.txt')
except:
all_output = ['file1.txt', 'file2.txt'] # Long running part
with open('tmp.txt', 'w') as fout:
for x in all_output:
fout.write(x + '\n')
print('List of input files created and written to tmp.txt')
return all_output
all_output = somefunc()
rule all:
input:
all_output,
rule one:
output:
all_output,
shell:
r"""
touch {output}
"""
onsuccess:
os.remove('tmp.txt')
onerror:
os.remove('tmp.txt')
Since jobs will be submitted in parallel, you should make sure that only one job writes tmp.txt and the others read it. I think the try/except above will do it but I'm not 100% sure. (Probably you want to use some better filename than tmp.txt, see the module tempfile. see also the module atexit) for exit handlers)
As discussed with #dariober it seems the cleanest to check whether the (hidden) snakemake directory has locks since they seem not to be generated until the first rule starts (assuming you are not using the --nolock argument).
import os
locked = len(os.listdir(".snakemake/locks")) > 0
However this results in a problem in my case:
import time
import os
def longfunc():
time.sleep(10)
return range(5)
locked = len(os.listdir(".snakemake/locks")) > 0
if not locked:
info = longfunc()
rule all:
input:
expand("test_{sample}", sample=info)
rule test:
output:
touch("test_{sample}")
run:
"""
sleep 1
"""
Somehow snakemake lets each rule reinterpret the complete snakefile, with the issue that all the jobs will complain that 'info is not defined'. For me it was easiest to store the results and load them for each job (pickle.dump and pickle.load).
I'm trying out $*ARGFILES.handles and it seems that it opens the files in binary mode.
I'm writing a zip-merge program, that prints one line from each file until there are no more lines to read.
#! /usr/bin/env perl6
my #handles = $*ARGFILES.handles;
# say $_.encoding for #handles;
while #handles
{
my $handle = #handles.shift;
say $handle.get;
#handles.push($handle) unless $handle.eof;
}
I invoke it like this: zip-merge person-say3 repeat repeat2
It fails with: Cannot do 'get' on a handle in binary mode in block at ./zip-merge line 7
The specified files are text files (encoded in utf8), and I get the error message for non-executable files as well as executable ones (with perl6 code).
The commented out line says utf8 for every file I give it, so they should not be binary,
perl6 -v: This is Rakudo version 2018.10 built on MoarVM version 2018.10
Have I done something wrong, or have I uncovered an error?
The IO::Handle objects that .handles returns are closed.
my #*ARGS = 'test.p6';
my #handles = $*ARGFILES.handles;
for #handles { say $_ }
# IO::Handle<"test.p6".IO>(closed)
If you just want get your code to work, add the following line after assigning to #handles.
.open for #handles;
The reason for this is the iterator for .handles is written in terms of IO::CatHandle.next-handle which opens the current handle, and closes the previous handle.
The problem is that all of them get a chance to be both the current handle, and the previous handle before you get a chance to do any work on them.
(Perhaps .next-handle and/or .handles needs a :!close parameter.)
Assuming you want it to work like roundrobin I would actually write it more like this:
# /usr/bin/env perl6
use v6.d;
my #handles = $*ARGFILES.handles;
# a sequence of line sequences
my $line-seqs = #handles.map(*.open.lines);
# Seq.new(
# Seq.new( '# /usr/bin/env perl6', 'use v6.d' ), # first file
# Seq.new( 'foo', 'bar', 'baz' ), # second file
# )
for flat roundrobin $line-seqs {
.say
}
# `roundrobin` without `flat` would give the following result
# ('# /usr/bin/env perl6', 'foo'),
# ('use v6.d', 'bar'),
# ('baz')
If you used an array for $line-seqs, you will need to de-itemize (.<>) the values before passing them to roundrobin.
for flat roundrobin #line-seqs.map(*.<>) {
.say
}
Actually I personally would be more likely to write something similar to this (long) one-liner.
$*ARGFILES.handles.eager».open».lines.&roundrobin.flat.map: *.put
:bin is always set in this type of objects. Since you are working on the handles, you should either read line by line as instructed on the example, or reset the handle so that it's not in binary mode.
I set CELERY_RESULT_BACKEND = "amqp" in celeryconfig.py
but I get:
>>> from tasks import add
>>> result = add.delay(3,5)
>>> result.ready()
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/djangoprojects/venv/local/lib/python2.7/site-packages/celery/result.py", line 105, in ready
return self.state in self.backend.READY_STATES
File "/djangoprojects/venv/local/lib/python2.7/site-packages/celery/result.py", line 184, in state
return self.backend.get_status(self.task_id)
File "/djangoprojects/venv/local/lib/python2.7/site-packages/celery/backends/base.py", line 414, in _is_disabled
raise NotImplementedError("No result backend configured. "
NotImplementedError: No result backend configured. Please see the documentation for more information.
I just went through this so I can shed some light on this. One might think for all of the great documentation stating some of this would have been a bit more obvious.
I'll assume you have both RabbitMQ up and functioning (it needs to be running), and that you have dj-celery installed.
Once you have that then all you need to do is to include this single line in your setting.py file.
BROKER_URL = "amqp://guest:guest#localhost:5672//"
Then you need to run syncdb and start this thing up using:
python manage.py celeryd -E -B --loglevel=info
The -E states that you want events captured and the -B states you want celerybeats running. The former enable you to actually see something in the admin window and the later allows you to schedule. Finally you need to ensure that you are actually going to capture the events and the status. So in another terminal run this:
./manage.py celerycam
And then finally your able to see the working example provided in the docs.. -- Again assuming you created the tasks.py that is says to.
>>> result = add.delay(4, 4)
>>> result.ready() # returns True if the task has finished processing.
False
>>> result.result # task is not ready, so no return value yet.
None
>>> result.get() # Waits until the task is done and returns the retval.
8
>>> result.result # direct access to result, doesn't re-raise errors.
8
>>> result.successful() # returns True if the task didn't end in failure.
True
Furthermore then you are able to view your status in the admin panel.
I hope this helps!! I would add one more thing which helped me. Watching the RabbitMQ Log file was key as it helped me identify that django-celery was actually talking to RabbitMQ.
Are you running django celery?
If so, you need to start a python shell in the context of django (or whatever the technical term is).
Type:
python manage.py shell
And try your commands from that shell
HI tried everything to work celery v3.1.25 with Django 1.8 version nothing worked..
Finally below line helped me ,feeling happy
app = Celery('documents',backend="celery.backends.amqp:AMQPBackend")
Setting backend="celery.backends.amqp:AMQPBackend" fixed my error.
I have nagios alerts set up to come through jabber with an http link to ack.
Is is possible there is a script I can run from a terminal on a remote workstation that takes the hostname as a parameter and acks the alert?
./ack hostname
The benefit, while seemingly mundane, is threefold. First, take http load off nagios. Secondly, nagios http pages can take up to 10-20 seconds to load, so I want to save time there. Thirdly, avoiding slower use of mouse + web interface + firefox/other annoyingly slow browser.
Ideally, I would like a script bound to a keyboard shortcut that simply acks the most recent alert. Finally, I want to take the inputs from a joystick, buttons and whatnot, and connect one to a big red button bound to the script so I can just ack the most recent nagios alert by hitting the button lol. (It would be rad too if the button had a screen on the enclosure that showed the text of the alert getting acked lol)
Make fun of me all you want, but this is actually something that would be useful to me. If I can save five seconds per alert, and I get 200 alerts per day I need to ack, that's saving me 15 minutes a day. And isn't the whole point of the sysadmin to automate what can be automated?
Thanks!
Yes, it's possible to ack nagios by parsing /var/lib/nagios3/retention.dat file.
See :
#!/usr/bin/env python
# -*- coding: utf8 -*-
# vim:ts=4:sw=4
import sys
file = "/var/lib/nagios3/retention.dat"
try:
sys.argv[1]
except:
print("Usage:\n"+sys.argv[0]+" <HOST>\n")
sys.exit(1)
f = open(file, "r")
line = f.readline()
c=0
name = {}
state = {}
host = {}
while line:
if "service_description=" in line:
name[c] = line.split("=", 2)[1]
elif "current_state=" in line:
state[c] = line.split("=", 2)[1]
elif "host_name=" in line:
host[c] = line.split("=", 2)[1]
elif "}" in line:
c+=1
line = f.readline()
for i in name:
num = int(state[i])
if num > 0 and sys.argv[1] == host[i].strip():
print(name[i].strip("\n"))
You simply have to put the host as parameter, and the script will displays the broken services.