I'm pretty new with eventlet and have some questions on sleep()
I tested with a small piece of code.
At first I spawned 3 greenthreads and then called greenthread.sleep(0), then these 3 greenthreads all came to execute the functions in them. what's going on?
does sleep() mean execute all the greenthread spawned? what does the argument 0 we passed in mean?
Here is the code:
import eventlet
from eventlet import greenthread
from eventlet import event
evt = event.Event()
def func1():
print "starting func1"
evt.wait()
print "stopping func1"
def func2():
print "starting func2"
evt.wait()
print "stopping func2"
def func3():
evt.send()
gt1 = greenthread.spawn(func1)
gt2 = greenthread.spawn(func2)
gt3 = greenthread.spawn(func3)
greenthread.sleep(0)
That's a great question, it deserves a special place in Eventlet documentation.
eventlet.sleep(0) reschedules the calling greenthread to the end of run queue. If there were any other greenthreads waiting to run, they will execute now.
Current implementation detail of Eventlet has a certain guarantee that if you call sleep, the calling greenthread will not continue until all other greenthreads that are ready to execute are finished or came to similar wait state. Started as implementation detail, we keep it as a public API now: call sleep(0) to let others run.
Related
I am trying to develop part of system that has the following requirement:
send health status to a remote server(every X seconds)
receive request for executing/canceling CPU bound job(s)(for example - clone git repo, compile(using conan) it.. etc).
I am using the socketio.AsyncClient to handle these requirements.
class CompileJobHandler(socketio.AsyncClientNamespace):
def __init__(self, namespace_val):
super().__init__(namespace_val)
// some init variables
async def _clone_git_repo(self, git_repo: str):
// clone repo and return its instance
return repo
async def on_availability_check(self, data):
// the health status
await self.emit('availability_check', " all good ")
async def on_cancel_job(self, data):
// cancel the current job
def _reset_job(self):
// reset job logics
def _reset_to_specific_commit(self, repo: git.Repo, commit_hash: str):
// reset to specific commit
def _compile(self, is_debug):
// compile logics - might be CPU intensive
async def on_execute_job(self, data):
// **request to execute the job(compile in our case)**
try:
repo = self._clone_git_repo(job_details.git_repo)
self._reset_to_specific_commit(repo, job_details.commit_hash)
self._compile(job_details.is_debug)
await self.emit('execute_job_response',
self._prepare_response("SUCCESS", "compile successfully"))
except Exception as e:
await self.emit('execute_job_response',
self._prepare_response(e.args[0], e.args[1]))
finally:
await self._reset_job()
The problem with the following code is that when execute_job message arrives, there is a blocking code running that blocks the whole async-io system.
to solve this problem, I have used the ProcessPoolExecutor and the asyncio event loop, as shown here: https://stackoverflow.com/questions/49978320/asyncio-run-in-executor-using-processpoolexecutor
after using it, the clone/compile functions are executed in another process - so that almost achieves my goals.
the questions I have are:
How can I design the code of the process more elegantly?(right now I have some static functions, and I don't like it...)
one approach is to keep it like that, another one is to pre-initialize an object(let's call it CompileExecuter and create instance of this type, and pre-iniailize it prior starting the process, and then let the process use it)
How can I stop the process in the middle of its execution?(if I received on_cancel_job request)
How can I handle the exception raised by the process correctly?
Other approaches to handle these requirements are welcomed
I have the following simplified code:
async def asynchronous_function(*args, **kwds):
statement = await prepare(query)
async with conn.transaction():
async for record in statement.cursor():
??? yield record ???
...
class Foo:
def __iter__(self):
records = ??? asynchronous_function ???
yield from records
...
x = Foo()
for record in x:
...
I don't know how to fill in the ??? above. I want to yield the record data, but it's really not obvious how to wrap asyncio code.
While it is true that asyncio is intended to be used across the board, sometimes it is simply impossible to immediately convert a large piece of software (with all its dependencies) to async. Fortunately there are ways to combine legacy synchronous code with newly written asyncio portions. A straightforward way to do so is by running the event loop in a dedicated thread, and using asyncio.run_coroutine_threadsafe to submit tasks to it.
With those low-level tools you can write a generic adapter to turn any asynchronous iterator into a synchronous one. For example:
import asyncio, threading, queue
# create an asyncio loop that runs in the background to
# serve our asyncio needs
loop = asyncio.get_event_loop()
threading.Thread(target=loop.run_forever, daemon=True).start()
def wrap_async_iter(ait):
"""Wrap an asynchronous iterator into a synchronous one"""
q = queue.Queue()
_END = object()
def yield_queue_items():
while True:
next_item = q.get()
if next_item is _END:
break
yield next_item
# After observing _END we know the aiter_to_queue coroutine has
# completed. Invoke result() for side effect - if an exception
# was raised by the async iterator, it will be propagated here.
async_result.result()
async def aiter_to_queue():
try:
async for item in ait:
q.put(item)
finally:
q.put(_END)
async_result = asyncio.run_coroutine_threadsafe(aiter_to_queue(), loop)
return yield_queue_items()
Then your code just needs to call wrap_async_iter to wrap an async iter into a sync one:
async def mock_records():
for i in range(3):
yield i
await asyncio.sleep(1)
for record in wrap_async_iter(mock_records()):
print(record)
In your case Foo.__iter__ would use yield from wrap_async_iter(asynchronous_function(...)).
If you want to receive all records from async generator, you can use async for or, for shortness, asynchronous comprehensions:
async def asynchronous_function(*args, **kwds):
# ...
yield record
async def aget_records():
records = [
record
async for record
in asynchronous_function()
]
return records
If you want to get result from asynchronous function synchronously (i.e. blocking), you can just run this function in asyncio loop:
def get_records():
records = asyncio.run(aget_records())
return records
Note, however, that once you run some coroutine in event loop you're losing ability to run this coroutine concurrently (i.e. parallel) with other coroutines and thus receive all related benefits.
As Vincent already pointed in comments, asyncio is not a magic wand that makes code faster, it's an instrument that sometimes can be used to run different I/O tasks concurrently with low overhead.
You may be interested in reading this answer to see main idea behind asyncio.
I am using eztext to gather input for my program, but the input window is preventing the output window from being opened(I can't see the display until the input window is closed)
Any help on the problem described above is much appreciated, after a long trawl through the internet I can't seem to find any solutions that I can modify to suit my purposes, though if anyone sees anything I've missed, I would be grateful for a link.
Thanks again in advance
A.J.P
You should try using 2 threads: one for the first window and a second for the other. Like this, both window will open.
edit:
For example, these two threads will communicate with a list. But i's better to use classes:
from time import sleep
from threading import Thread
informations = []
def funct_a (informations):
while (1):
sleep (0.5)
print informations
def funct_b (informations):
a = 0
while (1):
sleep (1)
informations.append (a)
a += 1
Thread (target = funct_a, args = (informations, )).start ()
Thread (target = funct_b, args = (informations, )).start ()
I want to test the next class:
from random import randint
class End(object):
def __init__(self):
self.quips=['You dead', 'You broke everything you can','You turn you head off']
def play(self):
print self.quips[randint(0, len(self.quips)-1)]
exit(1)
I want to test it with nosetests so I could see that the class exits correctly with code 1. I tried differents variants but nosetest returns error like
File "C:\Python27\lib\site.py", line 372, in __call__
raise SystemExit(code)
SystemExit: 1
----------------------------------------------------------------------
Ran 1 test in 5.297s
FAILED (errors=1)
Ofcourse I can assume that it exits but I want for test to return OK status not error. Sorry if my question may be stupid. Im very new to python and I try to test something my very first time.
I would recommend using the assertRaises context manager. Here is an example test that ensures that the play() method exits:
import unittest
import end
class TestEnd(unittest.TestCase):
def testPlayExits(self):
"""Test that the play method exits."""
ender = end.End()
with self.assertRaises(SystemExit) as exitexception:
ender.play()
# Check for the requested exit code.
self.assertEqual(exitexception.code, 1)
As you can see in the traceback, sys.exit()* raises an exception called SystemExit when you call it. So, that's what you want to test for with nose's assert_raises(). If you are writing tests with unittest2.TestCase that's self.assertRaises.
*actually you used plain built-in exit() but you really should use sys.exit() in a program.
I have a task that I need to run asynchronously from the web page that triggered it. This task runs rather long, and as the web page could be getting a lot of these requests, I'd like celery to only run one instance of this task at a given time.
Is there any way I can do this in Celery natively? I'm tempted to create a database table that holds this state for all the tasks to communicate with, but it feels hacky.
You probably can create a dedicated worker for that task configured with CELERYD_CONCURRENCY=1 then all tasks on that worker will run synchronously
You can use memcache/redis for that.
There is an example on the celery official site - http://docs.celeryproject.org/en/latest/tutorials/task-cookbook.html
And if you prefer redis (This is a Django realization, but you can also easily modify it for your needs):
from django.core.cache import cache
from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)
class SingletonTask(Task):
def __call__(self, *args, **kwargs):
lock = cache.lock(self.name)
if not lock.acquire(blocking=False):
logger.info("{} failed to lock".format(self.name))
return
try:
super(SingletonTask, self).__call__(*args, **kwargs)
except Exception as e:
lock.release()
raise e
lock.release()
And then use it as a base task:
#shared_task(base=SingletonTask)
def test_task():
from time import sleep
sleep(10)
This realization is nonblocking. If you want next task to wait for the previous task change blocking=False to blocking=True and add timeout