I'm using Twisted's adbapi to asynchronously write items to a SQL db in the item pipeline. What happens if I insert a "time.sleep(1000)" in a runInteraction of the Twisted dbapi?
Does Twisted just see the code is blocking and jump to doing something else until the block stops? (i.e. I can do any blocking thing I want within runInteraction), or have I just made my code blocking?
runInteraction runs in a thread and does not block the main reactor thread.
Related
We have an existing codebase that is heavily Thread based which we're trying to expose through flask-socketio. I can't find a mechanism to have a greenlet wait for data from a Thread without blocking all of gevent or through a polling loop.
I thought maybe I could use an unbounded gevent Queue and call put from the Thread but that didn't seem to work. Also, the application is not performant enough with long polling so we can't use threads for socketio.
Is there a mechanism to receive data from the Thread in a greenlet without the greenlet blocking all of gevent?
The GUI of my program freezes while the program is doing its work. I created a mass import which can send X-thousand datarows via a called webservice into a database. The code is already very big and I cannot rewrite it for multithreading purpose.
I don't know how to do it. Any suggestions? If needed I will show some code, but at the moment I don't know what to show.
Firstly, you should rewrite it to use avoid synchronously doing this on the UI thread. If you do a lot of work on the UI thread, it simply will freeze the UI thread. There are a few options here:
If your web service proxy supports asynchronous calls, and if you're using VB 11, you can use Async / Await to call the web service asynchronously from the UI thread in an asynchronous method, and control will return back to the UI thread at the same point in the asynchronous method when the call has completed. It takes a little while to get your head round asynchrony, but this is probably the best option if it's possible.
You can use the Task Parallel Library to make calls on a different thread, but then you'll need to think carefully about how that thread is going to interact with your UI thread.
You can use BackgroundWorker to run some code on another thread, but report progress and completion back on the UI thread
You could potentially call Application.DoEvents between each web service call, to let the UI handle events. This is dangerous - it can lead to re-entrant code, so locks won't behave as you expect them to, and similar hard-to-diagnose errors. This should be your last option, if all else fails.
I have a celery task that, when called, simply ignites the execution of some parallel code inside a twisted reactor. Here's some sample (not runnable) code to illustrate:
def run_task_in_reactor():
# this takes a while to run
do_something()
do_something_more()
#celery.task
def run_task():
print "Started reactor"
reactor.callFromThread(run_task_in_reactor)
(For the sake of simplicity, please assume that the reactor is already running when the task is received by the worker; I used the signal #worker_process_init.connect to start my reactor in another thread as soon as the worker comes up)
When I call run_task.delay(), the task finishes pretty quickly (since it does not wait for run_task_in_reactor() to finish, only schedules its execution in the reactor). And, when run_task_in_reactor() finally runs, do_something() or do_something_more() can throw an exception, which will go unoticed.
Using pika to consume from my queue, I can use an ACK inside do_something_more() to make the worker notify the correct completion of the task, for instance. However, inside Celery, this does not seems to be possible (or, at least, I do't know how to accomplish the same effect)
Also, I cannot remove the reactor, since it is a requirement of some third-party code I'm using. Other ways to achieve the same result are appreciated as well.
Use reactor.blockingCallFromThread instead.
If an application executes a code block asynchronously and notifies the completion state in a callback, do these all leave execution on the main thread (the callback)?
What is the correct way to handle Core Graphics drawing and file operations in this callback?
I currently have these instructions running in the callback, but would like to trigger a thread to handle these instructions instead.
If you are doing any UI drawing, that has to be on the main thread. If you hvae these asynchronous blocks doing work now (ostensibly in a concurrent dispatch queue), why not just do all the work there, and when you have a finished product to show, only then message back on the main block. If your callback calls other methods in your class, the safest way to deal with concurrency is to define helper objects - small objects that take some input and product an output.
Create the helper, attach whatever data it needs, kick it off in a block on a queue, and when its done it messages back on the main thread that extracts the finished product, renders it, then deletes the helper object.
I dont have much experience in Core Graphics, but I've done a good amount of work on blocks.
In my opinion, if your application executes a code block asynchronously (on a new thread), then it's a good practice to the make callback or notification on the main thread.
Does the following code run on the main thread? Does "main queue" refer to the main thread?
dispatch_async(dispatch_get_main_queue(),
^{
// Some code
});
The async part of dispatch async vs sync is different than concurrent vs serial. Async means that the function returns immediately, sync means that it'll wait until the block is executed. Since the main thread/queue is serial, things are going to get executed in order - I believe this means that since you're asking it to async dispatch on the same thread you're dispatching from, it'll return immediately, wait till the end of the current run loop and anything else in the queue, and then execute your block.
This is more useful for inside a queue than it is on the main thread - you can process your data, let the UI know to update, and continue processing without waiting for everything to redraw, etc. That's why you'll often see a dispatch_async call to the main thread inside another dispatch_async(concurrent queue) instead of just a dispatch_sync.
Yes. From Apple developer site:
The dispatch framework provides a default serial queue for the
application to use. This queue is accessed via
dispatch_get_main_queue().
This is documented in multiple places, including the docs for dispatch_get_main_queue() itself. The Concurrency Programming Guide says:
The main dispatch queue is a globally available serial queue that executes tasks on the application’s main thread.