QThread - changing data in global list - different values - locking

I am using QThread to do some time intensive calculations to keep GUI from freezing. In the QThread I am accessing and changing global lists many times during the thread life span, however I am unable to get the same result as if it were just on the main thread.
I would assume you had to perform some kind of lock, but i'm new to QThread and don't know how to implement it.
#Main Thread
self.runGasAnalysisThread = GasAnalysisThread()
self.runGasAnalysisThread.start()
#QThread
class GasAnalysisThread(QtCore.QThread):
"""Performs gas analysis function"""
def __init__(self,parent = None):
super().__init__(parent)
def run(self):
try:
boolValue = True
while True:
#Change lists here
float(Data.TestList[0])+ 1 #Data is another module I am using to store variables
Again, moving the code to the main thread works correctly, but as soon as I do it with QThread I get different results.
How would I implement a locking mechanism to keep this from happening?

It is common to be confused while using Qt's threading, as one would think that subclassing QThread would be the right path.
Truth is that a QThread is the Qt thread object where your process is actually run, which means that you'll need a separate class for it and move its instance in a QThread. Subclassing QThread is usually unnecessary.
If you need any kind of interaction between the "worker" (the object that does the processing) and the main thread (as in the GUI), it's good practice to use Qt's signals.
In this example I'm using a button to start the processing, once the processor is started it disables the button, and re-enables it once it signals that the process has finished.
class Worker(QtCore.QObject):
stateChanged = QtCore.pyqtSignal(bool)
def __init__(self):
super().__init__()
def run(self):
self.stateChanged.emit(True)
try:
boolValue = True
while True:
# this doesn't make much sense, as it doesn't seem to do anything;
# I'll leave it just for the sake of the argument
float(Data.TestList[0]) + 1
except:
pass
self.stateChanged.emit(False)
class SomeWidget(QtWidgets.QWidget):
def __init__(self, parent=None):
super().__init__(parent)
layout = QtWidgets.QHBoxLayout()
self.setLayout(layout)
self.startButton = QtWidgets.QPushButton('Start')
layout.addWidget(self.startButton)
self.worker = Worker()
self.workerThread = QtCore.QThread()
self.worker.moveToThread(self.workerThread)
self.workerThread.started.connect(self.worker.run)
self.startButton.clicked.connect(self.workerThread.start)
self.worker.stateChanged.connect(self.startButton.setDisabled)

Related

Passing a Queue with concurrent.futures regardless of executor type

Working up from threads to processes, I have switched to concurrent.futures, and would like to gain/retain flexibility in switching between a ThreadPoolExecutor and a ProcessPoolExecutor for various scenarios. However, despite the promise of a unified facade, I am having a hard time passing multiprocessing Queue objects as arguments on the futures.submit() when I switch to using a ProcessPoolExecutor:
import multiprocessing as mp
import concurrent.futures
def foo(q):
q.put('hello')
if __name__ == '__main__':
executor = concurrent.futures.ProcessPoolExecutor()
q = mp.Queue()
p = executor.submit(foo, q)
p.result()
print(q.get())
bumps into the following exception coming from multiprocessing's code:
RuntimeError: Queue objects should only be shared between processes through inheritance
which I believe means it doesn't like receiving the queue as an argument, but rather expects to (not in any OOP sense) "inherit it" on the multiprocessing fork rather than getting it as an argument.
The twist is that with bare-bones multiprocessing, meaning when not using it through the facade which concurrent.futures is ― there seems to be no such limitation, as the following code seamlessly works:
import multiprocessing as mp
def foo(q):
q.put('hello')
if __name__ == '__main__':
q = mp.Queue()
p = mp.Process(target=foo, args=(q,))
p.start()
p.join()
print(q.get())
I wonder what am I missing about this ― how can I make the ProcessPoolExecutor accept the queue as an argument when using concurrent.futures the same as it does when using the ThreadPoolExecutor or multiprocessing very directly like shown right above?

Ipywidgets not interacting in "while True" loop

1)This is the code snippet that I am trying to run.The main idea is that, I want the ipywidgets to be interactive, while the data is constantly being fetched from the data source. And the data is being updated at certain interval using the while loop.The objective is to plot the principal components interactively by changing the number of principal components(PC) to be considered, using the #interact function.
Also it runs perfectly fine without while loop, that is, when we are not considering the auto-update for the dataset, with the while loop. But when I include the while loop, it doesn't handle the interactive-ness of the widgets(that is, the interaction of the number of PC).
My feeling is that "while True" loop does not let the interaction of ipywidget to happen, due to some execution issue.
2) Also I looked into threading but i am unsure of how to use the fucntion which is in functools(select_data), be called using threading.Thread.
Any sort of help would be appreciated. Thanks
def data_import_date(start_date,end_date):
end_date1=end_date.strftime('%Y-%m-%dT%H:%M:%S')
start_date=pd.Timestamp(start_date)
end_date=pd.Timestamp(end_date1)
button=widgets.Button(description='Pull Data')
button.on_click(functools.partial(select_data,rs_=[start_date,end_date]))
vbox=widgets.VBox([button])
display(vbox,out)
def select_data(b,rs_):
# clear_output()
start_date=rs_[0]
end_date1=rs_[1]
print("Data pulling started")
with out:
clear_output()
seeq_login()
[item1,item2]=query_seeq_for_data()
i=0
while True:
end_date=end_date1.strftime('%Y-%m-%dT%H:%M:%S')
print("Start & End date: ",start_date,end_date)
if i==0:
[X_data,Y_data]=pull_data(item1,item2,start_date,end_date)
else:
[X_data_live,Y_data_live]=pull_data(item1,item2,start_date,end_date)
X_data=X_data.append(X_data_live)
Y_data=Y_data.append(Y_data_live)
print("Data pulling completed.\nNow you're ready for your analysis")
clear_output()
[X_train,X_test,Y_train,Y_test]=train_test(X_data,Y_data)
[Xp_train,components,explained_variance_ratio,_,_]=apply_PCA(X_train,X_test)
plot_PC_variance(X_data,explained_variance_ratio)
plot_PC(X_data,components)
time.sleep(20)
start_date=end_date
end_date1=end_date1+datetime.timedelta(days=1)
i+=1
a=interact(data_import_date,
start_date=widgets.DatePicker(value=pd.to_datetime(start_date)),
end_date=widgets.DatePicker(value=pd.to_datetime(end_date)))
def plot_PC(X_data,components):
Np_comp=(1,len(X_data.columns),1)
#interact
def principal_components(PC1=Np_comp,PC2=Np_comp):
fig,ax=plt.subplots(1,1,figsize=(10,10))
plt.figure(5)
print(PC1, PC2)
ax.set_xlabel("Principal Component {}".format(PC1), fontsize=14)
ax.set_ylabel("Principal Component {}".format(PC2), fontsize=14)
ax.set_title("Principal components {0} & {1}".format(PC1,PC2), fontsize=(20))
ax.scatter(components[:,PC1],components[:,PC2])
A fairly rudimentary example that shows how you can run a loop whilst polling for widget changes, but it's not really my area of expertise.
Drag the slider whilst the values are printing and you should see the printed value change to reflect current slider position.
import threading
from IPython.display import display
import ipywidgets as widgets
import time
float_val = widgets.FloatSlider()
display(float_val)
def work(progress):
total = 10
for i in range(total):
time.sleep(1)
print(float_val.value)
thread = threading.Thread(target=work, args=(progress,))
thread.start()

FuncAnimation doesn't respond when after dynamically sending data to plot to move a scatter point

So I'm using FuncAnimation from matplotlib so dynamically plot some data as it arrives from a serial port (in my project is the vehicle class from dronekit, which is displayed with the green dot), what I have basically is the animation called which every loop is receiving a new vehicle class with data changed so it can be plotted, but for some reason it plots but after a couple of seconds later after the thread of the mission(which allows the "refresh" of the vehicle data it pops up and kills python (Wheel of death), here's what I get:
I've put some tracking prints inside the function that is called when the FuncAnimation starts running, looks like this:
def droneAnimation(i, vehicle, droneScatter):
time.sleep(1)
lat = [vehicle.location.global_relative_frame.lat]
lon = [vehicle.location.global_relative_frame.lon]
alt = [vehicle.location.global_relative_frame.alt]
print("Alt received: " + str(alt))
droneScatter._offsets3d = (lat,lon,alt)
print("Changed pos")
As you can see those prints are triggered the first few seconds but still crashes after a few iterations.
The FuncAnimation is called like this:
fig,droneScatter = plotLiveSimpleFacade(vehicle,w,2)
ani = FuncAnimation(fig,droneAnimation, fargs = (vehicle,droneScatter))
plt.draw()
plt.pause(0.1)
m = threading.Thread(target=MissionStart(vehicle,hmax) , name = "MISSION")
m.start()
For reference: fig is a plt.figure(),droneScatter is just a scatter point, vehicle is the vehicle class containing the data that dynamically updates and the MissionStart thread is just a thread to make the vehicle class change overtime.
I would like to mention as well that the fig is on interactive mode on, and the axes limits are set well (I saw that when you dynamically change data but don't scale the axes may have problems) also, trying different combinations of plt.draw() and plt.plot(block = False) leads me to not plotting at all or just a blank plot.
Since I have no idea of what is causing this I'll put the dronekit tag on this and the threading to see if anyone has any idea!
I've looked onto threading with matplotlib and looks like threading with this said library it's not the best as it's not thread safe, the best bet is to look at multiprocessing with python or approaching the problem in a different manner.
You can find more information at this post

Scrapy. How to yield item after spider_close call?

I want to yield an item only when the crawling is finished.
I am trying to do it via
def spider_closed(self, spider):
item = EtsyItem()
item['total_sales'] = 1111111
yield item
But it does not yield anything, though the function is called.
How do I yield an item after the scraping is over?
Depending on what you want to do, there might be a veeeery hacky solution for this.
Instead of spider_closed you may want to consider using spider_idle signal which is fired before spider_closed. One difference between idle and close is that spider_idle allows execution of requests which then may contain a callback or errback to yield the desired item.
Inside spider class:
#classmethod
def from_crawler(cls, crawler, *args, **kwargs):
# ...
crawler.signals.connect(spider.spider_idle, signal=signals.spider_idle)
return spider
# ...
def yield_item(self, response):
yield MyItem(name='myname')
def spider_idle(self, spider):
req = Request('https://fakewebsite123.xyz',
callback=lambda:None, errback=self.yield_item)
self.crawler.engine.crawl(req, spider)
However this comes with several side effects so i discourage anyone from using this in production, for example the final request which will raise a DNSLookupError. I just want to show what is possible.
Oof, I'm afraid spider_closed is used for tearing down. I suppose you can do it by attaching some custom stuff to Pipeline to post-process your items.

QThread doesn't appear to start; PyQt5, Python 2.7.9

SUMMARY
PyQt5 doesn't appear to be creating a new thread corresponding to QThread object, or I haven't established Slot/Signal linkage correctly. Please help me to isolate my problem.
I'm a relatively casual user of Python, but I've been asked to create a utility for another team that wraps some of their Python libraries (which themselves wrap C++) in a GUI. Because this utility is for another team, I can't change versions of compilers etc, or at least, not without providing a decent reason.
The utility is intended to provide an interface for debugging into some hardware that my colleagues are developing.
After examining the options, I decided to use Qt and the PyQt bindings. The steps I followed were:
Install Visual Studio 2010 SP1 (required because other team's libraries are compiled using this version of the MS compiler).
Install Python 2.7.9 (their version of Python)
Install qt-opensource-windows-x86-msvc2010-5.2.1.exe
Get source for SIP-4.18.zip and compile and install
Get source for PyQt-gpl-5.2.1.zip, compile and install
Try to build a PyQt application that wraps the other team's comms and translation libraries. Those libraries aren't asynchronous as far as I can tell, so I think that I need to separate that part of the application from the GUI.
The code that I've written produces the UI and is responsive in the sense that if I put break points in the methods that are called from the QAction objects, then those break points are appropriately triggered. My problem is that the Worker object that I create doesn't appear to move to a separate thread, (despite the call to moveToThread) because if I make the connection of type BlockingQueuedConnection instead of QueuedConnection then I get a deadlock. Breakpoints that I put on the slots in the Worker type are never triggered.
Here's the code::
import os
import sys
import time
from PyQt5.QtWidgets import QMainWindow, QTextEdit, QAction, QApplication, QStatusBar, QLabel, QWidget, QDesktopWidget, QInputDialog
from PyQt5.QtGui import QIcon
from PyQt5.QtCore import Qt, QThread, QObject, pyqtSignal, pyqtSlot
class Worker(QObject):
def __init__(self):
super(Worker, self).__init__()
self._isRunning = True
self._connectionId = ""
self._terminate = False
#pyqtSlot()
def cmd_start_running(self):
"""This slot is used to send a command to the HW asking for it to enter Running mode.
It will actually work by putting a command in a queue for the main_loop to get to
in its own serialised good time. All the other commands will work in a similar fashion
Up until such time as it is implemented, I will fake it."""
self._isRunning = True
pass
#pyqtSlot()
def cmd_stop_running(self):
"""This slot is used to send a command to the HW asking for it to enter Standby mode.
Up until such time as it is implemented, I will fake it."""
self._isRunning = False
#pyqtSlot()
def cmd_get_version(self):
"""This slot is used to send a command to the HW asking for its version string"""
pass
#pyqtSlot()
def cmd_terminate(self):
"""This slot is used to notify this object that it has to join the main thread."""
pass
#pyqtSlot()
def main_loop(self):
"""This slot is the main loop that is attached to the QThread object. It has sleep periods
that allow the messages on the other slots to be processed."""
while not self._terminate:
self.thread().sleep(1)
# While there is stuff on the wire, get it off, translate it, then
# signal it
# For the mean while, pretend that _isRunning corresponds to when
# RT streams will be
# being received from the HW.
if self._isRunning:
pass
# Search queue for commands, if any found, translate, then put on
# the wire
class DemoMainWindow(QMainWindow):
sgnl_get_version = pyqtSignal()
sgnl_start_running = pyqtSignal()
sgnl_stop_running = pyqtSignal()
sgnl_terminate = pyqtSignal()
def __init__(self):
super(DemoMainWindow, self).__init__()
self.initUI()
self._workerObject = Worker()
self._workerThread = QThread()
self._workerObject.moveToThread(self._workerThread)
self._workerThread.started.connect(self._workerObject.main_loop, type=Qt.QueuedConnection)
# I changed the following connection to type BlockingQueuedConnection,
# and got a Deadlock error
# reported, so I assume that there is already a problem before I get to
# this point.
# I understand that the default for 'type' (Qt.AutoConnection) is
# supposed to correctly infer that a QueuedConnection is required.
# I was getting desperate.
self.sgnl_get_version.connect(self._workerObject.cmd_get_version, type=Qt.QueuedConnection)
self.sgnl_start_running.connect(self._workerObject.cmd_start_running, type=Qt.QueuedConnection)
self.sgnl_stop_running.connect(self._workerObject.cmd_stop_running, type=Qt.QueuedConnection)
self.sgnl_terminate.connect(self._workerObject.cmd_terminate, type=Qt.QueuedConnection)
def initUI(self):
textEdit = QTextEdit()
self.setCentralWidget(textEdit)
lbl = QLabel(self.statusBar())
lbl.setText("HW Version: ")
self.statusBar().addPermanentWidget(lbl)
exitAction = QAction(QIcon('exit24.png'), 'Exit', self)
exitAction.setShortcut('Ctrl+Q')
exitAction.setStatusTip('Exit application')
exitAction.triggered.connect(self.close)
connectAction = QAction(QIcon('connect24.png'), 'Connect', self)
connectAction.setStatusTip('Connect to HW')
connectAction.triggered.connect(self.establishCanConnection)
enterRunningAction = QAction(QIcon('start24.png'), 'Start Running', self)
enterRunningAction.setStatusTip('Start Running')
enterRunningAction.triggered.connect(self.enterRunning)
enterStandbyAction = QAction(QIcon('stop24.png'), 'Stop Running', self)
enterStandbyAction.setStatusTip('Stop Running')
enterStandbyAction.triggered.connect(self.enterStandby)
self.statusBar()
menubar = self.menuBar()
fileMenu = menubar.addMenu('&File')
fileMenu.addAction(exitAction)
hwMenu = menubar.addMenu('&Hardware')
hwMenu.addAction(connectAction)
hwMenu.addAction(enterRunningAction)
hwMenu.addAction(enterStandbyAction)
toolbar = self.addToolBar('Exit')
toolbar.addAction(exitAction)
toolbar.addAction(connectAction)
toolbar.addAction(enterRunningAction)
toolbar.addAction(enterStandbyAction)
self.setGeometry(300, 300, 400, 350) # x, y, width, height
self.setWindowTitle('Demo Prog')
self.show()
def establishCanConnection(self):
iDlg = QInputDialog(self)
iDlg.setInputMode(QInputDialog.IntInput)
idInt, ok = iDlg.getInt(self, 'CAN ID Selection', 'HW ID:')
canID = '%s%d' % ('HW', idInt)
if ok:
self._workerThread.start()
pass
# this would be where the channel is established
def enterRunning(self):
self.sgnl_start_running.emit()
# this would be where the command to start running is sent from
def enterStandby(self):
self.sgnl_stop_running.emit()
# send the command to stop running
if __name__ == '__main__':
app = QApplication(sys.argv)
mainWindow = DemoMainWindow()
sys.exit(app.exec_())
Note that the call to start the _workerThread is in the establishCanConnection method, but that shouldn't be a problem, should it?
I used the procmon utility to check if more threads are created if establishCanConnection is run, and it appears that there are more threads, but I found it hard to relate which thread (if any of them) related to the QThread object.
Don't use BlockingQueuedConnection unless you really need it. If you don't know whether you need it or not, then you don't need it.
Cross-thread signals are queued in the event-loop of the receiving thread. If that thread is running code that blocks, it won't be able to process any events. Thus, if you send a signal with BlockingQueuedConnection to a thread that is blocked, you'll get a deadlock.
Your example uses a worker object that runs a blocking while loop, so it is subject to the deadlock problem outlined above. If you want to send signals to a thread that is blocked, you will need to arrange for the blocking code to periodically allow the thread to process its events, like this:
while not self._terminate:
self.thread().sleep(1)
QApplication.processEvents()
PS:
If you want to check that the worker is running in a different thread, you can print the return value of QThread.currentThread() or QThread.currentThreadId() (these functions are static, so you don't need an instance of QThread to call them).