Related
I really like the simplicity with how ipywidgets.interactive works with pandas dataframe but I am having trouble getting data when a point in a scatter plot is selected.
I have looked at some examples that use matplotlib.widgets etc. but none that use it with interactive in Jupyter. It looks like this technique would be described here but it comes up just short:
http://minrk-ipywidgets.readthedocs.io/en/latest/examples/Using%20Interact.html
Here is an ipynb of what I am trying to accomplish:
from ipywidgets import interactive
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib.widgets import Button
from matplotlib.text import Annotation
from io import StringIO
data_ssv = """tone_amp_0 tone_freq_0 SNR
75.303 628.0 68.374
84.902 8000.0 61.292
92.856 288.0 70.545
70.000 2093.0 35.036
76.511 6834.0 66.952 """
data = pd.read_table(StringIO(data_ssv), sep="\s+", header=0)
col_names=list(data.columns.values)
plottable_col=( ['tone_amp_0', 'tone_freq_0', 'SNR'] )
def annotate(axis, text, x, y):
text_annotation = Annotation(text, xy=(x, y), xycoords='data')
axis.add_artist(text_annotation)
def onpick(event):
ind = event.ind
label_pos_x = event.mouseevent.xdata
label_pos_y = event.mouseevent.ydata
offset = 0 # just in case two dots are very close, this offset will help the labels not appear one on top of each other
for i in ind: # if the dots are to close one to another, a list of dots clicked is returned by the matplotlib library
label = "gen_labels" # generated_labels[i]
print( "index", i, label ) # step 4: log it for debugging purposes
ax=plt.gca()
annotate(ax,label,label_pos_x + offset,label_pos_y + offset)
ax.figure.canvas.draw_idle()
offset += 0.01 # alter the offset just in case there are more than one dots affected by the click
def update_plot(X='tone_amp_0', Y='tone_frq_0', Z='SNR'):
plt.scatter( data.loc[:, [X]],data.loc[:, [Y]], marker='.', edgecolors='none', c=data.loc[:,[Z]], picker=True, cmap='RdYlGn' )
plt.title(X+' vs '+Y); plt.xlabel(X); plt.ylabel(Y); plt.colorbar().set_label(Z, labelpad=+1)
plt.grid(); plt.show()
plt.gcf().canvas.mpl_connect('pick_event', onpick)
interactive(update_plot, X=plottable_col, Y=plottable_col, Z=plottable_col)
When I select a data point nothing is happening. Not sure how to debug this or understand what I am doing wrong. Can someone point out what I am doing wrong here?
Try put a semicolon at the end of plt.gcf().canvas.mpl_connect('pick_event', onpick).
I am using a wrapper of PyQt (pyqtgraph) to build a GUI application.
I wish to embed a Seaborn plot within it using the MatplotlibWidget. However, my problem is that the Seaborn wrapper method such as FacetGrid do not accept an external figure handle. Moreover, when I try to update the MatplotlibWidget object underlying figure (.fig) with the figure produced by the FacetGrid it doesn't work (no plot after draw). Any suggestion for a workaround?
Seaborn's Facetgrid provides a convenience function to quickly connect pandas dataframes to the matplotlib pyplot interface.
However in GUI applications you rarely want to use pyplot, but rather the matplotlib API.
The problem you are facing here is that Facetgrid already creates its own matplotlib.figure.Figure object (Facetgrid.fig). Also, the MatplotlibWidget
creates its own figure, so you end up with two figures.
Now, let's step back a bit:
In principle it is possible to use a seaborn Facetgrid plot in PyQt, by first creating the plot and then providing the resulting figure to the figure canvas (matplotlib.backends.backend_qt4agg.FigureCanvasQTAgg). The following is an example of how to do that.
from PyQt4 import QtGui, QtCore
from matplotlib.backends.backend_qt4agg import FigureCanvasQTAgg as FigureCanvas
import sys
import seaborn as sns
import matplotlib.pyplot as plt
tips = sns.load_dataset("tips")
def seabornplot():
g = sns.FacetGrid(tips, col="sex", hue="time", palette="Set1",
hue_order=["Dinner", "Lunch"])
g.map(plt.scatter, "total_bill", "tip", edgecolor="w")
return g.fig
class MainWindow(QtGui.QMainWindow):
send_fig = QtCore.pyqtSignal(str)
def __init__(self):
super(MainWindow, self).__init__()
self.main_widget = QtGui.QWidget(self)
self.fig = seabornplot()
self.canvas = FigureCanvas(self.fig)
self.canvas.setSizePolicy(QtGui.QSizePolicy.Expanding,
QtGui.QSizePolicy.Expanding)
self.canvas.updateGeometry()
self.button = QtGui.QPushButton("Button")
self.label = QtGui.QLabel("A plot:")
self.layout = QtGui.QGridLayout(self.main_widget)
self.layout.addWidget(self.button)
self.layout.addWidget(self.label)
self.layout.addWidget(self.canvas)
self.setCentralWidget(self.main_widget)
self.show()
if __name__ == '__main__':
app = QtGui.QApplication(sys.argv)
win = MainWindow()
sys.exit(app.exec_())
While this works fine, it is a bit questionable, if it's useful at all. Creating a plot inside a GUI in most cases has the purpose of beeing updated depending on user interactions. In the example case from above, this is pretty inefficient, as it would require to create a new figure instance, create a new canvas with this figure and replace the old canvas instance with the new one, adding it to the layout.
Note that this problematics is specific to those plotting functions in seaborn, which work on a figure level, like lmplot, factorplot, jointplot, FacetGrid and possibly others.
Other functions like regplot, boxplot, kdeplot work on an axes level and accept a matplotlib axes object as argument (sns.regplot(x, y, ax=ax1)).
A possibile solution is to first create the subplot axes and later plot to those axes, for example using the pandas plotting functionality.
df.plot(kind="scatter", x=..., y=..., ax=...)
where ax should be set to the previously created axes.
This allows to update the plot within the GUI. See the example below. Of course normal matplotlib plotting (ax.plot(x,y)) or the use of the seaborn axes level function discussed above work equally well.
from PyQt4 import QtGui, QtCore
from matplotlib.backends.backend_qt4agg import FigureCanvasQTAgg as FigureCanvas
from matplotlib.figure import Figure
import sys
import seaborn as sns
tips = sns.load_dataset("tips")
class MainWindow(QtGui.QMainWindow):
send_fig = QtCore.pyqtSignal(str)
def __init__(self):
super(MainWindow, self).__init__()
self.main_widget = QtGui.QWidget(self)
self.fig = Figure()
self.ax1 = self.fig.add_subplot(121)
self.ax2 = self.fig.add_subplot(122, sharex=self.ax1, sharey=self.ax1)
self.axes=[self.ax1, self.ax2]
self.canvas = FigureCanvas(self.fig)
self.canvas.setSizePolicy(QtGui.QSizePolicy.Expanding,
QtGui.QSizePolicy.Expanding)
self.canvas.updateGeometry()
self.dropdown1 = QtGui.QComboBox()
self.dropdown1.addItems(["sex", "time", "smoker"])
self.dropdown2 = QtGui.QComboBox()
self.dropdown2.addItems(["sex", "time", "smoker", "day"])
self.dropdown2.setCurrentIndex(2)
self.dropdown1.currentIndexChanged.connect(self.update)
self.dropdown2.currentIndexChanged.connect(self.update)
self.label = QtGui.QLabel("A plot:")
self.layout = QtGui.QGridLayout(self.main_widget)
self.layout.addWidget(QtGui.QLabel("Select category for subplots"))
self.layout.addWidget(self.dropdown1)
self.layout.addWidget(QtGui.QLabel("Select category for markers"))
self.layout.addWidget(self.dropdown2)
self.layout.addWidget(self.canvas)
self.setCentralWidget(self.main_widget)
self.show()
self.update()
def update(self):
colors=["b", "r", "g", "y", "k", "c"]
self.ax1.clear()
self.ax2.clear()
cat1 = self.dropdown1.currentText()
cat2 = self.dropdown2.currentText()
print cat1, cat2
for i, value in enumerate(tips[cat1].unique().get_values()):
print "value ", value
df = tips.loc[tips[cat1] == value]
self.axes[i].set_title(cat1 + ": " + value)
for j, value2 in enumerate(df[cat2].unique().get_values()):
print "value2 ", value2
df.loc[ tips[cat2] == value2 ].plot(kind="scatter", x="total_bill", y="tip",
ax=self.axes[i], c=colors[j], label=value2)
self.axes[i].legend()
self.fig.canvas.draw_idle()
if __name__ == '__main__':
app = QtGui.QApplication(sys.argv)
win = MainWindow()
sys.exit(app.exec_())
A final word about pyqtgraph: I wouldn't call pyqtgraph a wrapper for PyQt but more an extention. Although pyqtgraph ships with its own Qt (which makes it portable and work out of the box), it is also a package one can use from within PyQt. You can therefore add a GraphicsLayoutWidget to a PyQt layout simply by
self.pgcanvas = pg.GraphicsLayoutWidget()
self.layout().addWidget(self.pgcanvas)
The same holds for a MatplotlibWidget (mw = pg.MatplotlibWidget()). While you can use this kind of widget, it's merely a convenience wrapper, since all it's doing is finding the correct matplotlib imports and creating a Figure and a FigureCanvas instance. Unless you are using other pyqtgraph functionality, importing the complete pyqtgraph package just to save 5 lines of code seems a bit overkill to me.
Here is exact copy of the accepted answer but using PYQT5:
from PyQt5 import QtCore, QtGui, QtWidgets
from matplotlib.backends.backend_qt5agg import FigureCanvasQTAgg as FigureCanvas
from matplotlib.figure import Figure
import sys
import seaborn as sns
tips = sns.load_dataset("tips")
class MainWindow(QtWidgets.QMainWindow):
send_fig = QtCore.pyqtSignal(str)
def __init__(self):
super(MainWindow, self).__init__()
self.main_widget = QtWidgets.QWidget(self)
self.fig = Figure()
self.ax1 = self.fig.add_subplot(121)
self.ax2 = self.fig.add_subplot(122, sharex=self.ax1, sharey=self.ax1)
self.axes=[self.ax1, self.ax2]
self.canvas = FigureCanvas(self.fig)
self.canvas.setSizePolicy(QtWidgets.QSizePolicy.Expanding,
QtWidgets.QSizePolicy.Expanding)
self.canvas.updateGeometry()
self.dropdown1 = QtWidgets.QComboBox()
self.dropdown1.addItems(["sex", "time", "smoker"])
self.dropdown2 = QtWidgets.QComboBox()
self.dropdown2.addItems(["sex", "time", "smoker", "day"])
self.dropdown2.setCurrentIndex(2)
self.dropdown1.currentIndexChanged.connect(self.update)
self.dropdown2.currentIndexChanged.connect(self.update)
self.label = QtWidgets.QLabel("A plot:")
self.layout = QtWidgets.QGridLayout(self.main_widget)
self.layout.addWidget(QtWidgets.QLabel("Select category for subplots"))
self.layout.addWidget(self.dropdown1)
self.layout.addWidget(QtWidgets.QLabel("Select category for markers"))
self.layout.addWidget(self.dropdown2)
self.layout.addWidget(self.canvas)
self.setCentralWidget(self.main_widget)
self.show()
self.update()
def update(self):
colors=["b", "r", "g", "y", "k", "c"]
self.ax1.clear()
self.ax2.clear()
cat1 = self.dropdown1.currentText()
cat2 = self.dropdown2.currentText()
print (cat1, cat2)
for i, value in enumerate(tips[cat1].unique().get_values()):
print ("value ", value)
df = tips.loc[tips[cat1] == value]
self.axes[i].set_title(cat1 + ": " + value)
for j, value2 in enumerate(df[cat2].unique().get_values()):
print ("value2 ", value2)
df.loc[ tips[cat2] == value2 ].plot(kind="scatter", x="total_bill", y="tip",
ax=self.axes[i], c=colors[j], label=value2)
self.axes[i].legend()
self.fig.canvas.draw_idle()
if __name__ == '__main__':
import sys
app = QtWidgets.QApplication(sys.argv)
ex = MainWindow()
sys.exit(app.exec_())
While any matplotlib plots can be embedded in pyqt5 the same way, it's important to note that the UI could get slow as the sizeof the dataset grows. But I found such approaches handy to parse and plot log files by employing regex functionalities.
I understand that there have been one or two other questions posted that are related but not exactly what I need. I'm building this gui that activates a module by clicking a button. This python module that gets activated by pushing the button generates heatmaps from multiple pandas dataframes and saves those images, which in turn is then saved into an xlsx using pandas ExcelWriter.
I've tried to implement QThread, as other stackoverflow examples tried to explain similar problems but I continue getting this error: "It is not safe to use pixmaps outside the GUI thread". I understand that technically I'm not creating the heatmap inside the MAIN gui thread but I thought with QThread that I am still inside "a" gui thread. These dataframes that the heatmaps are based off of can be of a large size at times and I am somewhat grasping the concept of sending a signal to the main gui thread when a heatmap is to be created and have the heatmap function inside the main gui class...but I fear that will be troublesome later in passing so much data around..this is more like pipelining than threading. I just want this working thread to create these images and save them and then take those saved files and save them into an xlsx without interrupting the main gui..
(NOTE: This is a simplified version, in the real program there will be several of these threads created almost simultaneously and inside each thread several heatmaps will be created)
---main.py---
import sys
from MAIN_GUI import *
from PyQt4 import QtGui, QtCore
from excel_dummy import *
if __name__=="__main__":
app = QtGui.QApplication(sys.argv)
class MAIN_GUI(QtGui.QMainWindow):
def __init__(self):
super(MAIN_GUI, self).__init__()
self.uiM = Ui_MainWindow()
self.uiM.setupUi(self)
self.connect(self.uiM.updateALL_Button,QtCore.SIGNAL('clicked()'),self.newThread)
def newThread(self):
Excelify = excelify()
Excelify.start()
self.connect(Excelify,QtCore.SIGNAL('donethread(QString)'),(self.done))
def done(self):
print('done')
main_gui = MAIN_GUI()
main_gui.show()
main_gui.raise_()
sys.exit(app.exec_())
---excel_dummy.py---
import os, pandas as pd
from pandas import ExcelWriter
import numpy as np
import seaborn.matrix as sm
from PyQt4 import QtCore
from PyQt4.QtCore import QThread
from matplotlib.backends.backend_agg import FigureCanvas
from matplotlib.figure import Figure
import time
class excelify(QThread):
def __init__(self):
QThread.__init__(self)
def run(self):
path = 'home/desktop/produced_files'
with ExcelWriter(path + '/final.xlsx', engine='xlsxwriter') as writer:
workbook = writer.book
worksheet = workbook.add_worksheet()
heatit = self.heatmap()
worksheet.insert_image('C3',path + '/' + 'heat.jpg')
worksheet.write(2, 2, 'just write something')
writer.save()
print('file size: %s "%s"' % (os.stat(path).st_size, path))
time.slee(0.3)
self.emit(QtCore.SIGNAL('donethread(QString)'),'')
def heatmap(self):
df = pd.DataFrame(np.array([[1,22222,33333],[2,44444,55555],[3,44444,22222],[4,55555,33333]]),columns=['hour','in','out'])
dfu = pd.DataFrame(df.groupby([df.in,df.hour]).size())
dfu.reset_index(inplace=True)
dfu.rename(columns={'0':'Count'})
dfu.columns=['in','hour','Count']
dfu_2 = dfu.copy()
mask=0
fig = Figure()
ax = fig.add_subplot(1,1,1)
canvas = FigureCanvas(fig)
df_heatmap = dfu_2.pivot('in','hour','Count').fillna(0)
sm.heatmap(df_heatmap,ax=ax,square=True,annot=False,mask=mask)
fig.savefig(path + '/' + heat.jpg')
---MAIN_GUI.py---
from PyQt4 import QtCore,QtGui
try:
_fromUtf8 = QtCore.QString.fromUtf8
except AttributeError:
def _fromUtf8(s):
return s
try:
_encoding = QtGui.QApplication.unicodeUTF8
def _translate(context, text, disambig):
return QtGui.QApplication.translate(context, text, disambig, _encoding)
except AttributeError:
def _translate(context, text, disambig):
return QtGui.QApplication.translate(context, text, disambig)
class Ui_MainWindow(object):
def setupUi(self, MainWindow):
MainWindow.setObjectName(_fromUtf8("MainWindow"))
MainWindow.resize(320,201)
self.centralwidget = QtGui.QWidget(MainWindow)
self.centralwidget.setObjectName(_fromUtf8("centralwidget"))
self.updateALL_Button = QtGui.QPushButton(self.centralwidget)
self.updateALL_Button.setGeometry(QtCore.QRect(40,110,161,27))
self.updateALL_Button.setFocusPolicy(QtCore.Qt.NoFocus)
self.updateALL_Button.setObjectName(_fromUtf8("Options_updateALL_Button"))
MainWindow.setCentralWidget(self.centralwidget)
self.menubar = QtGui.QMenuBar(MainWindow)
self.menubar.setGeometry(QtCore.QRect(0, 0, 320, 24))
self.menubar.setObjectName(_fromUtf8("menubar"))
MainWindow.setMenuBar(self.menubar)
self.statusbar = QtGui.QStatusBar(MainWindow)
self.statusbar.setObjectName(_fromUtf8("statusbar"))
MainWindow.setStatusBar(self.statusbar)
self.retranslateUi(MainWindow)
QtCore.QMetaObject.connectSlotsByName(MainWindow)
def retranslateUi(self,MainWindow):
MainWindow.setWindowTitle(_translate("MainWindow", "MainWindow", None))
self.updateALL_Button.setText(_translate("MainWindow", "updateALL", None))
Even though you are explicitely using the Agg backend to generate your figure, it looks like Seaborn is still using the default backend on your system, which is most likely Qt4Agg, an interactive backend. We want Seaborn to use a non-interactive backend instead to avoid any error (see matplotlib documentation for more details about backends). To do so, tell Matplotlib in your imports to use the Agg backend and import Seaborn after Matplotlib.
You will also need to save your figure as a png, since jpg is not supported by the Agg backend. Unless you have some specific reasons for using jpg, png is usually a better format for graphs.
Finally, you could use a memory buffer instead of saving your images to a temporary file before saving them in an Excel Workbook. I haven't tested it, but it will probably be faster if you are working with large files.
Below is a MWE I've written which includes the aformentioned points and which does not give any error on my system in Python3.4:
import pandas as pd
import time
from pandas import ExcelWriter
import numpy as np
from PyQt4 import QtCore, QtGui
import matplotlib as mpl
mpl.use('Agg')
from matplotlib.backends.backend_agg import FigureCanvas
import seaborn.matrix as sm
try: # Python 2 (not tested)
from cStringIO import StringIO as BytesIO
except ImportError: # Python 3
from io import BytesIO
class MAIN_GUI(QtGui.QWidget):
def __init__(self):
super(MAIN_GUI, self).__init__()
self.worker = Excelify()
btn = QtGui.QPushButton('Run')
disp = QtGui.QLabel()
self.setLayout(QtGui.QGridLayout())
self.layout().addWidget(btn, 0, 0)
self.layout().addWidget(disp, 2, 0)
self.layout().setRowStretch(1, 100)
btn.clicked.connect(self.worker.start)
self.worker.figSaved.connect(disp.setText)
class Excelify(QtCore.QThread):
figSaved = QtCore.pyqtSignal(str)
def run(self):
self.figSaved.emit('Saving figure to Workbook.')
t1 = time.clock()
image_data = self.heatmap()
with ExcelWriter('final.xlsx', engine='xlsxwriter') as writer:
wb = writer.book
ws = wb.add_worksheet()
ws.insert_image('C3', 'heat.png', {'image_data': image_data})
writer.save()
t2 = time.clock()
self.figSaved.emit('Done in %f sec.' % (t2-t1))
def heatmap(self):
df = pd.DataFrame(np.array([[1, 22222, 33333], [2, 44444, 55555],
[3, 44444, 22222], [4, 55555, 33333]]),
columns=['hour', 'in', 'out'])
dfu = pd.DataFrame(df.groupby([df.out, df.hour]).size())
dfu.reset_index(inplace=True)
dfu.rename(columns={'0': 'Count'})
dfu.columns = ['in', 'hour', 'Count']
fig = mpl.figure.Figure()
fig.set_canvas(FigureCanvas(fig))
ax = fig.add_subplot(111)
df_heatmap = dfu.pivot('in', 'hour', 'Count').fillna(0)
sm.heatmap(df_heatmap, ax=ax, square=True, annot=False, mask=0)
buf= BytesIO()
fig.savefig(buf, format='png')
return(buf)
if __name__ == '__main__':
import sys
app = QtGui.QApplication(sys.argv)
w = MAIN_GUI()
w.show()
w.setFixedSize(200, 100)
sys.exit(app.exec_())
For years, I've been struggling to get efficient live plotting in matplotlib, and to this day I remain unsatisfied.
I want a redraw_figure function that updates the figure "live" (as the code runs), and will display the latest plots if I stop at a breakpoint.
Here is some demo code:
import time
from matplotlib import pyplot as plt
import numpy as np
def live_update_demo():
plt.subplot(2, 1, 1)
h1 = plt.imshow(np.random.randn(30, 30))
redraw_figure()
plt.subplot(2, 1, 2)
h2, = plt.plot(np.random.randn(50))
redraw_figure()
t_start = time.time()
for i in xrange(1000):
h1.set_data(np.random.randn(30, 30))
redraw_figure()
h2.set_ydata(np.random.randn(50))
redraw_figure()
print 'Mean Frame Rate: %.3gFPS' % ((i+1) / (time.time() - t_start))
def redraw_figure():
plt.draw()
plt.pause(0.00001)
live_update_demo()
Plots should update live when the code is run, and we should see the latest data when stopping at any breakpoint after redraw_figure(). The question is how to best implement redraw_figure()
In the implementation above (plt.draw(); plt.pause(0.00001)), it works, but is very slow (~3.7FPS)
I can implement it as:
def redraw_figure():
plt.gcf().canvas.flush_events()
plt.show(block=False)
And it runs faster (~11FPS), but plots are not up-to date when you stop at breakpoints (eg if I put a breakpoint on the t_start = ... line, the second plot does not appear).
Strangely enough, what does actually work is calling the show twice:
def redraw_figure():
plt.gcf().canvas.flush_events()
plt.show(block=False)
plt.show(block=False)
Which gives ~11FPS and does keep plots up-to-data if your break on any line.
Now I've heard it said that the "block" keyword is deprecated. And calling the same function twice seems like a weird, probably-non-portable hack anyway.
So what can I put in this function that will plot at a reasonable frame rate, isn't a giant kludge, and preferably will work across backends and systems?
Some notes:
I'm on OSX, and using TkAgg backend, but solutions on any backend/system are welcome
Interactive mode "On" will not work, because it does not update live. It just updates when in the Python console when the interpreter waits for user input.
A blog suggested the implementation:
def redraw_figure():
fig = plt.gcf()
fig.canvas.draw()
fig.canvas.flush_events()
But at least on my system, that does not redraw the plots at all.
So, if anybody has an answer, you would directly make me and thousands of others very happy. Their happiness would probably trickle through to their friends and relatives, and their friends and relatives, and so on, so that you could potentially improve the lives of billions.
Conclusions
ImportanceOfBeingErnest shows how you can use blit for faster plotting, but it's not as simple as putting something different in the redraw_figure function (you need to keep track of what things to redraw).
First of all, the code that is posted in the question runs with 7 fps on my machine, with QT4Agg as backend.
Now, as has been suggested in many posts, like here or here, using blit might be an option. Although this article mentions that blit causes strong memory leakage, I could not observe that.
I have modified your code a bit and compared the frame rate with and without the use of blit. The code below gives
28 fps when run without blit
175 fps with blit
Code:
import time
from matplotlib import pyplot as plt
import numpy as np
def live_update_demo(blit = False):
x = np.linspace(0,50., num=100)
X,Y = np.meshgrid(x,x)
fig = plt.figure()
ax1 = fig.add_subplot(2, 1, 1)
ax2 = fig.add_subplot(2, 1, 2)
img = ax1.imshow(X, vmin=-1, vmax=1, interpolation="None", cmap="RdBu")
line, = ax2.plot([], lw=3)
text = ax2.text(0.8,0.5, "")
ax2.set_xlim(x.min(), x.max())
ax2.set_ylim([-1.1, 1.1])
fig.canvas.draw() # note that the first draw comes before setting data
if blit:
# cache the background
axbackground = fig.canvas.copy_from_bbox(ax1.bbox)
ax2background = fig.canvas.copy_from_bbox(ax2.bbox)
plt.show(block=False)
t_start = time.time()
k=0.
for i in np.arange(1000):
img.set_data(np.sin(X/3.+k)*np.cos(Y/3.+k))
line.set_data(x, np.sin(x/3.+k))
tx = 'Mean Frame Rate:\n {fps:.3f}FPS'.format(fps= ((i+1) / (time.time() - t_start)) )
text.set_text(tx)
#print tx
k+=0.11
if blit:
# restore background
fig.canvas.restore_region(axbackground)
fig.canvas.restore_region(ax2background)
# redraw just the points
ax1.draw_artist(img)
ax2.draw_artist(line)
ax2.draw_artist(text)
# fill in the axes rectangle
fig.canvas.blit(ax1.bbox)
fig.canvas.blit(ax2.bbox)
# in this post http://bastibe.de/2013-05-30-speeding-up-matplotlib.html
# it is mentionned that blit causes strong memory leakage.
# however, I did not observe that.
else:
# redraw everything
fig.canvas.draw()
fig.canvas.flush_events()
#alternatively you could use
#plt.pause(0.000000000001)
# however plt.pause calls canvas.draw(), as can be read here:
#http://bastibe.de/2013-05-30-speeding-up-matplotlib.html
live_update_demo(True) # 175 fps
#live_update_demo(False) # 28 fps
Update:
For faster plotting, one may consider using pyqtgraph.
As the pyqtgraph documentation puts it: "For plotting, pyqtgraph is not nearly as complete/mature as matplotlib, but runs much faster."
I ported the above example to pyqtgraph. And although it looks kind of ugly, it runs with 250 fps on my machine.
Summing that up,
matplotlib (without blitting): 28 fps
matplotlib (with blitting): 175 fps
pyqtgraph : 250 fps
pyqtgraph code:
import sys
import time
from pyqtgraph.Qt import QtCore, QtGui
import numpy as np
import pyqtgraph as pg
class App(QtGui.QMainWindow):
def __init__(self, parent=None):
super(App, self).__init__(parent)
#### Create Gui Elements ###########
self.mainbox = QtGui.QWidget()
self.setCentralWidget(self.mainbox)
self.mainbox.setLayout(QtGui.QVBoxLayout())
self.canvas = pg.GraphicsLayoutWidget()
self.mainbox.layout().addWidget(self.canvas)
self.label = QtGui.QLabel()
self.mainbox.layout().addWidget(self.label)
self.view = self.canvas.addViewBox()
self.view.setAspectLocked(True)
self.view.setRange(QtCore.QRectF(0,0, 100, 100))
# image plot
self.img = pg.ImageItem(border='w')
self.view.addItem(self.img)
self.canvas.nextRow()
# line plot
self.otherplot = self.canvas.addPlot()
self.h2 = self.otherplot.plot(pen='y')
#### Set Data #####################
self.x = np.linspace(0,50., num=100)
self.X,self.Y = np.meshgrid(self.x,self.x)
self.counter = 0
self.fps = 0.
self.lastupdate = time.time()
#### Start #####################
self._update()
def _update(self):
self.data = np.sin(self.X/3.+self.counter/9.)*np.cos(self.Y/3.+self.counter/9.)
self.ydata = np.sin(self.x/3.+ self.counter/9.)
self.img.setImage(self.data)
self.h2.setData(self.ydata)
now = time.time()
dt = (now-self.lastupdate)
if dt <= 0:
dt = 0.000000000001
fps2 = 1.0 / dt
self.lastupdate = now
self.fps = self.fps * 0.9 + fps2 * 0.1
tx = 'Mean Frame Rate: {fps:.3f} FPS'.format(fps=self.fps )
self.label.setText(tx)
QtCore.QTimer.singleShot(1, self._update)
self.counter += 1
if __name__ == '__main__':
app = QtGui.QApplication(sys.argv)
thisapp = App()
thisapp.show()
sys.exit(app.exec_())
Here's one way to do live plotting: get the plot as an image array then draw the image to a multithreaded screen.
Example using a pyformulas screen (~30 FPS):
import pyformulas as pf
import matplotlib.pyplot as plt
import numpy as np
import time
fig = plt.figure()
screen = pf.screen(title='Plot')
start = time.time()
for i in range(10000):
t = time.time() - start
x = np.linspace(t-3, t, 100)
y = np.sin(2*np.pi*x) + np.sin(3*np.pi*x)
plt.xlim(t-3,t)
plt.ylim(-3,3)
plt.plot(x, y, c='black')
# If we haven't already shown or saved the plot, then we need to draw the figure first...
fig.canvas.draw()
image = np.fromstring(fig.canvas.tostring_rgb(), dtype=np.uint8, sep='')
image = image.reshape(fig.canvas.get_width_height()[::-1] + (3,))
screen.update(image)
#screen.close()
Disclaimer: I'm the maintainer of pyformulas
I want to speed up matplotlib.savefig() for many figures by multiprocessing module, and trying to benchmark the performance between parallel and sequence.
Below is the codes:
# -*- coding: utf-8 -*-
"""
Compare the time of matplotlib savefig() in parallel and sequence
"""
import numpy as np
import matplotlib.pyplot as plt
import multiprocessing
import time
def gen_fig_list(n):
''' generate a list to contain n demo scatter figure object '''
plt.ioff()
fig_list = []
for i in range(n):
plt.figure();
dt = np.random.randn(5, 4);
fig = plt.scatter(dt[:,0], dt[:,1], s=abs(dt[:,2]*1000), c=abs(dt[:,3]*100)).get_figure()
fig.FM_figname = "img"+str(i)
fig_list.append(fig)
plt.ion()
return fig_list
def savefig_worker(fig, img_type, folder):
file_name = folder+"\\"+fig.FM_figname+"."+img_type
fig.savefig(file_name, format=img_type, dpi=fig.dpi)
return file_name
def parallel_savefig(fig_list, folder):
proclist = []
for fig in fig_list:
print fig.FM_figname,
p = multiprocessing.Process(target=savefig_worker, args=(fig, 'png', folder)) # cause error
proclist.append(p)
p.start()
for i in proclist:
i.join()
if __name__ == '__main__':
folder_1, folder_2 = 'Z:\\A1', 'Z:\\A2'
fig_list = gen_fig_list(10)
t1 = time.time()
parallel_savefig(fig_list,folder_1)
t2 = time.time()
print '\nMulprocessing time : %0.3f'%((t2-t1))
t3 = time.time()
for fig in fig_list:
savefig_worker(fig, 'png', folder_2)
t4 = time.time()
print 'Non_Mulprocessing time: %0.3f'%((t4-t3))
And I meet problem "This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information." error caused by p = multiprocessing.Process(target=savefig_worker, args=(fig, 'png', folder)) .
Why ? And how to solve it ?
(Windows XP + Python: 2.6.1 + Numpy: 1.6.2 + Matplotlib: 1.2.0)
EDIT: (add error msg on python 2.7.3)
When run on IDLE of python 2.7.3, it gives below error msg:
>>>
img0
Traceback (most recent call last):
File "C:\Documents and Settings\Administrator\desktop\mulsavefig_pilot.py", line 61, in <module>
proc.start()
File "d:\Python27\lib\multiprocessing\process.py", line 130, in start
File "d:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "d:\Python27\lib\pickle.py", line 748, in save_global
(obj, module, name))
PicklingError: Can't pickle <function notify_axes_change at 0x029F5030>: it's not found as matplotlib.backends.backend_qt4.notify_axes_change
EDIT: (My solution demo)
inspired by Matplotlib: simultaneous plotting in multiple threads
# -*- coding: utf-8 -*-
"""
Compare the time of matplotlib savefig() in parallel and sequence
"""
import numpy as np
import matplotlib.pyplot as plt
import multiprocessing
import time
def gen_data(fig_qty, bubble_qty):
''' generate data for fig drawing '''
dt = np.random.randn(fig_qty, bubble_qty, 4)
return dt
def parallel_savefig(draw_data, folder):
''' prepare data and pass to worker '''
pool = multiprocessing.Pool()
fig_qty = len(draw_data)
fig_para = zip(range(fig_qty), draw_data, [folder]*fig_qty)
pool.map(fig_draw_save_worker, fig_para)
return None
def fig_draw_save_worker(args):
seq, dt, folder = args
plt.figure()
fig = plt.scatter(dt[:,0], dt[:,1], s=abs(dt[:,2]*1000), c=abs(dt[:,3]*100), alpha=0.7).get_figure()
plt.title('Plot of a scatter of %i' % seq)
fig.savefig(folder+"\\"+'fig_%02i.png' % seq)
plt.close()
return None
if __name__ == '__main__':
folder_1, folder_2 = 'A1', 'A2'
fig_qty, bubble_qty = 500, 100
draw_data = gen_data(fig_qty, bubble_qty)
print 'Mulprocessing ... ',
t1 = time.time()
parallel_savefig(draw_data, folder_1)
t2 = time.time()
print 'Time : %0.3f'%((t2-t1))
print 'Non_Mulprocessing .. ',
t3 = time.time()
for para in zip(range(fig_qty), draw_data, [folder_2]*fig_qty):
fig_draw_save_worker(para)
t4 = time.time()
print 'Time : %0.3f'%((t4-t3))
print 'Speed Up: %0.1fx'%(((t4-t3)/(t2-t1)))
You can try to move all of the matplotlib code(including the import) to a function.
Make sure you don't have a import matplotlib or import matplotlib.pyplot as plt at the top of your code.
create a function that does all the matplotlib including the import.
Example:
import numpy as np
from multiprocessing import pool
def graphing_function(graph_data):
import matplotlib.pyplot as plt
plt.figure()
plt.hist(graph_data.data)
plt.savefig(graph_data.filename)
plt.close()
return
pool = Pool(4)
pool.map(graphing_function, data_list)
It is not really a bug, per-say, more of a limitation.
The explanation is in the last line of your error mesage:
PicklingError: Can't pickle <function notify_axes_change at 0x029F5030>: it's not found as matplotlib.backends.backend_qt4.notify_axes_change
It is telling you that elements of the figure objects can not be pickled, which is how MultiProcess passes data between the processes. The objects are pickled in the main processes, shipped as pickles, and then re-constructed on the other side. Even if you fixed this exact issue (maybe by using a different backend, or stripping off the offending function (which might break things in other ways)) I am pretty sure there are core parts of Figure, Axes, or Canvas objects that can not be pickled.
As #bigbug point to, an example of how to get around this limitation, Matplotlib: simultaneous plotting in multiple threads. The basic idea is that you push your entire plotting routine off to the sub-process so you only push numpy arrays an maybe some configuration information across the process boundry.