1)This is the code snippet that I am trying to run.The main idea is that, I want the ipywidgets to be interactive, while the data is constantly being fetched from the data source. And the data is being updated at certain interval using the while loop.The objective is to plot the principal components interactively by changing the number of principal components(PC) to be considered, using the #interact function.
Also it runs perfectly fine without while loop, that is, when we are not considering the auto-update for the dataset, with the while loop. But when I include the while loop, it doesn't handle the interactive-ness of the widgets(that is, the interaction of the number of PC).
My feeling is that "while True" loop does not let the interaction of ipywidget to happen, due to some execution issue.
2) Also I looked into threading but i am unsure of how to use the fucntion which is in functools(select_data), be called using threading.Thread.
Any sort of help would be appreciated. Thanks
def data_import_date(start_date,end_date):
end_date1=end_date.strftime('%Y-%m-%dT%H:%M:%S')
start_date=pd.Timestamp(start_date)
end_date=pd.Timestamp(end_date1)
button=widgets.Button(description='Pull Data')
button.on_click(functools.partial(select_data,rs_=[start_date,end_date]))
vbox=widgets.VBox([button])
display(vbox,out)
def select_data(b,rs_):
# clear_output()
start_date=rs_[0]
end_date1=rs_[1]
print("Data pulling started")
with out:
clear_output()
seeq_login()
[item1,item2]=query_seeq_for_data()
i=0
while True:
end_date=end_date1.strftime('%Y-%m-%dT%H:%M:%S')
print("Start & End date: ",start_date,end_date)
if i==0:
[X_data,Y_data]=pull_data(item1,item2,start_date,end_date)
else:
[X_data_live,Y_data_live]=pull_data(item1,item2,start_date,end_date)
X_data=X_data.append(X_data_live)
Y_data=Y_data.append(Y_data_live)
print("Data pulling completed.\nNow you're ready for your analysis")
clear_output()
[X_train,X_test,Y_train,Y_test]=train_test(X_data,Y_data)
[Xp_train,components,explained_variance_ratio,_,_]=apply_PCA(X_train,X_test)
plot_PC_variance(X_data,explained_variance_ratio)
plot_PC(X_data,components)
time.sleep(20)
start_date=end_date
end_date1=end_date1+datetime.timedelta(days=1)
i+=1
a=interact(data_import_date,
start_date=widgets.DatePicker(value=pd.to_datetime(start_date)),
end_date=widgets.DatePicker(value=pd.to_datetime(end_date)))
def plot_PC(X_data,components):
Np_comp=(1,len(X_data.columns),1)
#interact
def principal_components(PC1=Np_comp,PC2=Np_comp):
fig,ax=plt.subplots(1,1,figsize=(10,10))
plt.figure(5)
print(PC1, PC2)
ax.set_xlabel("Principal Component {}".format(PC1), fontsize=14)
ax.set_ylabel("Principal Component {}".format(PC2), fontsize=14)
ax.set_title("Principal components {0} & {1}".format(PC1,PC2), fontsize=(20))
ax.scatter(components[:,PC1],components[:,PC2])
A fairly rudimentary example that shows how you can run a loop whilst polling for widget changes, but it's not really my area of expertise.
Drag the slider whilst the values are printing and you should see the printed value change to reflect current slider position.
import threading
from IPython.display import display
import ipywidgets as widgets
import time
float_val = widgets.FloatSlider()
display(float_val)
def work(progress):
total = 10
for i in range(total):
time.sleep(1)
print(float_val.value)
thread = threading.Thread(target=work, args=(progress,))
thread.start()
Related
In short, I am trying to run a live monitoring dashboard while calculating data in the background. Here is the code for my monitoring dashboard:
import matplotlib.pyplot as plt
import numpy as np
from matplotlib.animation import FuncAnimation
import psutil
import collections
# function to update the data
def monitor_machine_metrics(i):
cpu_metrics.popleft()
cpu_metrics.append(psutil.cpu_percent(interval=1))
ram_metrics.popleft()
ram_metrics.append(psutil.virtual_memory().percent)
# clear axis
ax.cla()
ax1.cla()
# plot cpu usage
ax.plot(cpu_metrics)
ax.text(len(cpu_metrics)-1, cpu_metrics[-1]+2, "{}%".format(cpu_metrics[-1]))
ax.set_ylim(0,100)
# plot memory usage
ax1.plot(ram_metrics)
ax1.text(len(ram_metrics)-1, ram_metrics[-1]+2, "{}%".format(ram_metrics[-1]))
ax1.set_ylim(0,100)
legend = plt.legend(['CPU Usage', 'RAM Usage'], loc='upper left')
return legend
# start collections with zeros
cpu_metrics = collections.deque(np.zeros(10))
ram_metrics = collections.deque(np.zeros(10))
usage_fig = plt.figure()
ax = plt.subplot()
ax1 = plt.subplot()
def display_dynamic_data(figure, funct, time_interval):
ani = FuncAnimation(figure, funct, interval=time_interval)
plt.show()
For the sake of keeping this simple, let's pretend I just want to print hi while the dashboard is running.
if __name__ == '__main__':
machine_monitoring_process =
multiprocessing.Process(target=display_dynamic_data,args=(usage_fig, monitor_machine_metrics, 1000))
machine_monitoring_process.start()
print('Hi')
Since FuncAnimation runs similar to an infinite loop, I purposely designed for this sub-process to be completely decoupled. While 'hi' prints, the figure comes up but no points will be displayed. However, display_dynamic_data works just fine if we just call the function instead of instantiating it as a sub-process. Unfortunately, I need this to run in parallel, since the real code is doing computations in the background.
Is there something that I am missing here?
Thank you!
When you use multiprocessing.Process, a new process with its own memory and objects is created. Code running in the new process has no effect in the original one. If you want to interchange information, you should use queues or other type of information exchange mechanism.
In your example, display_dynamic_data runs in the new process and exits. Remember that you must keep a reference to the FuncAnimation object to prevent it from being garbage collected.
You can check this answer to grasp what happens when you use multiprocessing.
I try to run multiple functions at the same time in Jupiter Notebook.
I have two web scraping functions that use Selenium and run for an infinite amount of time, both always creating an updated DataFrame. Another function merges the two DataFrames and does some calculations.
As the data always changes and the calculations from the different DataFrames need to be calculated within the same second (The two DataFrames update every 5 seconds), I wonder how I can run all functions at the same time.
As my code is mainly WebScraping I used this more to describe my goal and hopefully make it more readable. I already tried using 'multiprocessing' but it just does not do anything in the notebook.
def FirstWebScraping():
while True:
time.sleep(5).
#getting all data for DataFrame
def SecondtWebScraping():
while True:
time.sleep(5).
#getting all data for DataFrame
def Calculations():
while True:
#merging DataFrame from First- and SecondWebScraping
#doing calculations
#running this function infinite and looking for specific values
#Goal
def run_all_at_the_same_time()
FirstWebScraping()
SecondWebScraping()
Calculations()
Even though threading does not show the same benefits as multiprocessing it worked for me and with selenium. I put a waiting time at the beginning for the Calculations function and from there they were all looped infinitely.
from threading import Thread
if __name__ == '__main__':
Thread(target = FirstWebScraping).start()
Thread(target = SecondWebscraping).start()
Thread(target = Calculations).start()
You can run multiprocessing in Jupyter, if you follow two rules:
Put the worker functions in a separate module.
Protect the main process-only code with if __name__ == '__main__':
Assuming your three functions are moved to worker.py:
import multiprocessing as mp
import worker
if __name__ == '__main__':
mp.Process(target=worker.FirstWebScraping).start()
mp.Process(target=worker.SecondWebscraping).start()
mp.Process(target=worker.Calculations).start()
So I'm using FuncAnimation from matplotlib so dynamically plot some data as it arrives from a serial port (in my project is the vehicle class from dronekit, which is displayed with the green dot), what I have basically is the animation called which every loop is receiving a new vehicle class with data changed so it can be plotted, but for some reason it plots but after a couple of seconds later after the thread of the mission(which allows the "refresh" of the vehicle data it pops up and kills python (Wheel of death), here's what I get:
I've put some tracking prints inside the function that is called when the FuncAnimation starts running, looks like this:
def droneAnimation(i, vehicle, droneScatter):
time.sleep(1)
lat = [vehicle.location.global_relative_frame.lat]
lon = [vehicle.location.global_relative_frame.lon]
alt = [vehicle.location.global_relative_frame.alt]
print("Alt received: " + str(alt))
droneScatter._offsets3d = (lat,lon,alt)
print("Changed pos")
As you can see those prints are triggered the first few seconds but still crashes after a few iterations.
The FuncAnimation is called like this:
fig,droneScatter = plotLiveSimpleFacade(vehicle,w,2)
ani = FuncAnimation(fig,droneAnimation, fargs = (vehicle,droneScatter))
plt.draw()
plt.pause(0.1)
m = threading.Thread(target=MissionStart(vehicle,hmax) , name = "MISSION")
m.start()
For reference: fig is a plt.figure(),droneScatter is just a scatter point, vehicle is the vehicle class containing the data that dynamically updates and the MissionStart thread is just a thread to make the vehicle class change overtime.
I would like to mention as well that the fig is on interactive mode on, and the axes limits are set well (I saw that when you dynamically change data but don't scale the axes may have problems) also, trying different combinations of plt.draw() and plt.plot(block = False) leads me to not plotting at all or just a blank plot.
Since I have no idea of what is causing this I'll put the dronekit tag on this and the threading to see if anyone has any idea!
I've looked onto threading with matplotlib and looks like threading with this said library it's not the best as it's not thread safe, the best bet is to look at multiprocessing with python or approaching the problem in a different manner.
You can find more information at this post
I have a simple Pygame program:
#!/usr/bin/env python
import pygame
from pygame.locals import *
pygame.init()
win = pygame.display.set_mode((400,400))
pygame.display.set_caption("My first game")
But every time I try to run it, I get this:
pygame 2.0.0 (SDL 2.0.12, python 3.8.3)
Hello from the pygame community. https://www.pygame.org/contribute.html
And then nothing happens.
Why I can't run this program?
Your application works well. However, you haven't implemented an application loop:
import pygame
from pygame.locals import *
pygame.init()
win = pygame.display.set_mode((400,400))
pygame.display.set_caption("My first game")
clock = pygame.time.Clock()
run = True
while run:
# handle events
for event in pygame.event.get():
if event.type == pygame.QUIT:
run = False
# update game objects
# [...]
# clear display
win.fill((0, 0, 0))
# draw game objects
# [...]
# update display
pygame.display.flip()
# limit frames per second
clock.tick(60)
pygame.quit()
The typical PyGame application loop has to:
handle the events by calling either pygame.event.pump() or pygame.event.get().
update the game states and positions of objects dependent on the input events and time (respectively frames)
clear the entire display or draw the background
draw the entire scene (blit all the objects)
update the display by calling either pygame.display.update() or pygame.display.flip()
limit frames per second to limit CPU usage with pygame.time.Clock.tick
repl.it/#Rabbid76/PyGame-MinimalApplicationLoop See also Event and application loop
This is interesting. Computer read your program line by line[python]. When all the line are interpreted, the program closed. To solve this problem you need to add a while loop to make sure the program will continue until you close the program.
import pygame,sys
from pygame.locals import *
pygame.init()
pygame.display.set_caption("My first game")
win = pygame.display.set_mode((400,400))
#game loop keeps the game running until you exit the game.
game_running=True
while game_running:
for event in pygame.event.get():
if event.type==QUIT:
pygame.quit()
sys.exit()
win.fill((0, 250, 154)) #Fill the pygame window with specific color. You can use hex or rgb color
pygame.display.update() #Refresh the pygame window
You can check more pygame Examples.
https://github.com/01one/Pygame-Examples
I think this will be helpful.
I'm a novice using matplotlib as an embedded control in my PyQt4 application to display image data. I'd like to be able to allow the user to interactively draw a line on the image by clicking and dragging. I have it working but it is so slow as to be unusable, leading me to believe I'm not going about it the correct way. The only way I can get the line to appear is by forcing the canvas to redraw each time the mouse moves (I suspect this is the cause of the slowdown).
For example, on the mouse down event I store the current coordinates and add a Line2D object to the plot as follows:
def onMouseMove(self, event):
if self.drawingLine:
self.lineStartX = event.xdata
self.lineStopX = event.xdata
self.lineStartY = event.ydata
self.lineStopY = event.ydata
self.line = Line2D([self.lineStartX, self.lineStopX], [self.lineStartY, self.lineStopY], linewidth = 1.5, color = 'r')
self.axes.add_line(self.line)
Then, in my mouse move event I redraw the line as follows:
def onMouseMove(self, event):
if self.drawingLine:
self.lineStopX = event.xdata
self.lineStopY = event.ydata
# Adjust the line to the new endpoint:
self.line.set_data([self.lineStartX, self.lineStopX], [self.lineStartY, self.lineStopY])
# Force a redraw otherwise you don't see any changes:
self.fig.canvas.draw()
As I've stated this approach is unusably slow and hence probably wrong. Can somebody please clue me in to what the proper approach is here? Thank you all in advance.
First off, you will already gain a little by using
self.fig.canvas.draw_idle()
instead of draw(). This redraws the canvas only when it's not currently beeing repainted, saving you a lot of draws.
If this is not enough, you would need to use the technique of blitting. Now since you don't have a minimal example, I will not provide any complete solution for this here, but e.g. the answer to this question, why is plotting with Matplotlib so slow?, has an example of that.
The idea is to store the background, and only redraw the part that changes (here the line).
background = fig.canvas.copy_from_bbox(ax.bbox)
# then during mouse move
fig.canvas.restore_region(background)
line.set_data(...)
ax.draw_artist(line)
fig.canvas.blit(ax.bbox)
# only after mouse has stopped moving
fig.canvas.draw_idle()
This technique is also used internally by some matplotlib widgets, e.g. matplotlib.widgets.Cursor to let the lines follow the cursor quickly.
This brings me to the last point, which is: You don't need to reinvent the wheel. There is a matplotlib.widgets.RectangleSelector, which by defaut draws a rectangle for selection. But you may use its drawtype='line' argument, to change the selection to a line, together with the argument blit=True this should already give you what you need - you will just need to add the code to finally draw a line once the selection is finished.
Note that in the newest matplotlib version, there is even a matplotlib.widgets.PolygonSelector, which may directly be what you need.
matplotlib is built to be flexible and to work with multiple different backends. It is very slow at real-time plotting. The problem is that your mouse move events are very rapid. Anything trying to keep up with the mouse movement will probably be slow. You need to call the plot less often. You can do this by checking the time in your mouse move function and trying to limit the plotting calls to whatever works.
import time
def onMouseMove(self, event):
if self.drawingLine and time.time() - last_time > 0.03: # Change the 0.03 to change how often you plot.
last_time = time.time()
...
I highly suggest pyqtgraph. pyqtgraph has built in rate limiting signals that you can work with to do this.
Below is a basic example of how you can do this.
# Change the style to look like matplotlib
pyqtgraph.setConfigOption("background", QtGui.QColor.fromRgbF(230/255, 230/255, 234/255, 255/255))
pyqtgraph.setConfigOption("background", 'w')
pyqtgraph.setConfigOption("foreground", 'k')
pyqtgraph.setConfigOption("antialias", True)
# Create the widgets and plot items
glw = pyqtgraph.GraphicsLayoutWidget()
pg = glw.addPlot(0, 0)
class MyClass(object):
...
...
def onMouseMove(self, event):
if self.drawingLine:
scene_pos = event[0]
data_pos = pg.getViewBox().mapSceneToView(scene_pos)
x, y = data_pos.x(), data_pos.y()
self.lineStopX = x
self.lineStopY = y
# Adjust the line to the new endpoint:
if not self.line:
self.line = pg.plot(x=[], y=[])
self.line.setData(x=[self.lineStartX, self.lineStopX],
y=[self.lineStartY, self.lineStopY])
mouse_move_sig = pyqtgraph.SignalProxy(pg.scene().sigMouseMoved,
rateLimit=60, slot=onMouseMove)