I would expect when a function is vectorized by np.vectorize the total number of method run is the same as the input length. For example if the input is a scalar, the pre-vectorized method should only be run once. In a way, I expect a similar behaviour to map(func, input_array).
However, running the example, you will see that the vectorized method unnecessarily ran func multiple times when the input is only scalar.
Does anyone know if I am using the method wrong? I have also opened a github issue as well.
import numpy as np
import logging
logging.basicConfig(level=logging.DEBUG)
def func(x):
logging.debug(f"Computation started")
return x
func_v1 = np.vectorize(func)
func_v2 = np.vectorize(func_v1)
func_v1(1) # logging shows the method func is ran twice
func_v2(1) # logging shows the method func is ran four times.
According to the numpy documentation, the additional run is required when the otype is not provided. The additional run is to obtain the output type.
Related
I am trying to speed up a batched matrix multiplication problem with numba, but it keeps telling me that it's faster with contiguous code.
Note: I'm using numba version 0.55.1, and numpy version 1.21.5
Here's the problem:
import numpy as np
import numba as nb
def numbaFastMatMult(mat,vec):
result = np.zeros_like(vec)
for n in nb.prange(vec.shape[0]):
result[n,:] = np.dot(vec[n,:], mat[n,:,:])
return result
D,N = 10,1000
mat = np.random.normal(0,1,(N,D,D))
vec = np.random.normal(0,1,(N,D))
result = numbaFastMatMult(mat,vec)
print(mat.data.contiguous)
print(vec.data.contiguous)
print(mat[n,:,:].data.contiguous)
print(vec[n,:].data.contiguous)
clearly all the relevant data is contiguous (run the above code snippet and see the results of print()...
But, when I run this code, I get the following warning:
NumbaPerformanceWarning: np.dot() is faster on contiguous arrays, called on (array(float64, 1d, C), array(float64, 2d, A))
result[n,:] = np.dot(vec[n,:], mat[n,:,:])
2 Extra comments:
This is just a toy problem for replication. I'm actually using something with many more data points, so hoping this will speed up.
I think the "right" way to solve this is with np.tensordot. However, I want to understand what's going on for future reference. For example, this discussion addresses a similar issue, but as far as I can tell, doesn't address why the warning shows up directly.
I've tried adding a decorator:
nb.float64[:,::1](nb.float64[:,:,::1],nb.float64[:,::1]),
I've tried reordering the arrays so the batch index is first (n in the above code)
I've tried printing whether the "mat" variable is contiguous from inside the function
I'll leave this up, but I figured it out:
Outside of a numba function:
mat[n,:,:].data.contiguous==True
but inside numba, mat[n,:,:] is no longer continous.
Changing my code above to np.dot(vec[n], mat[n]) removed the warning.
I'm making this the "correct" answer since it solved my problem. However, according to max9111's response, this behavior may be a bug!
I try to run multiple functions at the same time in Jupiter Notebook.
I have two web scraping functions that use Selenium and run for an infinite amount of time, both always creating an updated DataFrame. Another function merges the two DataFrames and does some calculations.
As the data always changes and the calculations from the different DataFrames need to be calculated within the same second (The two DataFrames update every 5 seconds), I wonder how I can run all functions at the same time.
As my code is mainly WebScraping I used this more to describe my goal and hopefully make it more readable. I already tried using 'multiprocessing' but it just does not do anything in the notebook.
def FirstWebScraping():
while True:
time.sleep(5).
#getting all data for DataFrame
def SecondtWebScraping():
while True:
time.sleep(5).
#getting all data for DataFrame
def Calculations():
while True:
#merging DataFrame from First- and SecondWebScraping
#doing calculations
#running this function infinite and looking for specific values
#Goal
def run_all_at_the_same_time()
FirstWebScraping()
SecondWebScraping()
Calculations()
Even though threading does not show the same benefits as multiprocessing it worked for me and with selenium. I put a waiting time at the beginning for the Calculations function and from there they were all looped infinitely.
from threading import Thread
if __name__ == '__main__':
Thread(target = FirstWebScraping).start()
Thread(target = SecondWebscraping).start()
Thread(target = Calculations).start()
You can run multiprocessing in Jupyter, if you follow two rules:
Put the worker functions in a separate module.
Protect the main process-only code with if __name__ == '__main__':
Assuming your three functions are moved to worker.py:
import multiprocessing as mp
import worker
if __name__ == '__main__':
mp.Process(target=worker.FirstWebScraping).start()
mp.Process(target=worker.SecondWebscraping).start()
mp.Process(target=worker.Calculations).start()
I am typing this code in Google Colabs
def mytimer():
print("Python Program\n")
my_timer = threading.Timer(0.8, mytimer)
my_timer.start()
print("Bye\n")
In some cases I get the output as
Bye
At other times, I get
Bye
Python Program
Why is this difference occuring? Should I add any other line of code? Or any criteria I should be careful about?
You are missing few things here
import logging
import threading
import time
def mytimer():
print("Python Program\n")
my_timer = threading.Timer(0.2, mytimer) # change it to 0.8 will not print
my_timer.start()
print("Bye\n")
threading.Timer - As per docs Call a function after a specified number of seconds when it is 0.2 it will call that function and it prints both when it is 0.8 it is not calling that function so it is not calling, This has nothing to do with colab
you can refer this
Here is a simple DataFrame:
import numpy as np
dd=pd.DataFrame(np.arange(35).reshape(7,5), columns=list('xyzwv'))
dd['w']=list('AABBBCC')
,which is:
Now I try the following code
def func(x):
print(x)
return x
dd.groupby('w').apply(func)
then it prints out:
I think something goes wrong because
is being printed twice.
It looks as if func() is being called twice for the same group.
What mistake did I do?
apply calls the first group twice to work out whether it could do some optimizations, see
http://pandas.pydata.org/pandas-docs/stable/groupby.html#flexible-apply
I am using python 2.7 on windows 7 and I am currently trying to learn parallel processing.
I downloaded the multiprocessing 2.6.2.1 python package and installed it using pip.
When I try to run the foolowing very simple code, the program seems to get stuck, even after one hour it doesn't exit the execution despite the code to be super simple.
What am I missing?? thank you very much
from multiprocessing import Pool
def f(x):
return x*x
array =[1,2,3,4,5]
p=Pool()
result = p.map(f, array)
p.close()
p.join()
print result
The issue here is the way multiprocessing works. Think of it as python opening a new instance and importing all the modules all over again. You'll want to use the if __name__ == '__main__' convention. The following works fine:
import multiprocessing
def f(x):
return x * x
def main():
p = multiprocessing.Pool(multiprocessing.cpu_count())
result = p.imap(f, xrange(1, 6))
print list(result)
if __name__ == '__main__':
main()
I have changed a few other parts of the code too so you can see other ways to achieve the same thing, but ultimately you only need to stop the code executing over and over as python re-imports the code you are running.