Python v3.5, Windows 10
I'm using multiple processes and trying to captures user input. Searching everything I see there are odd things that happen when using input() with multiple processes. After 8 hours+ of trying, nothing I implement worked, I'm positive I am doing it wrong but I can't for the life of me figure it out.
The following is a very stripped down program that demonstrates the issue. Now it works fine when I run this program within PyCharm, but when I use pyinstaller to create a single executable it fails. The program constantly is stuck in a loop asking the user to enter something as shown below:.
I am pretty sure it has to do with how Windows takes in standard input from things I've read. I've also tried passing the user input variables as Queue() items to the functions but the same issue. I read you should put input() in the main python process so I did that under if __name__ = '__main__':
from multiprocessing import Process
import time
def func_1(duration_1):
while duration_1 >= 0:
time.sleep(1)
print('Duration_1: %d %s' % (duration_1, 's'))
duration_1 -= 1
def func_2(duration_2):
while duration_2 >= 0:
time.sleep(1)
print('Duration_2: %d %s' % (duration_2, 's'))
duration_2 -= 1
if __name__ == '__main__':
# func_1 user input
while True:
duration_1 = input('Enter a positive integer.')
if duration_1.isdigit():
duration_1 = int(duration_1)
break
else:
print('**Only positive integers accepted**')
continue
# func_2 user input
while True:
duration_2 = input('Enter a positive integer.')
if duration_2.isdigit():
duration_2 = int(duration_2)
break
else:
print('**Only positive integers accepted**')
continue
p1 = Process(target=func_1, args=(duration_1,))
p2 = Process(target=func_2, args=(duration_2,))
p1.start()
p2.start()
p1.join()
p2.join()
You need to use multiprocessing.freeze_support() when you produce a Windows executable with PyInstaller.
Straight out from the docs:
multiprocessing.freeze_support()
Add support for when a program which uses multiprocessing has been frozen to produce a Windows executable. (Has been tested with py2exe, PyInstaller and cx_Freeze.)
One needs to call this function straight after the if name == 'main' line of the main module. For example:
from multiprocessing import Process, freeze_support
def f():
print('hello world!')
if __name__ == '__main__':
freeze_support()
Process(target=f).start()
If the freeze_support() line is omitted then trying to run the frozen executable will raise RuntimeError.
Calling freeze_support() has no effect when invoked on any operating system other than Windows. In addition, if the module is being run normally by the Python interpreter on Windows (the program has not been frozen), then freeze_support() has no effect.
In your example you also have unnecessary code duplication you should tackle.
I have passed a DataFrame from an ipython notebook to a function inside a standard .py file.
In the function, i'm using df['column_name'].values to try to extract all the values from that column.
While debugging (in pycharm), I am running this line in the 'evaluate' tool pycharm provides and it works fine. However, when I run the same line normally (outside the tool window), i get an error:
"TypeError: list indices must be integers or slices, not str"
When looking at my dataframe in the workspace variables section, it is interpreted correctly as a dataframe.
The dataframe object I am passing has 3 columns ('x','y' and 'likelyhood'), each containing integers/floats. The line in question that unpacks the values inside one of these columns is the first line of the function.
Can anyone explain how this can happen? What is the difference that causes the fact that a line of code can work in one, and return an error in the other?
and also what might cause this specific TypeError and how can I solve this bug?
the data frame as printed by df.tail(10):
coords x y likelihood
105570 297.497525 332.355521 1.0
105571 297.463208 332.353797 1.0
105572 297.439774 332.383908 1.0
105573 297.457581 332.458205 1.0
105574 297.487260 332.402202 1.0
105575 297.519772 332.451551 1.0
105576 297.495998 332.431064 1.0
105577 297.516722 332.113481 1.0
105578 297.542539 332.080923 1.0
105579 297.528317 332.046282 1.0
full function:
import math
import numpy as np
import pandas as pd
def filter_jumps(bp, thresh=10):
"""
the function watches for jumps that cannot happen, and interpolates the location of the point acoordingly
"""
x=bp['x'].values
y=bp['y'].values
for i in range(1,len(x)):
if np.abs(x[i]+y[i]-x[i-1]-y[i-1]) > thresh:
start = i-1; end = None; idx=i+1
while not end:
if np.abs(x[idx]+y[idx]-x[idx-1]-y[idx-1]) > thresh:
end = idx
else:
idx += 1
rang = end-start
x[start:end] = np.linspace(x[start], x[end], rang)
y[start:end] = np.linspace(y[start], y[end], rang)
bp['x'] = x
bp['y'] = y
return bp
I am still learning Scilab (5.5.2), so I am writing and running test codes to familiarize myself with the software.
To test the numerical differential equation solver, I started easy from the equation dy/dx = A, which has as solution y = Ax+c (line equation).
This is the code I wrote:
// Function y = A*x+1
function ydot=fn(x, A)
ydot=A
endfunction
A=2;
//Initial conditions
x0=0;
y0=A*x0+1;
//Numerical Solution
x=[0:5];
y= ode(y0,x0,x,fn);
//Analytical solution
y2 = A*x+1;
clf(); plot(x, y); plot(x, y2, '-k');
//End
And these are the unexpected results:
y = 1. 2.7182824 7.3890581 20.085545 54.598182
148.41327
y2 = 1. 3. 5. 7. 9. 11.
It appears that y = e^x. Can someone explain what is going wrong, or what I did wrong?
Just renaming the variables does not change how they are used internally by the ODE solver. Since that solver expects a function with arguments time,state it will interpret the provided function that way.
Renaming the variables back, what you programmed is equivalent to
function ydot=fn(t,y)
ydot = y
endfunction
which indeed has the exponential function as solution.
From the manual you can see that the way to include parameters is to pass the function as a list,
The f argument can also be a list with the following structure: lst=list(realf,u1,u2,...un) where realf is a Scilab function with syntax: ydot = f(t,y,u1,u2,...,un)
function ydot=fn(t,y,A)
ydot = A
endfunction
y= ode(y0,x0,x,list(fn,A));
I'm having trouble solving a discrepancy between something breaking at runtime, but using the exact same data and operations in the python console, having it work fine.
# f_err - currently has value 1.11819388872025
# l_scales - currently a numpy array [1.17840183376334 1.13456764589809]
sq_euc_dists = self.se_term(x1, x2, l_scales) # this is fine. It calls cdists on x1/l_scales, x2/l_scales vectors
return (f_err**2) * np.exp(-0.5 * sq_euc_dists) # <-- errors on this line
The error that I get is
AttributeError: 'Zero' object has no attribute 'exp'
However, calling those exact same lines, with the same f_err, l_scales, and x1, x2 in the console right after it errors out, somehow does not produce errors.
I was not able to find a post referring to the 'Zero' object error specifically, and the non-'Zero' ones I found didn't seem to apply to my case here.
EDIT: It was a bit lacking in info, so here's an actual (extracted) runnable example with sample data I took straight out of a failed run, which when run in isolation works fine/I can't reproduce the error except in runtime.
Note that the sqeucld_dist function below is quite bad and I should be using scipy's cdist instead. However, because I'm using sympy's symbols for matrix elementwise gradients with over 15 partial derivatives in my real data, cdist is not an option as it doesn't deal with arbitrary objects.
import numpy as np
def se_term(x1, x2, l):
return sqeucl_dist(x1/l, x2/l)
def sqeucl_dist(x, xs):
return np.sum([(i-j)**2 for i in x for j in xs], axis=1).reshape(x.shape[0], xs.shape[0])
x = np.array([[-0.29932052, 0.40997373], [0.40203481, 2.19895326], [-0.37679417, -1.11028267], [-2.53012051, 1.09819485], [0.59390005, 0.9735], [0.78276777, -1.18787904], [-0.9300892, 1.18802775], [0.44852545, -1.57954101], [1.33285028, -0.58594779], [0.7401607, 2.69842268], [-2.04258086, 0.43581565], [0.17353396, -1.34430191], [0.97214259, -1.29342284], [-0.11103534, -0.15112815], [0.41541759, -1.51803154], [-0.59852383, 0.78442389], [2.01323359, -0.85283772], [-0.14074266, -0.63457529], [-0.49504797, -1.06690869], [-0.18028754, -0.70835799], [-1.3794126, 0.20592016], [-0.49685373, -1.46109525], [-1.41276934, -0.66472598], [-1.44173868, 0.42678815], [0.64623684, 1.19927771], [-0.5945761, -0.10417961]])
f_err = 1.11466725760716
l = [1.18388412685279, 1.02290811104357]
result = (f_err**2) * np.exp(-0.5 * se_term(x, x, l)) # This runs fine, but fails with the exact same calls and data during runtime
Any help greatly appreciated!
Here is how to reproduce the error you are seeing:
import sympy
import numpy
zero = sympy.sympify('0')
numpy.exp(zero)
You will see the same exception you are seeing.
You can fix this (inefficiently) by changing your code to the following to make things floating point.
def sqeucl_dist(x, xs):
return np.sum([np.vectorize(float)(i-j)**2 for i in x for j in xs],
axis=1).reshape(x.shape[0], xs.shape[0])
It will be better to fix your gradient function using lambdify.
Here's an example of how lambdify can be used on partial d
from sympy.abc import x, y, z
expression = x**2 + sympy.sin(y) + z
derivatives = [expression.diff(var, 1) for var in [x, y, z]]
derivatives is now [2*x, cos(y), 1], a list of Sympy expressions. To create a function which will evaluate this numerically at a particular set of values, we use lambdify as follows (passing 'numpy' as an argument like that means to use numpy.cos rather than sympy.cos):
derivative_calc = sympy.lambdify((x, y, z), derivatives, 'numpy')
Now derivative_calc(1, 2, 3) will return [2, -0.41614683654714241, 1]. These are ints and numpy.float64s.
A side note: np.exp(M) will calculate the element-wise exponent of each of the elements of M. If you are trying to do a matrix exponential, you need np.linalg.exmp.
I want to change the value of a variable on exit so that on the next run, it remains what was last set. This is a short version of my current code:
def example():
x = 1
while True:
x = x + 1
print x
On 'KeyboardInterrupt', I want the last value set in the while loop to be a global variable. On running the code next time, that value should be the 'x' in line 2. Is it possible?
This is a bit hacky, but hopefully it gives you an idea that you can better implement in your current situation (pickle/cPickle is what you should use if you want to persist more robust data structures - this is just a simple case):
import sys
def example():
x = 1
# Wrap in a try/except loop to catch the interrupt
try:
while True:
x = x + 1
print x
except KeyboardInterrupt:
# On interrupt, write to a simple file and exit
with open('myvar', 'w') as f:
f.write(str(x))
sys.exit(0)
# Not sure of your implementation (probably not this :) ), but
# prompt to run the function
resp = raw_input('Run example (y/n)? ')
if resp.lower() == 'y':
example()
else:
# If the function isn't to be run, read the variable
# Note that this will fail if you haven't already written
# it, so you will have to make adjustments if necessary
with open('myvar', 'r') as f:
myvar = f.read()
print int(myvar)
You could save any variables that you want to persist to a text file then read them back in to the script the next time it runs.
Here is a link for reading and writing to text files.
http://docs.python.org/2/tutorial/inputoutput.html#reading-and-writing-files
Hope it helps!