I have the following code section (appropriately simplified)
cpdef double func(double[:] x, double[:] y) nogil:
cdef:
double[:] _y
_y = y # Here's my trouble
_y[2] = 2. - y[1]
_y[1] = 1.
return func2(x, _y)
I'm trying to create a copy of y that I can manipulate in the function. The problem is, any changes made to _y get passed back to y. I don't want to make changes to y, just to this temporary copy of it.
The function is nogil, so I can't use _y = y.copy(). (already tried). I also tried _y[:] = y, based on the cython guidance pages, but I apparently can't do that if _y hasn't been initialized yet.
So... how do I make a copy of a 1d vector without invoking the gil?
Related
Consider the following code
#tf.function
def get_derivatives(function_to_diff,X):
f = function_to_diff(X)
## Derivatives
W = X[:,0]
Z = X[:,1]
V = X[:,2]
df_dW = tf.gradients(f, X[:,0])
return df_dW
I wanted get_derivatives to return the partial derivative of function_to_diff with respect to the first element of X.
However, when I run
def test_function(X):
return tf.pow(X[:,0],2) * X[:,1] * X[:,2]
get_derivatives(test_function,X)
I get None.
If I use unconnected_gradients='zero' for tf.graidents, I'd get zeros. In other words, the gradients are disconnected.
Questions
Why are the gradients disconnected?
How can I get the derivative with respect to the first element of X, i.e. how can I restore the connection? I know that if I wrote
def test_function(x,y,z)
return tf.pow(x,2) * y * z
#tf.function
def get_derivatives(function_to_diff,x,y,z):
f = function_to_diff(x,y,z)
df_dW = tf.gradients(f, x)
return df_dW
This could fix the problem. What if my function can only take in one argument, i.e. what if my function looks like test_function(X)? For example, test_function could be a trained neural network that takes in only one argument.
I am trying to filter evident measurement mistakes from my data using the 3-sigma rule. x is a numpy array of measurement points and y is an arrray of measured values. To remove wrong points from my data, I zip x.tolist() and y.tolist(), then filter by the second element of each tuple, then I need to convert my zip back into two lists. I tried to first covert my list of tuples into a list of lists, then convert it to numpy 2D array and then take two 1D-slices of it. It looks like the first slice is correct, but then it outputs the following:
x = np.array(list(map(list, list(filter(flt, list(zap))))))[:, 0]
IndexError: too many indices for array
I don't understand what am I doing wrong. Here's the code:
x = np.array(readCol(0, l))
y = np.array(readCol(1, l))
n = len(y)
stdev = np.std(y)
mean = np.mean(y)
print("Stdev is: " + str(stdev))
print("Mean is: " + str(mean))
def flt(n):
global mean
global stdev
global x
if abs(n[1] - mean) < 3*stdev:
return True
else:
print('flt function finds an error: ' + str(n[1]))
return False
def filtration(N):
print(Fore.RED + 'Filtration function launched')
global y
global x
global stdev
global mean
zap = zip(x.tolist(), y.tolist())
for i in range(N):
print(Fore.RED + ' Filtration step number ' + str(i) + Style.RESET_ALL)
y = np.array(list(map(list, list(filter(flt, list(zap))))))[:, 1]
print(Back.GREEN + 'This is y: \n' + Style.RESET_ALL)
print(y)
x = np.array(list(map(list, list(filter(flt, list(zap))))))[:, 0]
print(Back.GREEN + 'This is x: \n' + Style.RESET_ALL)
print(x)
print('filtration fuction main step')
stdev = np.std(y)
print('second step')
mean = np.mean(y)
print('third step')
Have you tried to test the problem line step by step?
x = np.array(list(map(list, list(filter(flt, list(zap))))))[:, 0]
for example:
temp = np.array(list(map(list, list(filter(flt, list(zap))))))
print(temp.shape, temp.dtype)
x = temp[:, 0]
Further break down might be needed, but since [:,0] is the only indexing operation in this line, I'd start there.
Without further study of the code and/or some examples, I'm not going to try to speculate what the nested lists are doing.
The error sounds like temp is not 2d, contrary to your expectations. That could be because temp is object dtype, and composed of lists the vary in length. That seems to be common problem when people make arrays from downloaded databases.
I have a loss function I would like to try and minimize:
def lossfunction(X,b,lambs):
B = b.reshape(X.shape)
penalty = np.linalg.norm(B, axis = 1)**(0.5)
return np.linalg.norm(np.dot(X,B)-X) + lambs*penalty.sum()
Gradient descent, or similar methods, might be useful. I can't calculate the gradient of this function analytically, so I am wondering how I can numerically calculate the gradient for this loss function in order to implement a descent method.
Numpy has a gradient function, but it requires me to pass a scalar field at pre determined points.
You could try scipy.optimize.minimize
For your case a sample call would be:
import scipy.optimize.minimize
scipy.optimize.minimize(lossfunction, args=(b, lambs), method='Nelder-mead')
You could estimate the derivative numerically by a central difference:
def derivative(fun, X, b, lambs, h):
return (fun(X + 0.5*h,b,lambs) - fun(X - 0.5*h,b,lambs))/h
And use it like this:
# assign values to X, b, lambs
# set the value of h
h = 0.001
print derivative(lossfunction, X, b, lambs, h)
The code above is valid for dimX = 1, some modifications are needed to account for multidimensional vector X:
def gradient(fun, X, b, lambs, h):
res = []
for i in range (0,len(X)):
t1 = list(X)
t1[i] = t1[i] + 0.5*h
t2 = list(X)
t2[i] = t2[i] - 0.5*h
res = res + [(fun(t1,b,lambs) - fun(t2,b,lambs))/h]
return res
Forgive the naivity of the code, I barely know how to write some python :-)
I'm trying to mirror an image. That is, if, e.g., a person is facing to the left, when the program terminates I want that person to now be facing instead to the right.
I understand how mirroring works in JES, but I'm unsure how to proceed here.
Below is what I'm trying; be aware that image is a global variable declared in another function.
def flipPic(image):
width = getWidth(image)
height = getHeight(image)
for y in range(0, height):
for x in range(0, width):
left = getPixel(image, x, y)
right = getPixel(image, width-x-1, y)
color = getColor(left)
setColor(right, color)
show(image)
return image
try this
width = getWidth(pic)
height = getHeight(pic)
for y in range (0,height):
for x in range (0, width/2):
left=getPixel(pic, x, y)
right=getPixel(pic, width-x-1,y)
color1=getColor(left)
color2=getColor(right)
setColor(right, color1)
setColor(left, color2)
repaint(pic)
I personally find that repaint is confusing for newbies (like me!).
I'd suggest something like this:
def mirrorImage(image):
width = getWidth(image)
height = getHeight(image)
for y in range (0,height):
for x in range (0, width/2):
left=getPixel(pic, x, y)
right=getPixel(pic, width-x-1,y)
color1=getColor(left)
color2=getColor(right)
setColor(right, color1)
setColor(left, color2)
show(image)
return image
mirrorImage(image)
This seems to work well.. I put some comments in so you can rewrite in your own style.
feel free to ask questions but I think your question may already be answered^^
#this function will take the pixel values for a selected picture and
#past them to a new canvas but fliped over!
def flipPic(pict):
#here we take the height and width of the original picture
width=getWidth(pict)
height=getHeight(pict)
#here we make and empty canvas
newPict=makeEmptyPicture(width,height)
#the Y for loop is setting the range to working for the y axes the started the X for loop
for y in range(0, height):
#the X for loop is setting the range to work in for the x axis
for x in range(0, width):
#here we are collecting the colour information for the origional pix in range of X and
colour=getColor(getPixel(pict,x,y))
#here we are setting the colour information to its new position on the blank canvas
setColor(getPixel(newPict,width-x-1,y),colour)
#setColor(getPixel(newPict,width-x-1,height-y-1),colour)#upsidedown
show(newPict)
#drive function
pict = makePicture(pickAFile())
show(pict)
flipPic(pict)
Might be easier to read if you copy it over to JES first :D
BTW I got full marks for this one in my intro to programming class ;)
Compare the following:
par(mfrow = 2)
image(x=as.POSIXct(1:100, origin = "1970-1-1"), z= matrix(rnorm(100*100), 100))
plot(x=as.POSIXct(1:100, origin = "1970-1-1"), (rnorm(100)))
It seems like image (and so, image.default) fails to take the class-defined Axis functions into account when plotting, while plot does. This is problematic, since I'm in the process of implementing some classes with custom pretty and format specifications that would have their own way of plotting an axis, so I want to having my own axis functions be called when image is used, than always use the numeric version.
I understand there's a way round this by plotting axis manually, calling image first with xaxt = "n", for instance. But this seems inconvenient and messy. Ideally, I'd like a solution that can just drop in to overlay the existing function while breaking as few things as possible. Any thoughts?
The simplest way is to suppress the axes on the call to image() with axes = FALSE then add them yourself. E.g.:
set.seed(42)
X <- as.POSIXct(1:100, origin = "1970-1-1")
Z <- matrix(rnorm(100*100), 100)
image(x = X, z = Z, axes = FALSE)
axis(side = 2)
axis.POSIXct(side = 1, x = X)
box()
This can also be done using the Axis() S3 generic:
image(x = X, z = Z, axes = FALSE)
axis(side = 2)
Axis(x = X, side = 1)
box()
So to actually try to Answer the Question, I would wrap this into a function that automates the various steps:
Image <- function(x = seq(0, 1, length.out = nrow(z)),
y = seq(0, 1, length.out = ncol(z)),
z, ...) {
image(x = X, z = Z, ..., axes = FALSE)
Axis(x = y, side = 2, ...)
Axis(x = X, side = 1, ...)
box()
}
Write your axis functions as S3 methods for the Axis() generic and class x and y appropriately do that your methods are called and the above should just work. All you need to remember is to change image() to Image().
You could also write your own image() method, and add your class to x to have it called instead of image.default() Depends on whether it makes sense for x to have a class or not?
The reason I would do this is that the only way to change image.default() R-wide is to edit the function and assign it to the graphics namespace or source your version and call it explicitly. This would need to be done each and every time you started R. A custom function could easily be sourced or added to your own local package of misc functions that you arrange to load as R is starting so that it is automagically available. See ?Startup for details of how you might arrange for this.