I can't not find the source of the difference in the handling of contrast for version 1.75.01 and 1.82. Here are two images that show what it used to look like (1.75),
and what it now looks like:
Unfortunately, rolling back is not trivial as I run into problems with dependencies (especially PIL v PILLOW). The images are created from a numpy array, and I suspect there is something related to how the numbers are getting handled (?type, rounding) when the conversion from array to image occurs, but I can't find the bug. Any help will be deeply appreciated.
Edited - New Minimal Example
#! /bin/bash
import numpy as np
from psychopy import visual,core
def makeRow (n,c):
cp = np.tile(c,[n,n,3])
cm = np.tile(-c,[n,n,3])
cpm = np.hstack((cp,cm))
return(cpm)
def makeCB (r1,r2,nr=99):
#nr is repeat number
(x,y,z) = r1.shape
if nr == 99:
nr = x/2
else:
hnr = nr/2
rr = np.vstack((r1,r2))
cb=np.tile(rr,[hnr,hnr/2,1])
return(cb)
def makeTarg(sqsz,targsz,con):
wr = makeRow(sqsz,1)
br = makeRow(sqsz,-1)
cb = makeCB(wr,br,targsz)
t = cb*con
return(t)
def main():
w = visual.Window(size = (400,400),units = "pix", winType = 'pyglet',colorSpace = 'rgb')
fullCon_np = makeTarg(8,8,1.0)
fullCon_i = visual.ImageStim(w, image = fullCon_np,size = fullCon_np.shape[0:2][::-1],pos = (-100,0),colorSpace = 'rgb')
fullCon_ih = visual.ImageStim(w, image = fullCon_np,size = fullCon_np.shape[0:2][::-1],pos = (-100,0),colorSpace = 'rgb')
fullCon_iz = visual.ImageStim(w, image = fullCon_np,size = fullCon_np.shape[0:2][::-1],pos = (-100,0),colorSpace = 'rgb')
fullCon_ih.contrast = 0.5
fullCon_ih.setPos((-100,100))
fullCon_iz.setPos((-100,-100))
fullCon_iz.contrast = 0.1
partCon_np = makeTarg(8,8,0.1)
partCon_i = visual.ImageStim(w, image = partCon_np,pos = (0,0), size = partCon_np.shape[0:2][::-1],colorSpace = 'rgb')
zeroCon_np = makeTarg(8,8,0.0)
zeroCon_i = visual.ImageStim(w, image = zeroCon_np,pos=(100,0), size = zeroCon_np.shape[0:2][::-1],colorSpace = 'rgb')
fullCon_i.draw()
partCon_i.draw()
fullCon_ih.draw()
fullCon_iz.draw()
zeroCon_i.draw()
w.flip()
core.wait(15)
core.quit()
if __name__ == "__main__":
main()
Which yields this:
The three checker-boards along the horizontal have the contrast changed in the array when generated before conversion to the image. The Vertical left shows that changing the image contrast afterwards works fine. The reason I can't use this is that a) I have collected a lot of data with the last version, and b) I want to grade the contrast of those big long bars in the centre programatically by multiplying one array against another, e.g. using a log scale or some other function, and doing the math is easier in numpy.
I still suspect the issue is in the conversion from np.array -> pil.image. The dtype of these array is float64, but even if I coerce to float32 nothing changes. If you examine the array before conversion at half contrast it is filled with 0.5 and -0.5 numbers, but all the negative numbers are getting turned to black and black is being set to zero at the time of conversion by psychopy.tools.imagetools.array2image I think.
OK, yes, the problem was to do with the issue of the scale for the array values. Basically, you've found a corner case that PsychoPy isn't handling correctly (i.e. a bug).
Explanation:
PsychoPy has a complex set of transformation rules for handling image/textures; it tries to deduce what you're going to do with this image and whether it should be stored in a way that supports colour manipulations (signed float) or not (can be an unsigned byte). In your case PsychoPy was getting this wrong; the fact that the array was filled with floats made PsychoPy think it could do color transforms, but the fact that it was NxNx3 suggest it shouldn't (we don't want to specify a "color" for something that already has its color specified for every pixel as rgb vals).
Workarounds (any one of these):
Just provide your array as NxN, not NxNx3. This is the right thing to do anyway; it means less for you to compute/store and by providing "intensity" values these can then be recolored on-the-fly. This is roughly what you had discovered already in providing just one slice of your NxNx3 array, but the point is that you could/should only create one slice in the first place.
Use GratingStim, which converts everything to signed floating point values rather than trying to work out what's best (potentially then you'd need to work out the spatial frequency stuff though)
You could add a line to fix it by rescaling your array (*0.5+0.5) but you'd have to set something so that this only occurred for this version (we'll fix it before the next release)
Basically, I'm suggesting you do (1) because that already works for past, present and future versions and is more efficient anyway. But thanks for letting us know - I'll try to make sure we catch this one in future
best wishes
Jon
The code is too long for me to read through and work out the actual problem.
What might be the problem is the issue of where zero should be. I think for a while numpy arrays were treated as having vals 0:1 whereas the rest of PsychoPy expects values to be -1:1 so it might be that you need to rescale your values with array=array*2-1 to get back to old (bad behaviour). Or check opacity too, which might have a similar issue. If you write a minimal example I'll read/test it properly
Thanks
Related
I am utilizing the pcolormesh function in Matplotlib to plot a series of gridded data (in parallel) across multiple map domains. The code snippet relevant to this question is as follows:
im = ax2.pcolormesh(xgrid, ygrid, data.variable.data[0], cmap=cmap, norm=norm, alpha=0.90, facecolor=None)
Where: xgrid = array of longitude points, ygrid = array of latitude points, data.variable.data[0] = array of corresponding data values, cmap = defined colormap, & norm = defined value normalization
Consider the following image generated from the provided code:
The undesired result I've found in the image above is what appears to be outlines around each grid square, or perhaps better described as patchwork that stands out slightly as the mesh alpha is reduced below 1.
I've set facecolor=None assuming that would remove these outlines, to no avail. What additions or corrections can I make to remove this feature?
Alright, I have some segmented image data s which is defined by a mask s, which is basically a 3D binary field where 1 defines a voxel which is part of the segment and 0 defines a voxel which is not part of it. I am trying to get representation of this segment which is as small as possible. this is rather easy, I can use the following:
compressed = s.flatten()[m.flatten() == 1]
my question is, given compressed and m, is there a similar Numpy method or an equally fast way to reconstruct s?
Alright feeling pretty dumb that I didn't realize that I could've just used the following:
a = np.zeros(m.shape)
a[m == 1] = compressed
>>> np.equal(a, s).all()
True
Hope this still is of some use to someone who isn't able to figure this out either!
I have been tinkering around a lot with tensorflow in the past few days however I am quite unsure whether a function I wrote would break the backpropagation in a Neural network. I thought I'd ask here before I try to integrate this function in a NN. So the basic setup is I want to add two matricies with
op = tf.add(tfObject, tfImageBackground)
where tfImageBackground is some constant image. (i.e. an RGBA image of size 800, 800 with R = G = B = A = 0) and the tfObject is again a matrix with the same dimenstion however we get that with the function I am unsure about
def getObject(vector):
objectId = vector[0]
x = vector[1]
y = vector[2]
xEnd = baseImageSize-(x+objectSize)
yStart =baseImageSize- (y+objectSize)
padding = tf.convert_to_tensor([[x, xEnd], [yStart, y],[0,0]])
RTensor = tfObjectMatrix[objectId,:,:,0:1]
GTensor = tfObjectMatrix[objectId,:,:,1:2]
BTensor = tfObjectMatrix[objectId,:,:,2:3]
ATensor = tfObjectMatrix[objectId,:,:,3:4]
paddedR = tf.pad(tensor = RTensor,
paddings= padding,
mode='Constant',
name='padAverageRed',
constant_values=255)
...
generates padding for every channel
...
finalTensor=tf.concat([paddedR, paddedG, paddedB, paddedA], 2)
return finalTensor
The tfObjectMatrix is a list of images which never change.
I did check wether I was able to generate a tf.gradient from the op, which turned out to work. I am unsure if that is sufficient for backpropagation to work though.
Thanks for you time and effort. Any input at all would be greatly appreciated.
TensorFlow will backpropagate to everything by default. As per your code, everything will receive gradients with a training operation from an optimizer. So to answer your question, backpropagation will work.
The only thing to consider, is that you say tfObjectMatrix is a list of images that will not change. So you might not want it to receive any gradients. Therefore you might want to look into tf.stop_gradient() and maybe use it like OM = tf.stop_gradient( tfObjectMatrix ) and work with that OM in your function.
I'm trying to make a powerspectrum from an experimental dataset which I am reading in, and then to fit it to an theoretical curve. Now everything is working fine and I'm not getting errors, except for the fact that my curve keeps differing by a factor of 1000 from the data and I have absolutely no idea what the problem could be. I've asked a few people, but to no avail. (I hope that you guys will be able to help)
Anyways, I'm pretty sure that its not the units, as they were tripple checked by me and 2 others. Basically, I need to fit a powerspectrum to an equation by using the least squares method.
I can't post the whole code, as its rather long and a bit messy, but this is the fourier part, I added comments to all arrays and vars which have not been declared in the code)
#Calculate stuff
Nm = 10**-6 #micro to meter
KbT = 4.10E-21 #Joule
T = 297. #K
l = zvalue*Nm #meter
meany = np.mean(cleandatay*Nm) #meter (cleandata is the array that I read in from a cvs at the start.)
SDy = sum((cleandatay*Nm - meany)**2)/len(cleandatay) #meter^2
FmArray[0][i] = ((KbT*l)/SDy) #N
#print FmArray[0][i]
print float((i*100/len(filelist)))#how many % done?
#fourier
dt = cleant[1]-cleant[0] #timestep
N = len(cleandatay) #Same for cleant, its the corresponding time to cleandatay
Here is where the fourier part starts, I take the fft and turn it into a powerspectrum. Then I calculate the corresponding freq steps with the array freqs
fouriery = np.fft.fft((cleandatay*(10**-6)))
fourierpower = (np.abs(fouriery))**2
fourierpower = fourierpower[1:N/2] #remove 0th datapoint and /2 (remove negative freqs)
fourierpower = fourierpower*dt #*dt to account for steps
freqs = (1.+np.arange((N/2)-1.))/50.
#Least squares method
eta = 8.9E-4 #pa*s
Rbead = 0.5E-6#meter
constant = 2*KbT/(3*eta*pi*Rbead)
omega = 2*pi*freqs #rad/s
Wcarray = 2.*pi*np.arange(0,30, 0.02003) #0.02 = 30/len(freqs)
ChiSq = np.zeros(len(Wcarray))
for k in range(0, len(Wcarray)):
Py = (constant / (Wcarray[k]**2 + omega**2))
ChiSq[k] = sum((fourierpower - Py)**2)
pylab.loglog(omega, Py)
print k*100/len(Wcarray)
index = np.where(ChiSq == min(ChiSq))
cutoffw = Wcarray[index]
Pygoed = (constant / (Wcarray[index]**2 + omega**2))
print cutoffw
print constant
print min(ChiSq)
pylab.loglog(omega,ChiSq)
So I have no idea what could be going wrong, I think its the fft, as nothing else can really go wrong.
Below is the pic I get when I plot all the fit lines against the spectrum, as you can see it is off by about 1000 (actually exactly 1000, as this leaves a least square residue of 10^-22, but I can't just randomly multiply without knowing why)
Just to elaborate on the picture. The green dots are the fft spectrum, the lines are the fits, the red dot is where it thinks the cutoff frequency is, and the blue line is the chi-squared fit, looking for the lowest value.
Take a look at the documentation for the FFT that you are using. Many FFTs introduce a scaling factor that is usually N * result (number of samples). Multiplying by 1/N will scale the results back in line. (You said that the result is 1000 too high....could it be that you are using a 1024 size FFT?)
Your library FFT routine might include a scale factor of 1/sqrt(n).
Check the documentation for the fft you used, as the proportion of the scale factor allocated between the fft and the ifft is arbitrary.
!I have values in the form of (x,y,z). By creating a list_plot3d plot i can clearly see that they are not quite evenly spaced. They usually form little "blobs" of 3 to 5 points on the xy plane. So for the interpolation and the final "contour" plot to be better, or should i say smoother(?), do i have to create a rectangular grid (like the squares on a chess board) so that the blobs of data are somehow "smoothed"? I understand that this might be trivial to some people but i am trying this for the first time and i am struggling a bit. I have been looking at the scipy packages like scipy.interplate.interp2d but the graphs produced at the end are really bad. Maybe a brief tutorial on 2d interpolation in sagemath for an amateur like me? Some advice? Thank you.
EDIT:
https://docs.google.com/file/d/0Bxv8ab9PeMQVUFhBYWlldU9ib0E/edit?pli=1
This is mostly the kind of graphs it produces along with this message:
Warning: No more knots can be added because the number of B-spline
coefficients
already exceeds the number of data points m. Probably causes:
either
s or m too small. (fp>s)
kx,ky=3,3 nx,ny=17,20 m=200 fp=4696.972223 s=0.000000
To get this graph i just run this command:
f_interpolation = scipy.interpolate.interp2d(*zip(*matrix(C)),kind='cubic')
plot_interpolation = contour_plot(lambda x,y:
f_interpolation(x,y)[0], (22.419,22.439),(37.06,37.08) ,cmap='jet', contours=numpy.arange(0,1400,100), colorbar=True)
plot_all = plot_interpolation
plot_all.show(axes_labels=["m", "m"])
Where matrix(c) can be a huge matrix like 10000 X 3 or even a lot more like 1000000 x 3. The problem of bad graphs persists even with fewer data like the picture i attached now where matrix(C) was only 200 x 3. That's why i begin to think that it could be that apart from a possible glitch with the program my approach to the use of this command might be totally wrong, hence the reason for me to ask for advice about using a grid and not just "throwing" my data into a command.
I've had a similar problem using the scipy.interpolate.interp2d function. My understanding is that the issue arises because the interp1d/interp2d and related functions use an older wrapping of FITPACK for the underlying calculations. I was able to get a problem similar to yours to work using the spline functions, which rely on a newer wrapping of FITPACK. The spline functions can be identified because they seem to all have capital letters in their names here http://docs.scipy.org/doc/scipy/reference/interpolate.html. Within the scipy installation, these newer functions appear to be located in scipy/interpolate/fitpack2.py, while the functions using the older wrappings are in fitpack.py.
For your purposes, RectBivariateSpline is what I believe you want. Here is some sample code for implementing RectBivariateSpline:
import numpy as np
from scipy import interpolate
# Generate unevenly spaced x/y data for axes
npoints = 25
maxaxis = 100
x = (np.random.rand(npoints)*maxaxis) - maxaxis/2.
y = (np.random.rand(npoints)*maxaxis) - maxaxis/2.
xsort = np.sort(x)
ysort = np.sort(y)
# Generate the z-data, which first requires converting
# x/y data into grids
xg, yg = np.meshgrid(xsort,ysort)
z = xg**2 - yg**2
# Generate the interpolated, evenly spaced data
# Note that the min/max of x/y isn't necessarily 0 and 100 since
# randomly chosen points were used. If we want to avoid extrapolation,
# the explicit min/max must be found
interppoints = 100
xinterp = np.linspace(xsort[0],xsort[-1],interppoints)
yinterp = np.linspace(ysort[0],ysort[-1],interppoints)
# Generate the kernel that will be used for interpolation
# Note that the default version uses three coefficients for
# interpolation (i.e. parabolic, a*x**2 + b*x +c). Higher order
# interpolation can be used by setting kx and ky to larger
# integers, i.e. interpolate.RectBivariateSpline(xsort,ysort,z,kx=5,ky=5)
kernel = interpolate.RectBivariateSpline(xsort,ysort,z)
# Now calculate the linear, interpolated data
zinterp = kernel(xinterp, yinterp)