I am getting confused about the argument order with RectBivariateSpline. I am reading a set of 2D data which has 343 values along the X axis and 373 values along the Y axis. The routine that reads the data returns it in the "correct" sense such that when I plot it it matplotlib I get a map that has the correct physical orientation. It also returns the X values in an array of 343 elements and the Y values in in array of 373 elements which makes sense.
The scipy documentation for RectBivariateSpline gives the arguments as:
scipy.interpolate.RectBivariateSpline(x, y, z)
However, when I execute
spln = scipy.interpolate.RectBivariateSpline(xval, yval, zval)
I get this error:
TypeError: x dimension of z must have same number of elements as x
I can remove the error by executing
spln = scipy.interpolate.RectBivariateSpline(yval, xval, zval)
but now the x and y values are the wrong way round (in a physical sense at least). Does this mean that the x argument to RectBivariateSpline refers to the first data dimension of the dataset rather than the physical x dimension? I am used to working with data in Fortran-style ordering, which probably is not helping.
In answer to hpaulj's comment, the shapes of the various arrays are:
xval (343,)
yval (373,)
zval (373, 343)
I think the issue is that I am getting confused between 'xy' and 'ij' ordering. Matpoltlib seems to be using 'xy' ordering so I guess I just need to be careful to transpose the ZVAL array when interpolating using scipy
Show us the values of xval.shape, yval.shape and zval.shape
Early in the RectBivariateSpline code it does:
x, y = ravel(x), ravel(y)
....
if not x.size == z.shape[0]:
raise TypeError('x dimension of z must have same number of '
'elements as x')
if not y.size == z.shape[1]:
raise TypeError('y dimension of z must have same number of '
'elements as y')
So the number of rows of z (1st dimension) must match the number of elements in x.
When you display a 2d array, rows are the 1st dimension, going down the page, columns the 2nd. But in a plot, we often expect the first axis, the x one to go across the page.
np.meshgrid lets you specify:
indexing : {'xy', 'ij'}, optional
Cartesian ('xy', default) or matrix ('ij') indexing of output.
See Notes for more details.
The difference between 'xy' indexing and 'ij' indexing might be confusing you. This Spline class is using the 'ij' kind.
An alternative to switching x and y is to use the transpose of z, z.T. Keep in mind that the interpolation points follow the same ordering rules.
Simple example
In [30]: x=np.arange(10)
In [31]: y=np.arange(15)
In [32]: z=x[:,None]*y[None,:]
In [33]: S=interpolate.RectBivariateSpline(x,y,z)
In [34]: S([1,2,3],[4,5,6])
Out[34]:
array([[ 4., 5., 6.],
[ 8., 10., 12.],
[ 12., 15., 18.]])
Contrast the xy v ij in meshgrid:
In [37]: np.meshgrid(x,y)[0].shape
Out[37]: (15, 10)
In [38]: np.meshgrid(x,y,indexing='ij')[0].shape
Out[38]: (10, 15)
z could have been constructed from the ij grids.
X,Y=np.meshgrid(x,y,indexing='ij')
Z = X*Y
Related
Please decipher what the tf.where documentation says about what it does when both x and y are provided.
I suppose it tries to say it will produce a result by:
Broadcast y to the result shape.
Broadcast x to the result shape.
Update y with x elements where the condition is true.
Is this correct?
If x and y are provided (both have non-None values):
tf.where will choose an output shape from the shapes of condition, x, and y that all three shapes are broadcastable to.
Returns
If x and y are provided: A Tensor with the same type as x and y, and shape that is broadcast from condition, x, and y. Otherwise, a Tensor with shape (num_true, dim_size(condition)).
According to latest document on tf.where
If x and y are provided (both have non-None values):
tf.where will choose an output shape from the shapes of condition, x,
and y that all three shapes are broadcastable to.
The condition tensor acts as a mask that chooses whether the
corresponding element / row in the output should be taken from x (if
the element in condition is True) or y (if it is false).
For Example:
tf.where([True, False, False, True], [1,2,3,4], [100,200,300,400])
#It is taking true index values from [1,2,3,4] that is [1,4] and false index values from [100,200,300,400] that is [200,300]
Output:
<tf.Tensor: shape=(4,), dtype=int32, numpy=array([ 1, 200, 300, 4],
dtype=int32)>
As you can see output tensor shape as input numpy arrays.
Simple question: I used a translation and rotation matrix and camera intrinsics matrix to get a 3x4 matrix used to transform 3d points to 2d points (notated as Tform)
I transformed the point [10,-5,1] with the matrix by adding one to the end, and the new point is notated as newpoint.
Now I want to use the newpoint data to transform back to 3D space, where old_est should be equal to old.
I'm looking for the solution to plug into the XXX matrix in my code below
import numpy as np
Tform=np.array([[4000,0,-1600,-8000],[500,5000,868,-8000],[.5,0,.8,-8]])
old=np.array([10,-5,1,1])
newpoint=np.dot(Tform,old)
print(newpoint)
old_est=np.dot(XXX,np.append(newpoint,1))
print(old_est)
Add a 4th row to Tform with the values 0 0 0 1, i.e. the last row of an identity matrix:
>>> m = np.vstack(Tform, np.array([0,0,0,1]))
>>> m
array([[ 4.00e+03, 0.00e+00, -1.60e+03, -8.00e+03],
[ 5.00e+02, 5.00e+03, 8.68e+02, -8.00e+03],
[ 5.00e-01, 0.00e+00, 8.00e-01, -8.00e+00],
[ 0.00e+00, 0.00e+00, 0.00e+00, 1.00e+00]])
Note that you cannot use append because it also flattens the input arrays.
Observe that, when multiplied with old, the 4th component of the result is 1, i.e. the result is equal to np.append(newpoint, 1):
>>> np.dot(m, old)
array([ 3.0400e+04, -2.7132e+04, -2.2000e+00, 1.0000e+00])
----------
It follows that XXX is the inverse of this new matrix:
>>> XXX = np.linalg.inv(m)
>>> np.dot(XXX, np.append(newpoint, 1))
array([10., -5., 1., 1.])
-------------
And we get the components of old back.
Alternatively you can subtract the 4th column of Tform from newpoint and multiply the result with the inverse of the left 3x3 sub-matrix of Tform, but this is slightly fiddly so we might as well let numpy do more of the work :)
In the NumPy v1.15 Reference Guide, the documentation for numpy.dot uses the concept of "sum product".
Namely, we read the following:
If a is an N-D array and b is a 1-D array, it is a sum product over the last axis of a and b.
If a is an N-D array and b is an M-D array (where M>=2), it is a sum product over the last axis of a and the second-to-last axis of b:
dot(a, b)[i,j,k,m] = sum(a[i,j,:] * b[k,:,m])
What is the definition for this "sum product" concept?
(Couldn't find such a definition on Wikipedia, for example.)
https://en.wikipedia.org/wiki/Matrix_multiplication
That is, the entry c[i,j] of the product is obtained by multiplying
term-by-term the entries of the ith row of A and the jth column of B,
and summing these m products. In other words, c[i,j] is the dot product
of the ith row of A and the jth column of B.
https://en.wikipedia.org/wiki/Dot_product
Algebraically, the dot product is the sum of the products of the
corresponding entries of the two sequences of numbers.
In early math classes did you learn to take the matrix product, by running one finger across the rows of A and down the columns of B, mulitplying pairs of numbers and summing them? That motion is part of my intuition of how that product is taken.
For the 1d second argument case, np.dot and np.matmul produce the same thing, but describe the action differently:
If a is an N-D array and b is a 1-D array, it is a sum product over
the last axis of a and b.
If the second argument is 1-D, it is promoted to a matrix by
appending a 1 to its dimensions. After matrix multiplication
the appended 1 is removed.
In [103]: np.dot([[1,2],[3,4]], [1,2])
Out[103]: array([ 5, 11])
In [104]: np.matmul([[1,2],[3,4]], [1,2])
Out[104]: array([ 5, 11])
Appending the dimension to B, does:
In [105]: np.matmul([[1,2],[3,4]], [[1],[2]])
Out[105]:
array([[ 5],
[11]])
This last is a (2,2) with (2,1) => (2,1)
Sometimes it is clearer to express the action in einsum terms:
In [107]: np.einsum('ij,j->i', [[1,2],[3,4]], [1,2])
Out[107]: array([ 5, 11])
j, the last axis of both arrays is the one that gets 'summed'.
For Numpy, "sum product" means "matrix multiplication". Not to confuse with SUMPRODUCT and MMULT functions in Excel, which have different meanings.
I'm struggling to clearly understand the concept of "element" in Pandas. Already went through the document of Pandas and googled around, and I'm guessing it's some sort of row? What do people mean when they say "apply function elment-wise"?
This question came up when I was reading this SO post : How to apply a function to two columns of Pandas dataframe
Pandas is designed for operating vector wise operations i.e. taking entire column and operate some function. This you can term as column wise operation.
But in some cases you may need to operate element by element (i.e. element wise operation). This type operation is not very efficient.
Here is an example:
import pandas as pd
df = pd.DataFrame([a for a in range(100)], columns=['mynum'])
column wise operation
%%timeit
df['add1'] = df.mynum +1
222 µs ± 3.31 µs per loop
When operated element wise
%%timeit
df['add1'] = df.apply(lambda a: a.mynum+1, axis = 1)
2.33 ms ± 85.4 µs per loop
I believe "element" in Pandas is an inherited concept of the "element" from NumPy. Give the first few paragraphs of the docs on ufuncs a read.
Each universal function takes array inputs and produces array outputs by performing the core function element-wise on the inputs (where an element is generally a scalar, but can be a vector or higher-order sub-array for generalized ufuncs).
In mathematics, element-wise operations refer to operations on individual elements of a matrix.
Examples:
import numpy as np
>>> x, y = np.arange(1,5).reshape(2,2), 3*np.eye(2)
>>> x, y
>>> x, y = np.arange(1,5).reshape(2,2), 3*np.eye(2)
>>> x, y
(array([[1, 2],
[3, 4]]),
array([[3., 0.],
[0., 3.]]))
>>> x + y # element-wise addition
array([[4., 2.],
[3., 7.]])
columns of y
>>> np.dot(x,y) # NOT element-wise multiplication (matrix multiplication)
# elements become dot products of the rows of x with columns of y
array([[ 3., 6.],
[ 9., 12.]])
>>> x * y # element-wise multiplication
array([[ 3., 0.],
[ 0., 12.]])
I realize your question was about Pandas, but element-wise in Pandas means the same thing it does in NumPy and in linear algebra (as far as I'm aware).
Element-wise means handling data element by element.
This is a mathematical question, but it is tied to the numpy implementation, so I decided to ask it at SO. Perhaps I'm hugely misunderstanding something, but if so I would like to be put straight.
numpy.ftt.ftt computes DFT according to equation:
numpy.ftt.fftfreq is supposed to return frequencies at which DFT was computed.
Say we have:
x = [0, 0, 1, 0, 0]
X = np.fft.fft(x)
freq = np.fft.fftfreq(5)
Then for signal x, its DFT transformation is X, and frequencies at which X is computed are given by freq. For example X[0] is DFT of x at frequency freq[0], X[1] is DFT of x at frequency freq[1], and so on.
But when I compute DFT of a simple signal by hand with the formula quoted above, my results indicate that X[1] is DFT of x at frequency 1, not at freq[1], X[2] is DFT of x at frequency 2, etc, not at freq[2], etc.
As an example:
In [32]: x
Out[32]: [0, 0, 1, 0, 0]
In [33]: X
Out[33]:
array([
1.00000000+0.j,
-0.80901699-0.58778525j,
0.30901699+0.95105652j, 0.30901699-0.95105652j,
-0.80901699+0.58778525j])
In [34]: freq
Out[34]: array([ 0. , 0.2, 0.4, -0.4, -0.2])
If I compute DFT of above signal for k = 0.2 (or freq[1]), I get
X at freq = 0.2: 0.876 - 0.482j, which isn't X[1].
If however I compute for k = 1 I get the same results as are in X[1] or -0.809 - 0.588j.
So what am I misunderstanding? If numpy.fft.fft(x)[n] is a DFT of x at frequency n, not at frequency numpy.fft.fttfreq(len(x))[n], what is the purpose of numpy.fft.fttfreq?
I think that because the values in the array returned by the numpy.fft.fttfreq are equal to the (k/n)*sampling frequency.
The frequencies of the dft result are equal to k/n divided by the time spacing, because the periodic function's period's amplitude will become the inverse of the original value after fft. You can consider the digital signal function is a periodic sampling function convoluted by the analog signal function. The convolution in time domain means multiplication in frequency domain, so that the time spacing of the input data will affect the frequency spacing of the dft result and the frequency spacing's value will become the original one divided by the time spacing. Originally, the frequency spacing of the dft result is equal to 1/n when the time spacing is equal to 1. So after the dft, the frequency spacing will become 1/n divided by the time spacing, which eqauls to 1/n multiplied by the sampling frequency.
To calculate that, the numpy.fft.fttfreq has two arguments, the length of the input and time spacing, which means the inverse of the sampling rate. The length of the input is equal to n, and the time spacing is equal to the value which the result k/n divided by (Default is 1.)
I have tried to let k = 2, and the result is equal to the X[2] in your example. In this situation, the k/n*1 is equal to the freq[2].
The DFT is a dimensionless basis transform or matrix multiplication. The output or result of a DFT has nothing to do with frequencies unless you know the sampling rate represented by the input vector (samples per second, per meter, per radian, etc.)
You can compute a Goertzel filter of the same length N with k=0.2, but that result isn't contained in an DFT or FFT result of length N. A DFT only contains complex Goertzel filter results for integer k values. And to get from k to the frequency represented by X[k], you need to know the sample rate.
Yours is not a SO question
You wrote
If I compute DFT of above signal for k = 0.2 .
and I reply "You shouldn't"... the DFT can be meaningfully computed only for integer values of k.
The relationship between an index k and a frequency is given by f_k = k Δf or, if you prefer circular frequencies, ω_k = k Δω where Δf = 1/T and Δω = 2πΔf, T being the period of the signal.
The arguments of fftfreq are a bit misleading... the required one is the number of samples n and the optional argument is the sampling interval, by default d=1.0, but at any rate T=n*d and Δf = 1/(n*d)
>>> fftfreq(5) # d=1
array([ 0. , 0.2, 0.4, -0.4, -0.2])
>>> fftfreq(5,2)
array([ 0. , 0.1, 0.2, -0.2, -0.1])
>>> fftfreq(5,10)
array([ 0. , 0.02, 0.04, -0.04, -0.02])
and the different T are 5,10,50 and the respective df are -.2,0.1,0.02 as (I) expected.
Why fftfreq doesn't simply require the signal's period? because it is mainly intended as an helper in demangling the Nyquist frequency issue.
As you know, the DFT is periodic, for a signal x of length N you have that
DFT(x,k) is equal to DFT(x,k+mN) where m is an integer.
This imply that there are only N/2 positive and N/2 negative distinct frequencies and that, when N/2<k<N, the frequency that must be associated with k in the most meaningful way is not k df but (k-N) df.
To perform this, fftfreq needs more information that the period T, hence the choice of requiring n and computing df from an assumption on sampling interval.