I'm trying to compare two frequency spectra but I am confused over a number of points.
One device samples at 40 Hz the other at 100 Hz and so I'm not sure if I need to take this into account. Anyway I have produced frequency spectra from both devices and now I wish to compare these. How can I do correlation at each point so that I get pearson correlations at each point. I know how to do an overall one of course but I want to see points of strong correlation and those less strong?
If you are calculating power spectral densities P(f), then it doesn't matter how your original signal x(t) is sampled. You can directly and quantitatuively compare both spectra. To make sure that you have calculated the spectral densities you can explicitly check Parsevals theorem:
$ \int P(f) df = \int x(t)^2 dt $
Of course you have to think about which frequencies are actually evaluated Remember that a fft gives you frequencies from f = 1/T until or below the Nyquist frequency f_ny = 1/(2 dt) depending on the number of samples in x(t) being even or odd.
Here's a python example code for psd
def psd(x,dt=1.):
"""Computes one-sided power spectral density of x.
PSD estimated via abs**2 of Fourier transform of x
Takes care of even or odd number of elements in x:
- if x is even both f=0 and Nyquist freq. appear once
- if x is odd f=0 appears once and Nyquist freq. does not appear
Note that there are no tapers applied: This may lead to leakage!
Parseval's theorem (Variance of time series equal to integral over PSD) holds and can be checked via
print ( np.var(x), sum(Px*f[1]) )
Accordingly, the etsimated PSD is independent of time series length
Author/date: M. von Papen / 16.03.2017
"""
N = np.size(x)
xf = np.fft.fft(x)
Px = abs(xf)**2./N*dt
f = np.arange(N/2+1)/(N*dt)
if np.mod(N,2) == 0:
Px[1:N/2] = 2.*Px[1:N/2]
else:
Px[1:N/2+1] = 2.*Px[1:N/2+1]
# Take one-sided spectrum
Px = Px[0:N/2+1]
return Px, f
Related
I'm trying to write a simple python script that recovers the amplitude and phase of a sine wave from it's fourier transformation.
I should be able to do this by calculating the magnitude, and direction of the vector defined by the real and imaginary numbers for the fourier transform, for a given frequency, i.e:
Amplitude_at_freq = sqrt(real_component_at_freq^2 + imag_component_at_freq^2)
Phase = arctan(imag_component_at_freq/real_component_at_freq)
Ref: 1 min 45 seconds into this video: https://www.youtube.com/watch?time_continue=106&v=IWQfj05i87g
I've written a simple python script using numpy's fft library to try and reproduce this, but despite writing out my derivation exactly as above, am failing to get the amplitude and phase, although I can recover the original frequency of my test sine wave correctly. This previous post Calculating amplitude from np.fft and this one Why FFT does not retrieve original amplitude when increasing signal length points to the same problem (where amplitude is off by factor of 2). Specifically the solution is to "multiply by 2 (half of spectrum is removed so energy must be preserved)," but I need clarification on what that means. Secondly there's no mention of my issue with recovering the phase change, and the amplitude is calculated differently from what I have here.
# Define amplitude, phase, frequency
_A = 4 # Amplitude
_p = 0 # Phase shift
_f = 8 # Frequency
# Construct a simple signal
t = np.linspace(0, 2*np.pi, 1024 + 1)[:-1]
g = _A * np.sin(_f * t + _p)
# Apply the fourier transform
ff = np.fft.fft(g)
# Get frequency of original signal
ff_ii = np.where(np.abs(ff) > 1.0)[0][0] # Just get one frequency, the other one is just mirrored freq at negative value
print('frequency of:', ff_ii)
# Get the complex vector at that frequency to retrieve amplitude and phase shift
yy = ff[ff_ii]
# Calculate the amplitude
T = t.shape[0] # domain of x; which we will divide height to get freq amplitude
A = np.sqrt(yy.real**2 + yy.imag**2)/T
print('amplitude of:', A)
# Calculate phase shift
phi = np.arctan(yy.imag/yy.real)
print('phase change:', phi)
However, the result I'm getting is:
>> frequency of: 8
>> amplitude of: 2.0
>> phase change: 1.5707963267948957
So the frequency is accurate, but I'm getting an amplitude of 2, when it should be 4, and phase change of pi/2, when it should be zero.
Is my math wrong, or is my understanding of numpy's fft implementation incorrect?
Fourier analyses a signal as a sum of exp(i.2.pi.f.t) terms, so it sees
A.sin(2.pi.f1.t) as:
-i.A/2.exp(i.2.pi.f1.t)+i.A/2.exp(-i.2.pi.f1.t),
which is mathematically equal. So in Fourier terms, you have both the positive frequency f1 and negative -f1 with complex values -A/2.i and A/2.i respectively. So each 'side' has only half the amplitude, but if you add them together (in the inverse Fourier transform) you get back amplitude A. This split in positive and negative frequency is where your missing factor 2 is if you only look at one (positive or negative) side of the spectrum. This is often done in practice because for real signals, the other half is trivial to derive given one.
Look into the exact mathematics Euler's formula and Fourier transform.
Is there a way to chose the x/y output axes range from np.fft2 ?
I have a piece of code computing the diffraction pattern of an aperture. The aperture is defined in a 2k x 2k pixel array. The diffraction pattern is basically the inner part of the 2D FT of the aperture. The np.fft2 gives me an output array same size of the input but with some preset range of the x/y axes. Of course I can zoom in by using the image viewer, but I have already lost detail. What is the solution?
Thanks,
Gert
import numpy as np
import matplotlib.pyplot as plt
r= 500
s= 1000
y,x = np.ogrid[-s:s+1, -s:s+1]
mask = x*x + y*y <= r*r
aperture = np.ones((2*s+1, 2*s+1))
aperture[mask] = 0
plt.imshow(aperture)
plt.show()
ffta= np.fft.fft2(aperture)
plt.imshow(np.log(np.abs(np.fft.fftshift(ffta))**2))
plt.show()
Unfortunately, much of the speed and accuracy of the FFT come from the outputs being the same size as the input.
The conventional way to increase the apparent resolution in the output Fourier domain is by zero-padding the input: np.fft.fft2(aperture, [4 * (2*s+1), 4 * (2*s+1)]) tells the FFT to pad your input to be 4 * (2*s+1) pixels tall and wide, i.e., make the input four times larger (sixteen times the number of pixels).
Begin aside I say "apparent" resolution because the actual amount of data you have hasn't increased, but the Fourier transform will appear smoother because zero-padding in the input domain causes the Fourier transform to interpolate the output. In the example above, any feature that could be seen with one pixel will be shown with four pixels. Just to make this fully concrete, this example shows that every fourth pixel of the zero-padded FFT is numerically the same as every pixel of the original unpadded FFT:
# Generate your `ffta` as above, then
N = 2 * s + 1
Up = 4
fftup = np.fft.fft2(aperture, [Up * N, Up * N])
relerr = lambda dirt, gold: np.abs((dirt - gold) / gold)
print(np.max(relerr(fftup[::Up, ::Up] , ffta))) # ~6e-12.
(That relerr is just a simple relative error, which you want to be close to machine precision, around 2e-16. The largest error between every 4th sample of the zero-padded FFT and the unpadded FFT is 6e-12 which is quite close to machine precision, meaning these two arrays are nearly numerically equivalent.) End aside
Zero-padding is the most straightforward way around your problem. But it does cost you a lot of memory. And it is frustrating because you might only care about a tiny, tiny part of the transform. There's an algorithm called the chirp z-transform (CZT, or colloquially the "zoom FFT") which can do this. If your input is N (for you 2*s+1) and you want just M samples of the FFT's output evaluated anywhere, it will compute three Fourier transforms of size N + M - 1 to obtain the desired M samples of the output. This would solve your problem too, since you can ask for M samples in the region of interest, and it wouldn't require prohibitively-much memory, though it would need at least 3x more CPU time. The downside is that a solid implementation of CZT isn't in Numpy/Scipy yet: see the scipy issue and the code it references. Matlab's CZT seems reliable, if that's an option; Octave-forge has one too and the Octave people usually try hard to match/exceed Matlab.
But if you have the memory, zero-padding the input is the way to go.
I'm looking for the most abundant frequency in a periodic signal.
I'm trying to understand what do I get if I perform a Fourier transformation on a periodic signal and filter for frequencies which have negative fft values.
In other words, what do the axis of plots 2 and 3 (see below) express? I'm plotting frequency (cycles/second) over the fft-transformed signal - what do negative values on the y axis mean, and would it make sense that I'd be interested in only those?
import numpy as np
import scipy
# generate data
time = scipy.linspace(0,120,4000)
acc = lambda t: 10*scipy.sin(2*pi*2.0*t) + 5*scipy.sin(2*pi*8.0*t) + 2*scipy.random.random(len(t))
signal = acc(time)
# get frequencies from decomposed fft
W = np.fft.fftfreq(signal.size, d=time[1]-time[0])
f_signal = np.fft.fft(signal)
# filter signal
# I'm getting only the "negative" part!
cut_f_signal = f_signal.copy()
# filter noisy frequencies
cut_f_signal[(W < 8.0)] = 0
cut_f_signal[(W > 8.2)] = 0
# inverse fourier to get filtered frequency
cut_signal = np.fft.ifft(cut_f_signal)
# plot
plt.subplot(221)
plt.plot(time,signal)
plt.subplot(222)
plt.plot(W, f_signal)
plt.subplot(223)
plt.plot(W, cut_f_signal)
plt.subplot(224)
plt.plot(time, cut_signal)
plt.show()
The FFT of a real-valued input signal will produce a conjugate symmetric result. (That's just the way the math works best.) So, for FFT result magnitudes only of real data, the negative frequencies are just mirrored duplicates of the positive frequencies, and can thus be ignored when analyzing the result.
However if you want to do the inverse and compute the IFFT, you will need to feed the IFFT a conjugate symmetric negative half (or upper half, above Fs/2) of frequency data, or else your IFFT result will end up producing a complex result (e.g. with non-zero imaginary (sqrt(-1)) components, rarely what one want when dealing with base-band real data).
If you want to filter the FFT data and end up with real results from an IFFT, you will need to filter the positive and negative frequencies symmetrically identically to maintain the needed symmetry.
The FFT also produces a complex result, where the value and sign the components (real and imaginary) of each result bin represents the phase as well as the magnitude of the component basis vector (complex sinusoid, or real cosine plus real sine components). Any negative value just represents a phase rotation from if the same result was positive.
As #hotpaw2 already wrote in his comment above, the result of a FFT performed on a real signal in time domain generates complex values in frequency domain.
The input value f_signal of your plot command is a vector of complex values.
plt.subplot(222)
plt.plot(W, f_signal)
This results in meaningless output.
You should plot the absolute values of f_signal.
If you are interested in the phase you should plot the angle, too.
In Matlab this would look like this:
% Plot the absolute values of f_signal
plot(W, abs(f_signal));
% Plot the phase of f_signal
plot(W, (unwrap(angle(f_signal)));
I am using the FFT function in NumPy to do some signal processing. I have array called signal
which has one data point for each hour and has a total of 576 data points. I use the following code on signal to look at its fourier transform.
t = len(signal)
ft = fft(signal,n=t)
mgft=abs(ft)
plot(mgft[0:t/2+1])
I see two peaks but I am unsure as to what the units of the x axis are i.e., how they map onto hours? Any help would be appreciated.
Given sampling rate FSample and transform blocksize N, you can calculate the frequency resolution deltaF, sampling interval deltaT, and total capture time capT using the relationships:
deltaT = 1/FSample = capT/N
deltaF = 1/capT = FSample/N
Keep in mind also that the FFT returns value from 0 to FSample, or equivalently -FSample/2 to FSample/2. In your plot, you're already dropping the -FSample/2 to 0 part. NumPy includes a helper function to calculate all this for you: fftfreq.
For your values of deltaT = 1 hour and N = 576, you get deltaF = 0.001736 cycles/hour = 0.04167 cycles/day, from -0.5 cycles/hour to 0.5 cycles/hour. So if you have a magnitude peak at, say, bin 48 (and bin 528), that corresponds to a frequency component at 48*deltaF = 0.0833 cycles/hour = 2 cycles/day.
In general, you should apply a window function to your time domain data before calculating the FFT, to reduce spectral leakage. The Hann window is almost never a bad choice. You can also use the rfft function to skip the -FSample/2, 0 part of the output. So then, your code would be:
ft = np.fft.rfft(signal*np.hanning(len(signal)))
mgft = abs(ft)
xVals = np.fft.fftfreq(len(signal), d=1.0) # in hours, or d=1.0/24 in days
plot(xVals[:len(mgft)], mgft)
Result of fft transformation doesn't map to HOURS, but to frequencies contained in your dataset. It would be beneficial to have your transformed graph so we can see where the spikes are.
You might be having spike at the beginning of the transformed buffer, since you didn't do any windowing.
In general, the dimensional units of frequency from an FFT are the same as the dimensional units of the sample rate attributed to the data fed to the FFT, for example: per meter, per radian, per second, or in your case, per hour.
The scaled units of frequency, per FFT result bin index, are N / theSampleRate, with the same dimensional units as above, where N is the length of the full FFT (you might only be plotting half of this length in the case of strictly real data).
Note that each FFT result peak bin represents a filter with a non-zero bandwidth, so you might want to add some uncertainty or error bounds to the result points you map onto frequency values. Or even use an interpolation estimation method, if needed and appropriate for the source data.
I am trying to write a simple low pass filter using scipy, but I need help defining the parameters.
I have 3.5 million records in the time series data that needs to be filtered, and the data is sampled at 1000 hz.
I am using signal.firwin and signal.lfilter from the scipy library.
The parameters I am choosing in the code below do not filter my data at all. Instead, the code below simply produces something that graphically looks like the same exact data except for a time phase distortion that shifts the graph to the right by slightly less than 1000 data points (1 second).
In another software program, running a low pass fir filter through graphical user interface commands produces output that has similar means for each 10 second (10,000 data point) segment, but that has drastically lower standard deviations so that we essentially lose the noise in this particular data file and replace it with something that retains the mean value while showing longer term trends that are not polluted by higher frequency noise. The other software's parameters dialog box contains a check box that allows you to select the number of coefficients so that it "optimizes based on sample size and sampling frequency." (Mine are 3.5 million samples collected at 1000 hz, but I would like a function that uses these inputs as variables.)
*Can anyone show me how to adjust the code below so that it removes all frequencies above 0.05 hz?* I would like to see smooth waves in the graph rather than just the time distortion of the same identical graph that I am getting from the code below now.
class FilterTheZ0():
def __init__(self,ZSmoothedPylab):
#------------------------------------------------------
# Set the order and cutoff of the filter
#------------------------------------------------------
self.n = 1000
self.ZSmoothedPylab=ZSmoothedPylab
self.l = len(ZSmoothedPylab)
self.x = arange(0,self.l)
self.cutoffFreq = 0.05
#------------------------------------------------------
# Run the filter
#------------------------------------------------------
self.RunLowPassFIR_Filter(self.ZSmoothedPylab, self.n, self.l
, self.x, self.cutoffFreq)
def RunLowPassFIR_Filter(self,data, order, l, x, cutoffFreq):
#------------------------------------------------------
# Set a to be the denominator coefficient vector
#------------------------------------------------------
a = 1
#----------------------------------------------------
# Create the low pass FIR filter
#----------------------------------------------------
b = signal.firwin(self.n, cutoff = self.cutoffFreq, window = "hamming")
#---------------------------------------------------
# Run the same data set through each of the various
# filters that were created above.
#---------------------------------------------------
response = signal.lfilter(b,a,data)
responsePylab=p.array(response)
#--------------------------------------------------
# Plot the input and the various outputs that are
# produced by running each of the various filters
# on the same inputs.
#--------------------------------------------------
plot(x[10000:20000],data[10000:20000])
plot(x[10000:20000],responsePylab[10000:20000])
show()
return
Cutoff is normalized to the Nyquist frequency, which is half the sampling rate. So with FS = 1000 and FC = 0.05, you want cutoff = 0.05/500 = 1e-4.
from scipy import signal
FS = 1000.0 # sampling rate
FC = 0.05/(0.5*FS) # cutoff frequency at 0.05 Hz
N = 1001 # number of filter taps
a = 1 # filter denominator
b = signal.firwin(N, cutoff=FC, window='hamming') # filter numerator
M = FS*60 # number of samples (60 seconds)
n = arange(M) # time index
x1 = cos(2*pi*n*0.025/FS) # signal at 0.025 Hz
x = x1 + 2*rand(M) # signal + noise
y = signal.lfilter(b, a, x) # filtered output
plot(n/FS, x); plot(n/FS, y, 'r') # output in red
grid()
The filter output is delayed half a second (the filter is centered on tap 500). Note that the DC offset added by the noise is preserved by the low-pass filter. Also, 0.025 Hz is well within the pass range, so the output swing from peak to peak is approximately 2.
The units of cutoff freq are probably [0,1) where 1.0 is equivalent to FS (the sampling frequency). So if you really mean 0.05Hz and FS=1000Hz, you'd want to pass cutoffFreq / 1000. You may need a longer filter to get such a low cutoff.
(BTW you are passing some arguments but then using the object attributes instead, but I don't see that introducing any obvious bugs yet...)