Related
I wrote a script to convert the EELS map to EELS line scan data, and it works well with DM 2.0. I can deal with it as directly collected EELS line scan data with DM2.0. But it does not work with DM 3.0 and the above version. It looks DM 3.0 still recognizes it as an EELS map file. DM3.0 still tried to generate elemental maps with multiple windows from it not generate line scan profiles with one single window and said the display type is incorrect. Not sure what code/command I need to add to fit the DM 3.0 and above versions. Appreciate any suggestions/comments.
image source
source := getFrontImage()
number sizeX,sizeY,sizeZ
source.Get3Dsize(sizeX,sizeY,sizeZ)
Result( "Original size:"+ sizeX +"; "+ sizeY+"; "+sizeZ+""+"\n" )
image sum
number regionsizeX = 1
number regionsizeY = sizeY
number row,col
Result( "new size:"+ regionsizeX +"; "+ regionsizeY+"; "+row+""+row+" "+"\n" )
sum := RealImage("Line Scan of [0,0,"+regionSizeY+","+regionSizeX+"]",4,sizeX/regionSizeX,sizey/regionsizeY,sizeZ)
//sum := ImageClone(source)
sum = 0
for (row=0;row<regionsizeY;row++) for (col=0;col<regionSizeX;col++)
{
OpenAndSetProgressWindow("Doing sub-block","x = "+col," y = "+row)
sum += Slice3(source,col,row,0,0,sizeX/regionSizeX,regionsizeX,1,sizeY/regionSizeY,regionSizeY,2,sizez,1)
}
OpenAndSetProgressWindow("","","")
ImageCopyCalibrationFrom(sum, source)
sum.setdisplaytype (1)
sum.SetStringNote( "Meta Data:Format", "Spectrum image" )
sum.SetStringNote( "Meta Data:Signal", "EELS" )
showimage(sum)
I'm also a bit confused by your terminology. When you write "Convert a Map into a LineScan" do you mean:
a) Convert a 3D Spectrum-Image (xy scan, one spectral dimension) into a 2D Line-Scan Spectrum-Image (one spatial dimension, one spectral dimension)
or
b) Convert a 2D Map (xy scan, one value) in a 1D Line-Trace (one spatial dimension, one value per point) ?
I suppose you mean a) and answer to that.
I'm surprised if/that your script would work without issues in GMS 2.
Your final (supposedly line-scan SI) data is still a 3D dataset with the dispersion running in Z-direction. This is not the typical LineScan SI data format (which is dispersion in X, spatial dimension in Y, no Z dimension).
Am I right in thinking that you want to "collapse" your 3D data along the y-dimension (by summing) ?
If so, what you want to do is:
// Get Input
image src3D := GetFrontImage()
number sizeX,sizeY,sizeZ
if ( 3 != src3D.ImageGetNumDimensions() ) Throw( "Input not 3D")
src3D.Get3Dsize(sizeX,sizeY,sizeZ)
// Optional: Use Rect-ROI on image to specify area
// If no selection, will return full FOV
number t,l,b,r
src3D.GetSelection(t,l,b,r)
// Prepare output (for summing 3D of rect-selection along y)
// NB: 2D container has:
// X dimension (spatial) along Y
// Z dimension (energy) along X
number nSpatial = r - l
number nSpectral = sizeZ
number eOrig, eScale, sOrig, sScale
string eUnit, sUnit
src3D.ImageGetDimensionCalibration(0, sOrig, sScale, sUnit, 0)
src3D.ImageGetDimensionCalibration(2, eOrig, eScale, eUnit, 0)
string name
if ( nSpatial != sizeX )
name = "Y-projection of [" + t + "," + l + "," + b + "," + r + "] over " + (b-t) + " rows"
else
name = "Y-projection over " + sizeY + " rows"
image dst2D := RealImage( name, 4, nSpectral, nSpatial )
dst2D.ImageSetDimensionCalibration(0, eOrig, eScale, eUnit, 0)
dst2D.ImageSetDimensionCalibration(1, sOrig, sScale, sUnit, 0)
// Copy Tags (contains necessary meta tags! Meta Data Format & Signal)
dst2D.ImageGetTagGroup().TagGroupCopyTagsFrom( src3D.ImageGetTagGroup() )
// Display (with captions)
dst2D.ShowImage()
dst2D.ImageGetImageDisplay(0).ImageDisplaySetCaptionOn(1)
number doFAST = 0
if ( !doFAST )
{
// Perform actuall summing (projection) by summing "line by line"
// into the LinePlot SI. Note the flipping of input and output dimensions!
for( number y = t; y<b; y++ )
{
number lineNumber = y - t
dst2D.slice2( 0,0,0, 0,nSpectral,1, 1,nSpatial,1 ) += src3D.slice2( l,y,0, 2,nSpectral,1, 0,nSpatial,1)
}
}
else
{
// Alternative (faster) projection. Use dedicated projection command.
image proj := src3D[l,t,0,r,b,nSpectral].Project(1) // Outcome of projectsion is having x=x and y=z, so need flip axis
dst2D = proj.slice2(0,0,0, 1,nSpectral,1, 0,nSpatial,1 ) // Flip axis
}
// Display (with captions)
dst2D.ShowImage()
dst2D.ImageGetImageDisplay(0).ImageDisplaySetCaptionOn(1)
Note that iterating using slice blocks is fast, but not as fast as the dedicated 'Project' command available in latest GMS versions. The example uses either, but lines #51-56 might not be available in older GMS.
Edit to address comment below:
Other relevant meta data for spectra is also found in the tags. For EELS, in particular the collection & convergence angle as well as the HT is of importance. You can find out about the tag-path by checking the tags of a properly acquired EELS spectrum.
Or, you can find out about their tag-paths by "converting" an empty 1D line-plot into an EELS spectrum and then attempting a quantification. You will get the prompt to fill in the data. After doing so, check the tags of the image:
I'm trying to calculate the average Luminance of an RGB image. To do this, I find the luminance of each pixel i.e.
L(r,g,b) = X*r + Y*g + Z*b (some linear combination).
And then find the average by summing up luminance of all pixels and dividing by width*height.
To speed this up, I'm using pyopencl.reduction.ReductionKernel
The array I pass to it is a Single Dimension Numpy Array so it works just like the example given.
import Image
import numpy as np
im = Image.open('image_00000001.bmp')
data = np.asarray(im).reshape(-1) # so data is a single dimension list
# data.dtype is uint8, data.shape is (w*h*3, )
I want to incorporate the following code from the example into it . i.e. I would make changes to datatype and the type of arrays I'm passing. This is the example:
a = pyopencl.array.arange(queue, 400, dtype=numpy.float32)
b = pyopencl.array.arange(queue, 400, dtype=numpy.float32)
krnl = ReductionKernel(ctx, numpy.float32, neutral="0",
reduce_expr="a+b", map_expr="x[i]*y[i]",
arguments="__global float *x, __global float *y")
my_dot_prod = krnl(a, b).get()
Except, my map_expr will work on each pixel and convert each pixel to its luminance value.
And reduce expr remains the same.
The problem is, it works on each element in the array, and I need it to work on each pixel which is 3 consecutive elements at a time (RGB ).
One solution is to have three different arrays, one for R, one for G and one for B ,which would work, but is there another way ?
Edit: I changed the program to illustrate the char4 usage instead of float4:
import numpy as np
import pyopencl as cl
import pyopencl.array as cl_array
deviceID = 0
platformID = 0
workGroup=(1,1)
N = 10
testData = np.zeros(N, dtype=cl_array.vec.char4)
dev = cl.get_platforms()[platformID].get_devices()[deviceID]
ctx = cl.Context([dev])
queue = cl.CommandQueue(ctx)
mf = cl.mem_flags
Data_In = cl.Buffer(ctx, mf.READ_WRITE, testData.nbytes)
prg = cl.Program(ctx, """
__kernel void Pack_Cmplx( __global char4* Data_In, int N)
{
int gid = get_global_id(0);
//Data_In[gid] = 1; // This would change all components to one
Data_In[gid].x = 1; // changing single component
Data_In[gid].y = 2;
Data_In[gid].z = 3;
Data_In[gid].w = 4;
}
""").build()
prg.Pack_Cmplx(queue, (N,1), workGroup, Data_In, np.int32(N))
cl.enqueue_copy(queue, testData, Data_In)
print testData
I hope it helps.
I have a question regarding the math that Apple is using in it's speak here example.
A little background: I know that average power and peak power returned by the AVAudioRecorder and AVAudioPlayer is in dB. I also understand why the RMS power is in dB and that it needs to be converted into amp using pow(10, (0.5 * avgPower)).
My question being:
Apple uses this formula to create it's "Meter Table"
MeterTable::MeterTable(float inMinDecibels, size_t inTableSize, float inRoot)
: mMinDecibels(inMinDecibels),
mDecibelResolution(mMinDecibels / (inTableSize - 1)),
mScaleFactor(1. / mDecibelResolution)
{
if (inMinDecibels >= 0.)
{
printf("MeterTable inMinDecibels must be negative");
return;
}
mTable = (float*)malloc(inTableSize*sizeof(float));
double minAmp = DbToAmp(inMinDecibels);
double ampRange = 1. - minAmp;
double invAmpRange = 1. / ampRange;
double rroot = 1. / inRoot;
for (size_t i = 0; i < inTableSize; ++i) {
double decibels = i * mDecibelResolution;
double amp = DbToAmp(decibels);
double adjAmp = (amp - minAmp) * invAmpRange;
mTable[i] = pow(adjAmp, rroot);
}
}
What are all the calculations - or rather, what do each of these steps do? I think that mDecibelResolution and mScaleFactor are used to plot 80dB range over 400 values (unless I'm mistaken). However, what's the significance of inRoot, ampRange, invAmpRange and adjAmp? Additionally, why is the i-th entry in the meter table "mTable[i] = pow(adjAmp, rroot);"?
Any help is much appreciated! :)
Thanks in advance and cheers!
It's been a month since I've asked this question, and thanks, Geebs, for your response! :)
So, this is related to a project that I've been working on, and the feature that is based on this was implemented about 2 days after asking that question. Clearly, I've slacked off on posting a closing response (sorry about that). I posted a comment on Jan 7, as well, but circling back, seems like I had a confusion with var names. >_<. Thought I'd give a full, line by line answer to this question (with pictures). :)
So, here goes:
//mDecibelResolution is the "weight" factor of each of the values in the meterTable.
//Here, the table is of size 400, and we're looking at values 0 to 399.
//Thus, the "weight" factor of each value is minValue / 399.
MeterTable::MeterTable(float inMinDecibels, size_t inTableSize, float inRoot)
: mMinDecibels(inMinDecibels),
mDecibelResolution(mMinDecibels / (inTableSize - 1)),
mScaleFactor(1. / mDecibelResolution)
{
if (inMinDecibels >= 0.)
{
printf("MeterTable inMinDecibels must be negative");
return;
}
//Allocate a table to store the 400 values
mTable = (float*)malloc(inTableSize*sizeof(float));
//Remember, "dB" is a logarithmic scale.
//If we have a range of -160dB to 0dB, -80dB is NOT 50% power!!!
//We need to convert it to a linear scale. Thus, we do pow(10, (0.05 * dbValue)), as stated in my question.
double minAmp = DbToAmp(inMinDecibels);
//For the next couple of steps, you need to know linear interpolation.
//Again, remember that all calculations are on a LINEAR scale.
//Attached is an image of the basic linear interpolation formula, and some simple equation solving.
//As per the image, and the following line, (y1 - y0) is the ampRange -
//where y1 = maxAmp and y0 = minAmp.
//In this case, maxAmp = 1amp, as our maxDB is 0dB - FYI: 0dB = 1amp.
//Thus, ampRange = (maxAmp - minAmp) = 1. - minAmp
double ampRange = 1. - minAmp;
//As you can see, invAmpRange is the extreme right hand side fraction on our image's "Step 3"
double invAmpRange = 1. / ampRange;
//Now, if we were looking for different values of x0, x1, y0 or y1, simply substitute it in that equation and you're good to go. :)
//The only reason we were able to get rid of x0 was because our minInterpolatedValue was 0.
//I'll come to this later.
double rroot = 1. / inRoot;
for (size_t i = 0; i < inTableSize; ++i) {
//Thus, for each entry in the table, multiply that entry with it's "weight" factor.
double decibels = i * mDecibelResolution;
//Convert the "weighted" value to amplitude using pow(10, (0.05 * decibelValue));
double amp = DbToAmp(decibels);
//This is linear interpolation - based on our image, this is the same as "Step 3" of the image.
double adjAmp = (amp - minAmp) * invAmpRange;
//This is where inRoot and rroot come into picture.
//Linear interpolation gives you a "straight line" between 2 end-points.
//rroot = 0.5
//If I raise a variable, say myValue by 0.5, it is essentially taking the square root of myValue.
//So, instead of getting a "straight line" response, by storing the square root of the value,
//we get a curved response that is similar to the one drawn in the image (note: not to scale).
mTable[i] = pow(adjAmp, rroot);
}
}
Response Curve image: As you can see, the "Linear curve" is not exactly a curve. >_<
Hope this helps the community in some way. :)
No expert, but based on physics and math:
Assume the max amplitude is 1 and minimum is 0.0001 [corresponding to -80db, which is what min db value is set to in the apple example : #define kMinDBvalue -80.0 in AQLevelMeter.h]
minAmp is the minimum amplitude = 0.0001 for this example
Now, all that is being done is the amplitudes in multiples of the decibel resolution are being adjusted against the minimum amplitude:
adjusted amplitude = (amp-minamp)/(1-minamp)
This makes the range of the adjusted amplitude = 0 to 1 instead of 0.0001 to 1 (if that was desired).
inRoot is set to 2 here. rroot=1/2 - raising to power 1/2 is square root. from apple's file:
// inRoot - this controls the curvature of the response. 2.0 is square root, 3.0 is cube root. But inRoot doesn't have to be integer valued, it could be 1.8 or 2.5, etc.
Essentially gives you a response between 0 and 1 again, and the curvature of that varies based on what value you set for inRoot.
How to apply a filter on CMRotationMatrix? maybe kalman filter. I need fix the noise of CMRotationMatrix (transformFromCMRotationMatrix), to get a linear values of result matrix
This matrix values will be convert to XYZ, in my case I'm simulate 3D on 2D screen like that:
// Casting matrix to x, y
vec4f_t v;
multiplyMatrixAndVector(v, projectionCameraTransform, boxMatrix);
float x = (v[0] / v[3] + 1.0f) * 0.5f;
float y = (v[1] / v[3] + 1.0f) * 0.5f;
CGPointMake(x * self.bounds.size.width, self.bounds.size.height - (y * self.bounds.size.height));
code:
// define variable
mat4f_t cameraTransform;
// start the display link loop
- (void)startDisplayLink
{
displayLink = [CADisplayLink displayLinkWithTarget:self selector:#selector(onDisplayLink:)];
[displayLink setFrameInterval:1];
[displayLink addToRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode];
}
// stop the display link loop
- (void)stopDisplayLink
{
[displayLink invalidate];
displayLink = nil;
}
// event of display link
- (void)onDisplayLink:(id)sender
{
CMDeviceMotion *d = motionManager.deviceMotion;
if (d != nil) {
CMRotationMatrix r = d.attitude.rotationMatrix;
transformFromCMRotationMatrix(cameraTransform, &r);
[self setNeedsDisplay];
}
}
// function trigger before [self setNeedDisplay];
void transformFromCMRotationMatrix(vec4f_t mout, const CMRotationMatrix *m)
{
mout[0] = (float)m->m11;
mout[1] = (float)m->m21;
mout[2] = (float)m->m31;
mout[3] = 0.0f;
mout[4] = (float)m->m12;
mout[5] = (float)m->m22;
mout[6] = (float)m->m32;
mout[7] = 0.0f;
mout[8] = (float)m->m13;
mout[9] = (float)m->m23;
mout[10] = (float)m->m33;
mout[11] = 0.0f;
mout[12] = 0.0f;
mout[13] = 0.0f;
mout[14] = 0.0f;
mout[15] = 1.0f;
}
// Matrix-vector and matrix-matricx multiplication routines
void multiplyMatrixAndVector(vec4f_t vout, const mat4f_t m, const vec4f_t v)
{
vout[0] = m[0]*v[0] + m[4]*v[1] + m[8]*v[2] + m[12]*v[3];
vout[1] = m[1]*v[0] + m[5]*v[1] + m[9]*v[2] + m[13]*v[3];
vout[2] = m[2]*v[0] + m[6]*v[1] + m[10]*v[2] + m[14]*v[3];
vout[3] = m[3]*v[0] + m[7]*v[1] + m[11]*v[2] + m[15]*v[3];
}
In general I would distinguish between improving the signal noise ratio and smooting the signal.
Signal Improvement
If you really want to be better than Apple's Core Motion which has already a sensor fusion algorithm implemented, stay prepared for a long term project with an uncertain outcome. In this case you would be better off to take the raw accelerometer and gyro signals to build your own sensor fusion algorithm but you have to care about a lot of problems like drift, hardware dependency of different iPhone versions, hardware differences of the sensors within same generation, ... So my advice: try everything to avoid it.
Smoothing
This just means interpolating two or more signals and building kind of average. I don't know about any suitable approach to use for rotation matrices directly (maybe there is one) but you can use quaternions instead (more resources: OpenGL Tutorial Using Quaternions to represent rotation or Quaternion FAQ).
The resulting quaternion of such an interpolation can be multiplied with your vector to get the projection similarly to the matrix way (you may look at Finding normal vector to iOS device for more information).
Interpolation between two unit quaternions representing rotations can be accomplished with Slerp. In practice you will use what is described as Geometric Slerp in Wikipedia. If you have two points in time t1 and t2 and the corresponding quaternions q1 and q2 and the angular distance omega between them, the formula is:
q'(q1, q2, t) = sin((1- t) * omega) / sin(omega) * q0 + sin(t * omega) / sin(omega) * q1
t should be 0.5 because you want the average between both rotations. Omega can be calculated by the dot product:
cos(omega) = q1.q2 = w1*w2 + x1*x2 + y1*y2 + z1*z2
If this approach using two quaternions still doesn't match your needs, you can repeat this by using slerp (slerp (q1, q2), slerp (q3, q4)). Some notes:
From a performance point fo view it's not that cheap to perform three sin and one arccos call in your run loop 1/frequency times per second. Thus you should avoid using too many points
In your case all signals are close to each other especially when using high sensor frequencies. You have to take care about angles that are very small and let 1/sin(omega) explode. In this case set sin(x) ≈ x
Like in other filters like low-pass filter the more points in time you use the more time delay you get. So if you have frequency f you will get about 0.5/f sec delay when using two points and 1.5/f for the double slerp.
If something appears weird, check that your resulting quaternions are unit quaternions i.e. ||q|| = 1
If you are running into performance issues you might have a look at Hacking Quaternions
The C++ project pbrt at github contains a quaternion class to get some inspiration from.
I'm working with GPS data, getting values every second and displaying current position on a map. The problem is that sometimes (specially when accuracy is low) the values vary a lot, making the current position to "jump" between distant points in the map.
I was wondering about some easy enough method to avoid this. As a first idea, I thought about discarding values with accuracy beyond certain threshold, but I guess there are some other better ways to do. What's the usual way programs perform this?
Here's a simple Kalman filter that could be used for exactly this situation. It came from some work I did on Android devices.
General Kalman filter theory is all about estimates for vectors, with the accuracy of the estimates represented by covariance matrices. However, for estimating location on Android devices the general theory reduces to a very simple case. Android location providers give the location as a latitude and longitude, together with an accuracy which is specified as a single number measured in metres. This means that instead of a covariance matrix, the accuracy in the Kalman filter can be measured by a single number, even though the location in the Kalman filter is a measured by two numbers. Also the fact that the latitude, longitude and metres are effectively all different units can be ignored, because if you put scaling factors into the Kalman filter to convert them all into the same units, then those scaling factors end up cancelling out when converting the results back into the original units.
The code could be improved, because it assumes that the best estimate of current location is the last known location, and if someone is moving it should be possible to use Android's sensors to produce a better estimate. The code has a single free parameter Q, expressed in metres per second, which describes how quickly the accuracy decays in the absence of any new location estimates. A higher Q parameter means that the accuracy decays faster. Kalman filters generally work better when the accuracy decays a bit quicker than one might expect, so for walking around with an Android phone I find that Q=3 metres per second works fine, even though I generally walk slower than that. But if travelling in a fast car a much larger number should obviously be used.
public class KalmanLatLong {
private final float MinAccuracy = 1;
private float Q_metres_per_second;
private long TimeStamp_milliseconds;
private double lat;
private double lng;
private float variance; // P matrix. Negative means object uninitialised. NB: units irrelevant, as long as same units used throughout
public KalmanLatLong(float Q_metres_per_second) { this.Q_metres_per_second = Q_metres_per_second; variance = -1; }
public long get_TimeStamp() { return TimeStamp_milliseconds; }
public double get_lat() { return lat; }
public double get_lng() { return lng; }
public float get_accuracy() { return (float)Math.sqrt(variance); }
public void SetState(double lat, double lng, float accuracy, long TimeStamp_milliseconds) {
this.lat=lat; this.lng=lng; variance = accuracy * accuracy; this.TimeStamp_milliseconds=TimeStamp_milliseconds;
}
/// <summary>
/// Kalman filter processing for lattitude and longitude
/// </summary>
/// <param name="lat_measurement_degrees">new measurement of lattidude</param>
/// <param name="lng_measurement">new measurement of longitude</param>
/// <param name="accuracy">measurement of 1 standard deviation error in metres</param>
/// <param name="TimeStamp_milliseconds">time of measurement</param>
/// <returns>new state</returns>
public void Process(double lat_measurement, double lng_measurement, float accuracy, long TimeStamp_milliseconds) {
if (accuracy < MinAccuracy) accuracy = MinAccuracy;
if (variance < 0) {
// if variance < 0, object is unitialised, so initialise with current values
this.TimeStamp_milliseconds = TimeStamp_milliseconds;
lat=lat_measurement; lng = lng_measurement; variance = accuracy*accuracy;
} else {
// else apply Kalman filter methodology
long TimeInc_milliseconds = TimeStamp_milliseconds - this.TimeStamp_milliseconds;
if (TimeInc_milliseconds > 0) {
// time has moved on, so the uncertainty in the current position increases
variance += TimeInc_milliseconds * Q_metres_per_second * Q_metres_per_second / 1000;
this.TimeStamp_milliseconds = TimeStamp_milliseconds;
// TO DO: USE VELOCITY INFORMATION HERE TO GET A BETTER ESTIMATE OF CURRENT POSITION
}
// Kalman gain matrix K = Covarariance * Inverse(Covariance + MeasurementVariance)
// NB: because K is dimensionless, it doesn't matter that variance has different units to lat and lng
float K = variance / (variance + accuracy * accuracy);
// apply K
lat += K * (lat_measurement - lat);
lng += K * (lng_measurement - lng);
// new Covarariance matrix is (IdentityMatrix - K) * Covarariance
variance = (1 - K) * variance;
}
}
}
What you are looking for is called a Kalman Filter. It's frequently used to smooth navigational data. It is not necessarily trivial, and there is a lot of tuning you can do, but it is a very standard approach and works well. There is a KFilter library available which is a C++ implementation.
My next fallback would be least squares fit. A Kalman filter will smooth the data taking velocities into account, whereas a least squares fit approach will just use positional information. Still, it is definitely simpler to implement and understand. It looks like the GNU Scientific Library may have an implementation of this.
This might come a little late...
I wrote this KalmanLocationManager for Android, which wraps the two most common location providers, Network and GPS, kalman-filters the data, and delivers updates to a LocationListener (like the two 'real' providers).
I use it mostly to "interpolate" between readings - to receive updates (position predictions) every 100 millis for instance (instead of the maximum gps rate of one second), which gives me a better frame rate when animating my position.
Actually, it uses three kalman filters, on for each dimension: latitude, longitude and altitude.
They're independent, anyway.
This makes the matrix math much easier: instead of using one 6x6 state transition matrix, I use 3 different 2x2 matrices. Actually in the code, I don't use matrices at all. Solved all equations and all values are primitives (double).
The source code is working, and there's a demo activity.
Sorry for the lack of javadoc in some places, I'll catch up.
You should not calculate speed from position change per time. GPS may have inaccurate positions, but it has accurate speed (above 5km/h). So use the speed from GPS location stamp.
And further you should not do that with course, although it works most of the times.
GPS positions, as delivered, are already Kalman filtered, you probably cannot improve, in postprocessing usually you have not the same information like the GPS chip.
You can smooth it, but this also introduces errors.
Just make sure that your remove the positions when the device stands still, this removes jumping positions, that some devices/Configurations do not remove.
I usually use the accelerometers. A sudden change of position in a short period implies high acceleration. If this is not reflected in accelerometer telemetry it is almost certainly due to a change in the "best three" satellites used to compute position (to which I refer as GPS teleporting).
When an asset is at rest and hopping about due to GPS teleporting, if you progressively compute the centroid you are effectively intersecting a larger and larger set of shells, improving precision.
To do this when the asset is not at rest you must estimate its likely next position and orientation based on speed, heading and linear and rotational (if you have gyros) acceleration data. This is more or less what the famous K filter does. You can get the whole thing in hardware for about $150 on an AHRS containing everything but the GPS module, and with a jack to connect one. It has its own CPU and Kalman filtering on board; the results are stable and quite good. Inertial guidance is highly resistant to jitter but drifts with time. GPS is prone to jitter but does not drift with time, they were practically made to compensate each other.
One method that uses less math/theory is to sample 2, 5, 7, or 10 data points at a time and determine those which are outliers. A less accurate measure of an outlier than a Kalman Filter is to to use the following algorithm to take all pair wise distances between points and throw out the one that is furthest from the the others. Typically those values are replaced with the value closest to the outlying value you are replacing
For example
Smoothing at five sample points A, B, C, D, E
ATOTAL = SUM of distances AB AC AD AE
BTOTAL = SUM of distances AB BC BD BE
CTOTAL = SUM of distances AC BC CD CE
DTOTAL = SUM of distances DA DB DC DE
ETOTAL = SUM of distances EA EB EC DE
If BTOTAL is largest you would replace point B with D if BD = min { AB, BC, BD, BE }
This smoothing determines outliers and can be augmented by using the midpoint of BD instead of point D to smooth the positional line. Your mileage may vary and more mathematically rigorous solutions exist.
As for least squares fit, here are a couple other things to experiment with:
Just because it's least squares fit doesn't mean that it has to be linear. You can least-squares-fit a quadratic curve to the data, then this would fit a scenario in which the user is accelerating. (Note that by least squares fit I mean using the coordinates as the dependent variable and time as the independent variable.)
You could also try weighting the data points based on reported accuracy. When the accuracy is low weight those data points lower.
Another thing you might want to try is rather than display a single point, if the accuracy is low display a circle or something indicating the range in which the user could be based on the reported accuracy. (This is what the iPhone's built-in Google Maps application does.)
Going back to the Kalman Filters ... I found a C implementation for a Kalman filter for GPS data here: http://github.com/lacker/ikalman I haven't tried it out yet, but it seems promising.
You can also use a spline. Feed in the values you have and interpolate points between your known points. Linking this with a least-squares fit, moving average or kalman filter (as mentioned in other answers) gives you the ability to calculate the points inbetween your "known" points.
Being able to interpolate the values between your knowns gives you a nice smooth transition and a /reasonable/ approximation of what data would be present if you had a higher-fidelity. http://en.wikipedia.org/wiki/Spline_interpolation
Different splines have different characteristics. The one's I've seen most commonly used are Akima and Cubic splines.
Another algorithm to consider is the Ramer-Douglas-Peucker line simplification algorithm, it is quite commonly used in the simplification of GPS data. (http://en.wikipedia.org/wiki/Ramer-Douglas-Peucker_algorithm)
Here's a Javascript implementation of #Stochastically's Java implementation for anyone needing it:
class GPSKalmanFilter {
constructor (decay = 3) {
this.decay = decay
this.variance = -1
this.minAccuracy = 1
}
process (lat, lng, accuracy, timestampInMs) {
if (accuracy < this.minAccuracy) accuracy = this.minAccuracy
if (this.variance < 0) {
this.timestampInMs = timestampInMs
this.lat = lat
this.lng = lng
this.variance = accuracy * accuracy
} else {
const timeIncMs = timestampInMs - this.timestampInMs
if (timeIncMs > 0) {
this.variance += (timeIncMs * this.decay * this.decay) / 1000
this.timestampInMs = timestampInMs
}
const _k = this.variance / (this.variance + (accuracy * accuracy))
this.lat += _k * (lat - this.lat)
this.lng += _k * (lng - this.lng)
this.variance = (1 - _k) * this.variance
}
return [this.lng, this.lat]
}
}
Usage example:
const kalmanFilter = new GPSKalmanFilter()
const updatedCoords = []
for (let index = 0; index < coords.length; index++) {
const { lat, lng, accuracy, timestampInMs } = coords[index]
updatedCoords[index] = kalmanFilter.process(lat, lng, accuracy, timestampInMs)
}
Mapped to CoffeeScript if anyones interested. **edit -> sorry using backbone too, but you get the idea.
Modified slightly to accept a beacon with attribs
{latitude: item.lat,longitude: item.lng,date: new
Date(item.effective_at),accuracy: item.gps_accuracy}
MIN_ACCURACY = 1
# mapped from http://stackoverflow.com/questions/1134579/smooth-gps-data
class v.Map.BeaconFilter
constructor: ->
_.extend(this, Backbone.Events)
process: (decay,beacon) ->
accuracy = Math.max beacon.accuracy, MIN_ACCURACY
unless #variance?
# if variance nil, inititalise some values
#variance = accuracy * accuracy
#timestamp_ms = beacon.date.getTime();
#lat = beacon.latitude
#lng = beacon.longitude
else
#timestamp_ms = beacon.date.getTime() - #timestamp_ms
if #timestamp_ms > 0
# time has moved on, so the uncertainty in the current position increases
#variance += #timestamp_ms * decay * decay / 1000;
#timestamp_ms = beacon.date.getTime();
# Kalman gain matrix K = Covarariance * Inverse(Covariance + MeasurementVariance)
# NB: because K is dimensionless, it doesn't matter that variance has different units to lat and lng
_k = #variance / (#variance + accuracy * accuracy)
#lat = _k * (beacon.latitude - #lat)
#lng = _k * (beacon.longitude - #lng)
#variance = (1 - _k) * #variance
[#lat,#lng]
I have transformed the Java code from #Stochastically to Kotlin
class KalmanLatLong
{
private val MinAccuracy: Float = 1f
private var Q_metres_per_second: Float = 0f
private var TimeStamp_milliseconds: Long = 0
private var lat: Double = 0.toDouble()
private var lng: Double = 0.toDouble()
private var variance: Float =
0.toFloat() // P matrix. Negative means object uninitialised. NB: units irrelevant, as long as same units used throughout
fun KalmanLatLong(Q_metres_per_second: Float)
{
this.Q_metres_per_second = Q_metres_per_second
variance = -1f
}
fun get_TimeStamp(): Long { return TimeStamp_milliseconds }
fun get_lat(): Double { return lat }
fun get_lng(): Double { return lng }
fun get_accuracy(): Float { return Math.sqrt(variance.toDouble()).toFloat() }
fun SetState(lat: Double, lng: Double, accuracy: Float, TimeStamp_milliseconds: Long)
{
this.lat = lat
this.lng = lng
variance = accuracy * accuracy
this.TimeStamp_milliseconds = TimeStamp_milliseconds
}
/// <summary>
/// Kalman filter processing for lattitude and longitude
/// https://stackoverflow.com/questions/1134579/smooth-gps-data/15657798#15657798
/// </summary>
/// <param name="lat_measurement_degrees">new measurement of lattidude</param>
/// <param name="lng_measurement">new measurement of longitude</param>
/// <param name="accuracy">measurement of 1 standard deviation error in metres</param>
/// <param name="TimeStamp_milliseconds">time of measurement</param>
/// <returns>new state</returns>
fun Process(lat_measurement: Double, lng_measurement: Double, accuracy: Float, TimeStamp_milliseconds: Long)
{
var accuracy = accuracy
if (accuracy < MinAccuracy) accuracy = MinAccuracy
if (variance < 0)
{
// if variance < 0, object is unitialised, so initialise with current values
this.TimeStamp_milliseconds = TimeStamp_milliseconds
lat = lat_measurement
lng = lng_measurement
variance = accuracy * accuracy
}
else
{
// else apply Kalman filter methodology
val TimeInc_milliseconds = TimeStamp_milliseconds - this.TimeStamp_milliseconds
if (TimeInc_milliseconds > 0)
{
// time has moved on, so the uncertainty in the current position increases
variance += TimeInc_milliseconds.toFloat() * Q_metres_per_second * Q_metres_per_second / 1000
this.TimeStamp_milliseconds = TimeStamp_milliseconds
// TO DO: USE VELOCITY INFORMATION HERE TO GET A BETTER ESTIMATE OF CURRENT POSITION
}
// Kalman gain matrix K = Covarariance * Inverse(Covariance + MeasurementVariance)
// NB: because K is dimensionless, it doesn't matter that variance has different units to lat and lng
val K = variance / (variance + accuracy * accuracy)
// apply K
lat += K * (lat_measurement - lat)
lng += K * (lng_measurement - lng)
// new Covarariance matrix is (IdentityMatrix - K) * Covarariance
variance = (1 - K) * variance
}
}
}