Algorithm for calculating most stable, consecutive values from a database - sql

Question

I have some questions and I'm in need of your input.
Say I have a database table filled with 2000-3000 rows and each row has a value and some identifiers. I am in need of withdrawing ~100 consecutive rows with the most stable values (lowest spread). It's okay with a few jumper values if you can exclude them.
How would you do this and what algorithm would you use?
I'm currently using SAS Enterprise Guide for my DB which runs on Oracle. I don't really know that much of the generic SAS language but I don't know what other language I could use for this? Some scripting language? I have limited programming knowledge but this task seems pretty easy, correct?
The algorithms I've been thinking of is:
Select 100 consecutive rows and calculate standard deviation. Increment select statement by 1 and calculate standard deviation again. Loop trough the whole table.
Export the rows with the lowest standard deviation
Same as 1, but calculate variance instead of standard deviation (basically the same thing). When the whole table has been looped, do it again but exclude 1 row which has the highest value from avg. Repeat process until 5 jumpers has been excluded and compare the results.
Pros and cons compared to method 1?
Questions:
Best & easiest method?
Prefered language? Possible in SAS?
Do you have any other method you would recommend?
Thanks in advance
/Niklas

Answer 1

The below code will do what you are asking. It is just using some sample data and only calcs it for 10 observations (rather than 100). I'll leave it to you to adapt as required.
Create some sample data. available to all sas installations:
data xx;
set sashelp.stocks;
where stock = 'IBM';
obs = _n_;
run;
Create row numbers and sort it descending. Makes it easier to calc stddev:
proc sort data=xx;
by descending obs;
run;
Use an array to keep the subsequent 10 obs for every row. Calculate the stddev for each row using the array (except for the last 10 rows. Remember we are working backwards through the data.
data calcs;
set xx;
array a[10] arr1-arr10;
retain arr1-arr10 .;
do tmp=10 to 2 by -1;
a[tmp] = a[tmp-1];
end;
a[1] = close;
if _n_ ge 10 then do;
std = std(of arr1-arr10);
end;
run;
Find which obs (ie. row) had the lowest stddev calc. Save it to a macro var.
proc sql noprint;
select obs into :start_row
from calcs
having std = min(std)
;
quit;
Select the 10 observations from the sample data that were involved in calcing the lowest stddev.
proc sql noprint;
create table final as
select *
from xx
where obs between &start_row and %eval(&start_row+10)
order by obs
;
quit;

Answer 2

An addition to Robert's solution but with part 2 included as well, creating a second array and then looping through and removing the top 5 values. You'll still the last parts of Roberts solution to extract the row with the minimum standard deviation and then the corresponding attached rows. You didn't specify how you wanted to deal with the variances that have the max removed so they are left in the dataset.
data want;
*set arrays for looping;
/*used to calculate the std*/
array p{0:9} _temporary_;
/*used to copy the array over to reduce variables*/
array ps(1:10) _temporary_;
/*used to store the var with 5 max values removed*/
array s{1:5} var1-var5;
set sample;
p{mod(_n_,10)} = open;
if _n_ ge 10 then std = std(of p{*});
*remove max values to calculate variance;
if _n_ ge 10 then do;
*copy array over to remove values;
do i=1 to 10;
ps(i)=p(i-1);
end;
do i=1 to 5;
index=whichn(max(of ps(*)), of ps(*));
ps(index)=.;
s(i)=var(of ps(*));
end;
end;
run;