I wish to drop the columns in a SAS dataset which has a sum less than a particular value. Consider the case below.
Column_A Pred_1 Pred_2 Pred_3 Pred_4 Pred_5
A 1 1 0 1 0
A 0 1 0 1 0
A 0 1 0 1 0
A 0 1 0 1 1
A 0 1 0 0 1
Let us assume that our threshold is 4, so I wish to drop predictors having sum of active observations less than 4, so the output would look like
Column_A Pred_2 Pred_4
A 1 1
A 1 1
A 1 1
A 1 1
A 1 0
Currently I am using a very inefficient method of using multiple transposes to drop the predictors. There are multiple datasets with records > 30,000 so transpose approach is taking time. Would appreciate if anyone has a more efficient solution!
Thanks!
Seems like you could do:
Run PROC MEANS or similar proc to get the sums
Create a macro variable that contains all variables in the dataset with sum < threshhold
Drop those variables
Then no TRANSPOSE or whatever, just regular plain old summarization and drops. Note you should use ODS OUTPUT not the OUT= in PROC MEANS, or else you will have to PROC TRANSPOSE the normal PROC MEANS OUT= dataset.
An example using a trivial dataset:
data have;
array x[20];
do _n_ = 1 to 20;
do _i = 1 to dim(x);
x[_i] = rand('Uniform') < 0.2;
end;
output;
end;
run;
ods output summary=have_sums; *how we get our output;
ods html select none; *stop it from going to results window;
proc means data=have stackodsoutput sum; *stackodsoutput is 9.3+ I believe;
var x1-x20;
run;
ods html select all; *reenable normal html output;
%let threshhold=4; *your threshhold value;
proc sql;
select variable
into :droplist_threshhold separated by ' '
from have_sums
where sum lt &threshhold; *checking if sum is over threshhold;
quit;
data want;
set have;
drop &droplist_threshhold.; *and now, drop them!;
run;
Just use PROC SUMMARY to get the sums. You can then use a data step to generate the list of variable names to drop.
%let threshhold=4;
%let varlist= pred_1 - pred_5;
proc summary data=have ;
var &varlist ;
output out=sum sum= ;
run;
data _null_;
set sum ;
array x &varlist ;
length droplist $500 ;
do i=1 to dim(x);
if x(i) < &threshhold then droplist=catx(' ',droplist,vname(x(i)));
end;
call symputx('droplist',droplist);
run;
You can then use the macro variable to generate a DROP statement or a DROP= dataset option.
drop &droplist;
Related
I have a dataset containing a variable X, made up of multiple numbers separated by a comma. The number of item is different among rows. I created a count words. Now I would like to see the numbers in different columns.
Here the example:
X Num_of_X Var1 Var2 Var3 Var4 ... Varn
3,10,165 3 3 10 165
1 1 1
15,100 2 15 100
10,52,63,90 4 10 52 63 90
I tried this way:
%let max_num_X=max(num_of_x);
data have;
set have;
length var1-var&max_num_X $10.;
array Var(&max_num_X) $;
do i=1 to &max_num_X;
Var[i]=scan(X,i,',');
output;
end;
run;
Could you help me?
Thank you
Do something like this
data have;
input X :$20.;
datalines;
3,10,165
1
15,100
10,52,63,90
;
data long;
set have;
n = _N_;
do i = 1 to countw(X, ',');
xx = scan(X, i, ',');
output;
end;
run;
proc transpose data = long out = want(drop=_:) prefix=Var;
by n;
id i;
var xx;
run;
you could use a macro to find the maximum number and create the variables:
%macro create_vars();
proc sql noprint; select max(countw(X)) into :max_num_X from have; quit;
data have; set have;
%do i = 1 %to &max_num_X.; Var&i. = scan(X,&i.,','); %end;
run;
%mend;
%create_vars();
I have a dataframe
ID value1
1 12
2 345
3 342
i have a second dataframe
value2
3823
how do I get the following result?
ID value1 value2
1 12 3823
2 345 3823
3 342 3823
any joins I have done have given me
ID value1 value2
1 12 .
2 345 .
3 342 .
. . 3823
No need for joins or helper variables:
data have;
do i = 1 to 3;
output;
end;
run;
data lookup;
j = 1;
run;
data want;
set have;
if _n_ = 1 then set lookup;
run;
Without the if _n_ = 1, the data step stops after one iteration when it tries to read a second row from the lookup dataset and finds that there are no rows remaining.
N.B. this requires that the have dataset doesn't already contain a variable with the same name as the variable(s) attached from the lookup dataset.
By far the easiest way to do this is to utilize PROC SQL and defining the condition 1=1, which is always true for each comparison:
data first;
input ID value1 ##;
cards;
1 12 2 345 3 342
run;
data second;
input value2 ;
cards;
3823
run;
proc sql;
create table wanted as
select * from first
left join second
on 1 =1
;quit;
Edit: As far as I know, there isn't direct way to merge datasets by each row, but you can do the following trick:
Add variable Help:
data second_trick;
set second;
help=1;
run;
data first_trick;
set first;
help=1;
run;
Then we just perform the merge by the static variable:
data wanted_trick;
merge first_trick(in=a) second_trick;
by help;
if a; /*Left join, just to be sure.*/
run;
now this only works if you want to add single static value. Don't try to use it your Second set has more rows.
For more on Merges and joins see: https://support.sas.com/resources/papers/proceedings/proceedings/sugi30/249-30.pdf
I want to assign a new variable from existing highest n variable.
So if we have a table that has increasing number of columns -
data have;
input uid $ var1 $ var2 $ var3 $;
datalines;
1111 1 0 1
2222 1 0 0
3333 0 0 0
4444 1 1 1
5555 0 0 0
6666 1 1 1
;
I want derive the variable var3 as final_code.
data want;
set have;
final_code = max(of var1-var3);
run;
Above doesn't make sense here as I want only var3 column to remain. Similarly, if var4 is there, I wish to have var4 only.
Does somebody want to help me here ?
If I understand you right, you don't want max of the values but the value from the highest-numbered-variable.
Lots of ways to do this, which way depends on how the variables are named. Here's the easiest, if they're actually named as you say.
data want;
set have;
array var[*] var:;
final_code = var[dim(var)];
run;
Here we make an array out of var: and then choose the last element in the array using dim (to say the size of the array).
I think this is what you are looking for is:
%let n=3
data want;
set have;
var&n = max(of var1-var&n);
drop var1-var%eval(&n-1);
run;
The macro variable &n holds the value of n. This acts as a substitution during the compilation phase of the code.
The DROP statement tells the data step to drop those variable.
The %eval() macro function performs integer math on macro values. So we are dropping 1 through N-1.
I have 90 variables in the data, I want to do the following in SAS.
Here is my SAS code:
data test;
length id class sex $ 30;
input id $ 1 class $ 4-6 sex $ 8 survial $ 10;
cards;
1 3rd F Y
2 2nd F Y
3 2nd F N
4 1st M N
5 3rd F N
6 2nd M Y
;
run;
data items2;
set test;
length tid 8;
length item $8;
tid = _n_;
item = class;
output;
item = sex;
output;
item = survial;
output;
keep tid item;
run;
What if I have 90 variables to input the data like this? There should be a very long list. I want to simplify it.
You could use an ARRAY or alternately a PROC TRANSPOSE.
The following is untested, because you haven't provided an exxample of your input dataset.
DATA ITEMS;
ARRAY VARS {*} VAR1-VAR90;
SET REPLACE;
DO I = LBOUND(VARS) TO HBOUUND(VARS);
ITEM = VARS{I};
OUTPUT;
END;
RUN;
OR
PROC TRANSPOSE DATA = TEST OUT = WANT;
BY ID;
VAR CLASS -- SURVIAL;
RUN;
In the future it would be best is you could supply your input and desired output.
I don't seem to be able to add another comment to the above answer, as such I am adding one here.
You need to extend the VAR statement to include all variables that you want transposed.
CLASS -- SURVIAL means all variables between CLASS and SURVIVAL inclusive.
Post your code and the error so that I can help you better.
I have two tables:
data a;
input a b c;
datalines;
1 2 .
;
run;
data b;
input a b c;
datalines;
1 . 3
;
run;
The result I want from these tables is replacing the missings by the values that are not missing:
a b c
-----
1 2 3
How can I do it with mostly less piece of code?
EDIT:
I wrote the code and it works, but may be there is more simple code for this.
%macro x;
%macro dummy; %mend dummy;
data _null_;
set x end=Last;
call symputx("name"||left(_N_),name);
if Last then call symputx("num",_n_);
run;
data c;
set a b;
run;
data c;
set c;
%do i=1 %to #
x&i=lag(&&name&i);
%end;
n=_n_;
run;
data c1 (drop= n %do i=1 %to # x&i %end;);
set c (where=(n=2));
%do i=1 %to #
if missing(&&name&i) and not missing(x&i) then &&name&i=x&i;
%end;
run;
%mend;
%x;
If the values are consistent, ie, you never have:
1 2 3
1 3 .
and/or are happy for them to be overwritten, then UPDATE is excellent for this.
data c;
update a b;
by a;
run;
UPDATE will only replace values with non-missing values, so . gets replaced by 3 but 2 is not replaced by .. Again assuming a is the ID variable as Gordon assumes.
You also can easily do this:
data c;
set a b;
by a;
retain b_1 c_1;
if first.a then do; *save the first b and c;
b_1=b;
c_1=c;
end;
else do; *now fill in missings using COALESCE which only replaces if missing;
b_1=coalesce(b_1,b); *use coalescec if this is a char var;
c_1=coalesce(c_1,c); *same;
end;
if last.a then output; *output last row;
drop b c;
rename
b_1=b
c_1=c
;
run;
This makes sure you keep the first instance of any particular value, if they may be different (the opposite of update which keeps the last instance, and different from the SQL solution which takes MAX specifically). All three should give the same result if you have only identical values. Data step options should be a bit faster than the SQL option, I expect, as they're both one pass solutions with no matching required (though it probably doesn't matter).
Using proc SQL, you can do this with aggregation:
proc sql;
select max(a) as a, max(b) as b, max(c) as c
from (select a, b, c from a union all
select a, b, c from b
) x;
If, as I suspect, the first column is an id for matching the two tables, you should instead do:
proc sql;
select coalesce(a.a, b.a), coalesce(a.b, b.b) as b, coalesce(a.c, b.c) as c
from a full join
b
on a.a = b.a;
I'm going to post how to do your approach with some details here: I wouldn't consider this the best approach for this, but you can perhaps learn more easily by starting with what you have, and it's not a horrible approach certainly - just not optimal.
Starting:
%macro x;
%macro dummy; %mend dummy;
data _null_;
set x end=Last;
call symputx("name"||left(_N_),name);
if Last then call symputx("num",_n_);
run;
data c;
set a b;
run;
data c; *NOTE 1;
set c;
%do i=1 %to #
x&i=lag(&&name&i); *NOTE 2;
%end;
n=_n_;
run;
data c1 (drop= n %do i=1 %to # x&i %end;); *NOTE 3;
set c (where=(n=2));
%do i=1 %to #
if missing(&&name&i) and not missing(x&i) then &&name&i=x&i;
%end;
run;
%mend;
%x;
Ending:
*You can still do the first datastep to figure out the dimensions of the arrays,
if you want, use &num instead of the 3s hardcoded in there (but do not need &name list).;
data c;
set a(in=in_a) b(in=in_b);
array x[3] _temporary_; *NOTE 4;
array var[3] a b c;
do i = 1 to dim(x); *NOTE 5;
x[i] = lag(vars[i]);
end;
if in_b then do; *NOTE 6;
do i=1 to dim(x);
if missing(vars[i]) then vars[i]=x[i]; *NOTE 7;
end;
output;
end;
run;
Notes:
NOTE 1: You can combine the two c datasteps here with no difference at all. In general have as few data steps as you can, as they're slow - this is a difference from R or similar which use in memory processing, in SAS you use disk processing which is nice for ability to do 200GB of data but not as fast for multiple steps like this - so make fewer steps.
NOTE 2: This is basically a macro implementation of an array. SAS datastep has an array already! Use it.
NOTE 3: You don't need to do the drop like that. drop=n x: works fine as long as none of your real variables start with x (and if they do, use an _ before all of your dummy variables and it will be the same). : is a wild card for 'starts with'.
NOTE 4: Here is the array implementation of your x array. I use temporary because that means the variables will be dropped automatically for you.
NOTE 5: Here we do the lags. I don't like using lag for this where retain does a better job of the same thing, but it works fine.
NOTE 6: This if in_b is like your if last from your step. This identifies records in b only - if there's only one then it will only happen once.
NOTE 7: This is doing the replacement for missing. COALESCE \ COALESCEC would also work for this purpose (though in some cases you might need to use this method if you are unsure of the variable type). No reason to check if not missing unless you're using special missings in some fashion - no harm in replacing . with ..