loop to conditionally export tables SAS (Large number of Variables) - sql

Question

So I am trying to break up a large dataset (70,000 obs with 1,790 variables) based on a specific variable grouping. Excel or CSV is the ideal format to export to, but there is a limitation on variable numbers (260 or something). Any ideas how I can do this in SAS (or R / SQL otherwise)?
I know the macro works, I have used it before. The error message reads the limit on variables has been reached.

Answer 1

There is certainly a limit on creating an Excel file, but not a CSV file. Here is an example using a dummy SAS data set:
data a;
array x(*) x1-x1790;
do j=1 to 5;
do i=1 to dim(x);
x(i) = ranuni(0);
end;
output;
end;
run;
proc export data=a
outfile="c:\temp\tempfile.csv"
dbms=CSV
replace;
run;
And here is the relevant log:
NOTE: The file 'c:\temp\tempfile.csv' is:
Filename=c:\temp\tempfile.csv,
RECFM=V,LRECL=32767,File Size (bytes)=0,
Last Modified=23Jan2013:15:27:13,
Create Time=23Jan2013:15:27:13
NOTE: 6 records were written to the file 'c:\temp\tempfile.csv'.
The minimum record length was 9636.
The maximum record length was 23087.
NOTE: There were 5 observations read from the data set WORK.A.
NOTE: DATA statement used (Total process time):
real time 0.26 seconds
cpu time 0.09 seconds
5 records created in c:\temp\tempfile.csv from A.
NOTE: "c:\temp\tempfile.csv" file was successfully created.
NOTE: PROCEDURE EXPORT used (Total process time):
real time 2.04 seconds
cpu time 0.26 seconds
Note the first row contains column headers.
UPDATE: If you have a recent version of SAS (9.3 TS1M1 or later) you can create an Office 2010 Excel spreadsheet, which has a maximum of 1,048,576 rows by 16,384 columns. In that case, you would use DBMS=XLSX.

Answer 2

Bob's answer is good if you are okay with XLSX or a CSV. If you do want to make a .xls excel file (255 column limit), or don't have 9.3TS1M1, it's fairly easy to do that. How exactly depends on how you want to specify the columns that go into each file.
Say you just want each 255 columns into a separate file, and two files split at the midpoint (35000 records into file A, 35001-end into file B, per set of variables). You would do something like this:
options mprint symbolgen;
data test;
array xs x1-x1700;
do id = 1 to 70000;
do _t = 1 to dim(xs);
xs[_t]=ranuni(7);
end;
output;
end;
run;
%macro export_file(varstart=,varend=,varnumstart=0,varnumend=0,recstart=1,recend=0,keeplist=,dset=, libname=WORK, outfile=,sheet="sheet1");
%if &varnumstart ne 0 %then %do;
proc sql noprint;
select name into :varstart from dictionary.columns
where libname=upcase("&libname.") and memname=upcase("&dset.") and varnum=&varnumstart.;
select name into :varend from dictionary.columns
where libname=upcase("&libname.") and memname=upcase("&dset.") and varnum=&varnumend.;
quit;
%end;
%if &varstart=%str() or &varend=%str() %then %do;
%put "ERROR: MISSING PARAMETERS. PLEASE CHECK YOUR MACRO CALL AND RERUN. MUST HAVE VARSTART AND VAREND OR VARNUMSTART AND VARNUMEND.";
%abort;
%end;
data _for_Export/view=_for_export;
set &libname..&dset;
keep &varstart.--&varend.
%if &keeplist ne %str() %then %do;
&keeplist
%end;
;
if _N_ ge &recstart.;
%if &recend ne 0 %then %do;
if _N_ le &recend.;
%end;
run;
proc export data=_for_export file=&outfile. dbms=excel replace;
sheet=&sheet.;
run;
proc datasets nolist noprint lib=work;
delete _for_export/memtype=view;
quit;
%mend export_file;
%export_file(varnumstart=1,varnumend=250, keeplist=id,recstart=1,recend=35000,dset=test,outfile="c:\temp\test.xls",sheet="sheet1");
%export_file(varnumstart=1,varnumend=250, keeplist=id,recstart=35001,recend=99999,dset=test,outfile="c:\temp\test.xls",sheet="sheet2");
%export_file(varnumstart=251,varnumend=500, keeplist=id,recstart=1,recend=35000,dset=test,outfile="c:\temp\test.xls",sheet="sheet3");
%export_file(varnumstart=251,varnumend=500, keeplist=id,recstart=35001,recend=99999,dset=test,outfile="c:\temp\test.xls",sheet="sheet4");
Mine fails when I try to export sheet4, not sure if there's some limit to the total size of an .xls file, but you can easily modify this to create separate files. This wouldn't work if you needed to specify specific variable names that are nonconsecutive for each separate file, but you could fairly easily modify the SQL code that pulls from dictionary.columns to instead pull from a table you create that holds the variable names you want in each file.