GAMS - writing output to excel file using GDX - gams-math

I am in the midst of exporting data from GAMS to excel. As such, I am using the GDX utilities.
Say that I have two variables, x and y, defined over the set i. The set i={1,2,3,4}. As such, x(i) and y(i) have four data points each.
I first unload the output to a GDX file:
execute_unload "output_ex.gdx"
x
y
;
Thereafter, I write the variable levels to Excel file. Currently, I write it as
execute 'gdxxrw.exe Output.gdx par=XXX rng=XXX !a1'
execute 'gdxxrw.exe output_ex.gdx par=x rng=x!a1'
execute 'gdxxrw.exe output_ex.gdx par=y rng=y!a1'
This creates an excel file, with 2 sheets, x and y, each containing the 4 values of x and y, starting in cell a1, with the set indices (i=1..4) in the first column. I would like to have instead, a single sheet. This can be achieved, by say
execute 'gdxxrw.exe Output.gdx par=XXX rng=XXX !a1'
execute 'gdxxrw.exe output_ex.gdx par=x rng=x!a1'
execute 'gdxxrw.exe output_ex.gdx par=y rng=x!a3'
Here, I obtain x and y in a single sheet, entitled x. However, the issue is that the indices repeat, i.e. the index is created in duplicate on cell a3, so I have a separate row of the same indices for x and y. Is there any way of having the index only in the first row, and the subsequent rows containing just the data, identified by variable names? This would make it aesthetically more pleasing and easier to manipulate.
Many thanks!

You could combine the two parameters in one in GAMS already before exporting it, like this:
Set i / i1*i4 /
header / x,y /;
Parameter
x(i)
y(i)
xy(i, header);
* Fill with random data
x(i) = uniformInt(2,6);
y(i) = uniformInt(4,7);
* Combine in one parameter
xy(i,'x') = x(i);
xy(i,'y') = y(i);
execute_unload "output_ex.gdx" xy;
execute 'gdxxrw.exe output_ex.gdx par=xy rng=xy!a1'

Related

SAS Server behaving strangely

I'm playing around with SAS (version: 7.11 HF2), I've a dataset which has columns A and B, variable A is decimal. When I run the below code, strangely I get a . (dot) in the first row of output.
Input data:
a, b
2.4, 1
1.2, 2
3.6, 3
Code:
data test;
c = a;
set abcd.test_data;
run;
Output data:
c, a, b
., 2.4, 1
2.4, 1.2, 2
1.2, 3.6, 3
3.6, ,
Strange things:
Derived variable is always generated on the right side, this one is being generated on left.
. (dot) is coming and the values are shifting by a row in the derived column.
Any help?
Looks like it did want you asked it to do.
On the first iteration of the data step it will set C to the value of A. The value of A is missing since you have not yet given it any value. Then the SET statement will read the first observation from your input dataset. Since there is no explicit OUTPUT statement the observation is written when the iteration reaches the end.
On the rest of the iterations of the data step the value that A will have when it is assigned to C will be the value as last read from the input dataset. Any variable that is part of an input dataset is "retained", which really just means it is not set to missing when a new iteration starts.
If the goal was to create C with the previous value of A you could have created the same output by using the LAG() function.
data test;
set abcd.test_data;
c=lag(a);
run;
Your set statement is after your variable assignment statement. SAS is first trying to assign the value of a to c, which has not yet been read. Place your set statement first, then do variable manipulation.
data test;
set abcd.test_data;
c = a;
run;
Nothing strange here, just put the SET statement before.
Datastep processing consists of 2 phases.
Compilation Phase
Execution Phase
During compilation phase, each of the statements within the data step are scanned for syntax errors.
During execution phase, a dataset's data portion is created.
It initializes variables to missing and finally executes other statements in the order determined by their location in the data step.
In your case, the set statement comes after the assignment of c. At that time a and b are set to missing, hence giving a missing value for c. Finally, the SET statement will be executed and that is why you end up with a value for both a and b on the first line.
data test;
set abcd.test_data;
c = a;
run;
Note that the first variable in your dataset is c, because this is the first stated in your code.

Performing calculations on multiple columns in dataframe and create new columns

I'm trying to perform calculations based on the entries in a pandas dataframe. The dataframe looks something like this:
and it contains 1466 rows. I'll have to run similar calculations on other dfs with more rows later.
What I'm trying to do, is calculate something like mag='(U-V)/('R-I)' (but ignoring any values that are -999), put that in a new column, and then z_pred=10**((mag-c)m) in a new column (mag, c and m are just hard-coded variables). I have other columns I need to add too, but I figure that'll just be an extension of the same method.
I started out by trying
for i in range(1):
current = qso[:]
mag = (U-V)/(R-I)
name = current['NED']
z_pred = 10**((mag - c)/m)
z_meas = current['z']
but I got either a Series for z, which I couldn't operate on, or various type errors when I tried to print the values or write them to a file.
I found this question which gave me a start, but I can't see how to apply it to multiple calculations, as in my situation.
How can I achieve this?
Conditionally adding calculated columns row wise are usually performed with numpy's np.where;
df['mag'] = np.where(~df[['U', 'V', 'R', 'I']].eq(-999).any(1), (df.U - df.V) / (df.R - df.I), -999)
Note; assuming here that when any of the columns contain '-999' it will not be calculated and a '-999' is returned.

Output to a text file

I need to output lots of different datasets to different text files. The datasets share some common variables that need to be output but also have quite a lot of different ones. I have loaded these different ones into a macro variable separated by blanks so that I can macroize this.
So I created a macro which loops over the datasets and outputs each into a different text file.
For this purpose, I used a put statement inside a data step. The PUT statement looks like this:
PUT (all the common variables shared by all the datasets), (macro variable containing all the dataset-specific variables);
E.g.:
%MACRO OUTPUT();
%DO N=1 %TO &TABLES_COUNT;
DATA _NULL_;
SET &&TABLE&N;
FILE 'PATH/&&TABLE&N..txt';
PUT a b c d "&vars";
RUN;
%END;
%MEND OUTPUT;
Where &vars is the macro variable containing all the variables needed for outputting for a dataset in the current loop.
Which gets resolved, for example, to:
PUT a b c d special1 special2 special5 ... special329;
Now the problem is, the quoted string can only be 262 characters long. And some of my datasets I am trying to output have so many variables to be output that this macro variable which is a quoted string and holds all those variables will be much longer than that. Is there any other way how I can do this?
Do not include quotes around the list of variable names.
put a b c d &vars ;
There should not be any limit to the number of variables you can output, but if the length of the output line gets too long SAS will wrap to a new line. The default line length is currently 32,767 (but older versions of SAS use 256 as the default line length). You can actually set that much higher if you want. So you could use 1,000,000 for example. The upper limit probably depends on your operating system.
FILE "PATH/&&TABLE&N..txt" lrecl=1000000 ;
If you just want to make sure that the common variables appear at the front (that is you are not excluding any of the variables) then perhaps you don't need the list of variables for each table at all.
DATA _NULL_;
retain a b c d ;
SET &&TABLE&N;
FILE "&PATH/&&TABLE&N..txt" lrecl=1000000;
put (_all_) (+0) ;
RUN;
I would tackle this but having 1 put statement per variable. Use the # modifier so that you don't get a new line.
For example:
data test;
a=1;
b=2;
c=3;
output;
output;
run;
data _null_;
set test;
put a #;
put b #;
put c #;
put;
run;
Outputs this to the log:
800 data _null_;
801 set test;
802 put a #;
803 put b #;
804 put c #;
805 put;
806 run;
1 2 3
1 2 3
NOTE: There were 2 observations read from the data set WORK.TEST.
NOTE: DATA statement used (Total process time):
real time 0.07 seconds
cpu time 0.03 seconds
So modify your macro to loop through the two sets of values using this syntax.
Not sure why you're talking about quoted strings: you would not quote the &vars argument.
put a b c d &vars;
not
put a b c d "&vars";
There's a limit there, but it's much higher (64k).
That said, I would do this in a data driven fashion with CALL EXECUTE. This is pretty simple and does it all in one step, assuming you can easily determine which datasets to output from the dictionary tables in a WHERE statement. This has a limitation of 32kiB total, though if you're actually going to go over that you can work around it very easily (you can separate out various bits into multiple calls, and even structure the call so that if the callstr hits 32000 long you issue a call execute with it and then continue).
This avoids having to manage a bunch of large macro variables (your &VAR will really be &&VAR&N and will be many large macro variables).
data test;
length vars callstr $32767;
do _n_ = 1 by 1 until (last.memname);
set sashelp.vcolumn;
where memname in ('CLASS','CARS');
by libname memname;
vars = catx(' ',vars,name);
end;
callstr = catx(' ',
'data _null_;',
'set',cats(libname,'.',memname),';',
'file',cats('"c:\temp\',memname,'.txt"'),';',
'put',vars,';',
'run;');
call execute(callstr);
run;

Ti-84 syntax error relating to a single list conversion to a matrix

:N-remainder(dim(L1),N→ dim(L2)
:Fill(23,L2
:augment(L1, L2->L1
:{1,1→dim([A]
:For(x,1,dim(L1)/N
:augment([A],List▶matr(seq(L1(I),I,Nx-N+1,Nx),[B]
:End
I get a syntax error when running this Ti-basic code and I cannot figure out why (happens somewhere when List is being converted to matrix). Basically this code is suppose to take a L1 (add 23 until I dim(L1) is a multiply of N), then create a matrix with N rows and -int(-dim(L1)/n) columns.
Example:
Let N=3 and
L1 = {9,12,15,22,5,9,14,4,9,1,14,7,9,18,12,19}
dim(L1) = 16 which is not a multiply of 3 (18 is so add 23 to L1 twice)
L1 = {9,12,15,22,5,9,14,4,9,1,14,7,9,18,12,19,23,23}
dim(L1) = 18 which is a multiple of 3
Create a 3x6 matrix with Col1 = {9,12,15}, Col2 = {22,5,9}, ..., Col6 = {19,23,23}
http://tibasicdev.wikidot.com/forum/t-1039272/comments/show?from=activities#post-2131820
Read full convo. here
There are at least two issues with your code:
(1) For the augment command both matrices must share the same number of rows. In your program matrix [A] is set to dimension {1,1} (Why?), but the columns you want to append are of different size. So you'll get a "dimension error".
(2) The List▶matr command doesn't return a matrix (actually it doesn't return anything). So you can't use it as second parameter for the augment command. Instead you must run it first and then use something like augment([A],[B])▶[C].

Using CONTAINS with variables sql

Ok so I am trying to reference one variable with another in SQL.
X= a,b,c,d (x is a string variable with a list of things in it)
Y= b ( Y is a string variable that may or may not have a vaue that appears in X)
I tried this:
Case when Y in (X) then 1 else 0 end as aa
But it doesnt work since it looks for exact matches between X and Y
also tried this:
where contains(X,#Y)
but i cant create Y globally since it is a variable that changes in each row of the table.( x also changes)
A solution in SAS would also be useful.
Thanks
Maybe like will help
select
*
from
t
where
X like ('%'+Y+'%')
or
select
case when (X like ('%'+Y+'%')) then 1 else 0 end
from
t
SQLFiddle example
In SAS I would use the INDEX function, either in a data step or proc sql. This returns the position within the string in which it finds the character(s), or zero if there is no match. Therefore a test if the value returned is greater than zero will result in a binary 1:0 output. You need to use the compress function with the variable containing the search characters as SAS pads the value with blanks.
Data step solution :
aa=index(x,compress(y))>0;
Proc Sql solution :
index(x,compress(y))>0 as aa