We have a teradata database that is filled with BIGINT datatypes. SAS is the primary analytic engine for the organization. The SAS access library chokes on BIGINT and therefore forces all code to cast variables into a integer or decimal before returning them to SAS. Example:
proc sql;
connect to database (blah blah);
create table sas_dataset as
select * from connection to database(
select
cast(bigint_var as integer),
varchar_var,
char_var,
from some_table
);
disconnect from database;
quit;
Does anyone know a way to fix this? Possibly a macro for casting variables or SAS access fix? Keep in mind that there are literally thousands of variables in this database that are bigint and making changes to the database is out of the question.
If you can't fix it on the SAS side, you can always create a set of views on the Teradata side to do the casting. Then have SAS access the views:
create view sas_cast_db.some_table as
select col1, col2, cast(bigint_var as decimal(18)), col3
from real_db.some_table;
Since you have lots of tables, it may be possible to write some SQL to auto-generate these from the data dictionary.
The issue is that SAS can only handle a maximum of 15 digits for a BIGINT data type.
SAS have provided a few work-arounds for SAS 9.2 (one mentioned above) for this issue. You can also get your SAS Platform Admin to arrange for the SAS server to be updated so that it truncates the BIGINT fields to 15 digits (obvious caveats apply), or update your LIBNAME statement to set all BIGINTs to be cast automatically.
http://support.sas.com/kb/39/831.html
Sample code
data temp1;
set mylib.bigclass (dbsastype= (id='char(20)'));
run;
Related
The database setup at my organisation is SQL tables copied onto our SAS server. The SQL tables were setup to run pre-programmed SQL queries, now SAS is the tool used. This however creates an issue with some tables having variables that are too long for SAS, but work in SQL. The label for the source variable is correct and not shortened.
The source table (in SQL Server) names:
Consolidated_Arrears_Vs_Portfolio_Balance_Ltd
Consolidated_Arrears_Vs_Portfolio_Balance_Pure
In SAS:
Consolidated_Arrears_Vs_Portfoli
Consolidated_Arrears_Vs_Portfoli
SAS Labels:
Consolidated_Arrears_Vs_Portfolio_Balance_Ltd
Consolidated_Arrears_Vs_Portfolio_Balance_Pure
So, how do I tell the difference in code between these two?
Thanks in advance.
To use the data as native in SAS, one approach would be to write a macro to map the original SQL names (per label) to the corresponding new SAS names. If the original table names got mangled as well you have a lot more issues.
Original SQL
select Abracadabra_Magical_Unity_Formation_SequenceId from AMUF_Master
Replace with
select %nameFor(Abracadabra_Magical_Unity_Formation_SequenceId) from AMUF_Master
The macro %nameFor would either do a dynamic lookup against the tables in the library, or perhaps better, when a static table design, create a fixed mapping table from a one time lookup
* presume SQL data now in libref MIGRATED;
* do once to get the variable metadata that includes LABEL and NAME;
proc sql;
create table static.nameFor as
select * from sashelp.vcolumn
where libnames = 'MIGRATED';
* use as needed;
%macro nameFor(SQL_Name);
%sysfunc(dosubl(select NAME from static.nameFor where LABEL="&SQL_Name"))
%mend;
You could also use the static.nameFor to discover all the SQL names that got changed during migration. Those would be where name ne label.
An automated approach would be to create a search and replace program that makes changes to a copy of the original SQL queries on-hand.
The search and replace would be either
find <long-named column>, replace with %nameFor(<long-named column>) , or
find <long-named column>, replace with <migrated to SAS column name>
The first replacement way adds noise.
The second way loses some of the original queries 'true-flavor'
I read different posts about this problem but it didn't help me with my problem.
I am on a local db (Microsoft SQL Server) and query data on remote db (ORACLE).
In this data, there is a CLOB type.
CLOB type column shows me only 7 correct data the others show me <null>
I tried to CAST(DEQ_COMMENTAIRE_REFUS_IMPORT AS VARCHAR(4000))
I tried to SUBSTRING(DEQ_COMMENTAIRE_REFUS_IMPORT, 4000, 1)
Can you help me, please ?
Thank you
No MSSQL but in my case we were pulling data into MariaDB using the ODBC Connect engine from Oracle.
For CLOBs, we did the following (in outline):
Create PLSQL function get_clob_chunk ( clobin CLOB, chunkno NUMBER) RETURN VARCHAR2.
This will return the the specified nth chunk of 1000 chars for the CLOB.
We found 1,000 worked best with multibyte data. If the data is all plain text single byte that chunks of 4,000 are safe.
Apologies for the absence of actual code, as I'm a bit rushed for time.
Create a Oracle VIEW which calls the get_clob_chunk function to split the CLOB into 1,000 char chunk columns chunk1, chunk2, ... chunkn, CAST as VARCHAR2(1000).
We found that Oracle did not like having more than 16 such columns, so we had to split the views into sets of 16 such columns.
What this means is that you must check what the maximum size of data in the CLOB is so you know how many chunks/views you need. To do this dynamically adds complexity, needless to say.
Create a view in MariaDB querying the view.
Create table/view in MariaDB that joins the chunks up into a single Text column.
Note, in our case, we found that copying Text type columns between MariaDB databases using the ODBC Connect engine was also problematic, and required a similar splitting method.
Frankly, I'd rather use Java/C# for this.
I'm fairly new to SAS and recently we migrated some of our SAS datasets to a SQL Server table but we are still using SAS to do our analysis. I have run into a problem when SAS is trying to bring in the data from the SQL Server table and have SAS check if the srv_edt date is between the SAS dates of dos_beg_dt1 and dos_end_dt1.
When SAS tries to compare the dates I get an error of: ERROR: WHERE clause operator requires compatible variables.
The dos_beg_dt1, dos_end_dt1, and srv_edt (SQL date format) all "appear" in the format of yyyy-mm-dd. When I bring the srv_edt into a SAS table it reads it as a character date. So I've tried changing the format of the dates and then I will get an error like:ERROR: Variable srv_edt has been defined as both character and numeric. I can't seem to find the correct format or function to get SAS to do the comparison to see if the srv_edt (SQL) is between the dos_beg_dt1 and dos_end_dt1 SAS dates.
The code I use is as follows:
libname sql odbc dsn=test schema=dbo;
%let dos_beg_dt1 = %sysfunc(intnx(qtr,&date,-1,beginning),yymmdd10.);
%let dos_end_dt1 = %sysfunc(intnx(qtr,&date,-1,end),yymmdd10.);
data sample;
set sql.table;
where &dos_beg_dt1 <= srv_edt <= &dos_end_dt1;
run;
For reference I am using SAS 9.2 to connect via odbc to SQL Server 2008.
Any help or advice would be greatly appreciated.
SAS stores and uses dats as numeric variables. If you had not identified the column srv_edt as a date column when you migrated the database to SQL server everything would now process correctly.
I assume that currently and into the future you will just store the tables in SQL server and all the processing will be in SAS.
You have a few options.
1/ re-migrate the SAS tables but identify all the date, time and datetime columns as just numeric. They all can be stored as 8 byte floating point. The date variables may also be stored (in SQL Server) is long integers. The code would need a slight change so that the macro variables would be numeric.
%let dos_beg_dt1 = %sysfunc(intnx(qtr,&date,-1,beginning));
%let dos_end_dt1 = %sysfunc(intnx(qtr,&date,-1,end));
2/ keep the date, time, and datetime variables in SQL Server format and change the data type of the column when using the data. (Note the reverse will be necessary on output). SQL Server will present the date variables as strings (character) so that your expression above will need to be -
%let dos_beg_dt1 = %sysfunc(intnx(qtr,&date,-1,beginning));
%let dos_end_dt1 = %sysfunc(intnx(qtr,&date,-1,end));
data sample;
set sql.table;
where &dos_beg_dt1 <= (input(srv_edt, yymmdd10.0)) <= &dos_end_dt1;
That to make sure when using SAS processing the type is numeric which is what the input function will do.
3/ keep the date, time, and datetime variables in SQL Server format and change your working to accommodate that fact. That is comparisions will be using character data and output will need to produce characters. SQL Server will present the date variables as strings (character) so that your expression above will need to be -
%let dos_beg_dt1 = %sysfunc(intnx(qtr,&date,-1,beginning), yymmdd10.);
%let dos_end_dt1 = %sysfunc(intnx(qtr,&date,-1,end), yymmdd10.);
data sample;
set sql.table;
where ("&dos_beg_dt1" <= srv_edt) and
(srv_edt <= "&dos_end_dt1");`
Note double quotes " " required surrounding macro variables as this
comparison is numeric.
If the column srv_edt is showing up in your SAS data set as a character variable, that means it is really a character variable or it's been converted to character by the ODBC driver you are using (possibly because the native data type is not supported by ODBC).
You'd be better off changing this to a PROC SQL pass-thru query if possible. You would need to figure out the native syntax that corresponds to the SAS intnx function (and I cannot help you there). As written, the entire table must be read (because you are using a SAS function). If you use a pass-thru query, SAS will only receive the rows that match the whee clause.
There might be setting in the ODBC driver that control this behavior. I'll add the ODBC and SQL Server tags to your question; you may get more "hits".
SQL Server introduced new date and datetime types in SQL Server 2008 (prior, there was only one type for all date/datetime variables). This usage note suggests that you need to install a new set of SQL Server ODBC drivers for SAS to read the date variables correctly. It suggests this would be installed normally if you have SQL Server 2008 Tools (like SQL Server Management Studio) on the machine that is doing the ODBC connection, but you might have multiple drivers installed and need to ensure you are using the right one.
That said, it is not a bad idea to use pass-through SQL to pull the data across, as that might make it easier to do the pull (as you don't have to worry as much about the ODBC driver). The generalized pass through connection string is
proc sql;
connect to odbc (required="driver=sql server native client 10.0;
Server=server;Trusted_Connection=Yes;DATABASE=database;");
create table X as select * from connection to odbc(... sql server native code here ...);
quit;
From your question it sounds like you're more of a SQL person and can then construct the query yourself; if you are not, either edit the question to include that request (and then either a SQL Server person or myself will answer). You can use SAS macro variables in that query (ie, to pass the current date) as long as you do not enclose them or the query in single quotes.
I am trying to use a Teradata UDF (User Defined Function) in a SAS Implicit SQL which establishes the connection to Teradata using LIBNAME Statement.Assume that the function is called PTY_DECRYPT and is defined in a Database called TEST in Teradata. The Purpose of this function is to decrypt values in a Column of a View in Teradata.
What works is using the UDF in an Explicit Sql .Below I am using the function on a column called SSN_NBR in a view called V_TEST_PERS present in the Database called SAMPLE.
Explcit Sql:
Options debug=DBMS_TIMERS sastrace=',,,d'
sastraceloc=saslog no$stsuffix fullstimer;
Proc Sql;
Connect to TERADATA(User=XXXXX pwd=XXXXX server=XXXXX);
Create Table Final as
select * from connection to teradata
(
Select
sub_id,
SSN_NBR,
TEST.PTY_DECRYPT(SSN_NBR,'T_ssn_test',400,0,0 ) as SSN_NBR_Decrypt
from SAMPLE.V_TEST_PERS
);
disconnect from teradata;
Quit;
But I would like to use the same function in an Implicit SQL but it does not work. Any ideas as to how to make it work in an Implicit Sql with minimum changes to the Implicit SQL?
Implicit Sql
Options debug=DBMS_TIMERS sastrace=',,,d'
sastraceloc=saslog no$stsuffix fullstimer;
Libname Td Teradata User=XXXXX pwd=XXXXX server=XXXXX database=SAMPLE ;
Proc sql;
Create table Final as
select
sub_id,
SSN_NBR,
TEST.PTY_DECRYPT(SSN_NBR,'T_ssn_test',400,0,0 ) as SSN_NBR_Decrypt
from Td.V_TEST_PERS;
Quit;
In your implicit SQL you reference the view with the LIBNAME alias TD, however when you reference the UDF you are not aliasing the TEST database containing the UDF with the LIBNAME alias. Syntactically, you may not be able to do that in SAS. (e.g. TD.TEST.PTY_DECRYPT() - in fact I wouldn't expect this to work)
The UDF may need to be placed in SYSLIB or TD_SYSFNLIB so that it is in a default search path for the database optimizer to find the UDF without being fully qualified. (e.g. TD_WEEK_BEGIN()) Alternatively, the UDF could be placed in database SAMPLE but that likely violates how UDFs are maintained in your environment, as it would in my environment.
Otherwise, the UDF call could be embedded in a view on the database, but then you have other issues to consider with the security of that column if your environment is not granting security on a column level basis to views containing encrypted data elements. (e.g. PHI, PII, etc.) Without a row-column level security mechanism in place to dynamically filter a users ability to see the column you are decrypting in the view putting the UDF into the view isn't going to work.
I asked the same question the SAS Communities Forum and I am glad to say that i did find a Solution to this Problem.
Please see the link below :
https://communities.sas.com/t5/Base-SAS-Programming/Using-a-Teradata-UDF-in-SAS-Implicit-Sql-Pass-Thru/m-p/266850/highlight/false#M52685
I've been attempting to write embedded SQL statements for DB2 that ultimately gets compiled in C.
I couldn't find a tutorial or manual on the embedded SQL syntax for C for reference. One case I would like to do is to insert data into a table. I know most embedded sql statements need the initalizer EXEC SQL, but that's the extent of my knowledge generally. I'm doing this for an assignment and would appreciate if there are more information regarding this or solution.
Example of a statement to query the database:
EXEC SQL SELECT SNAME, AGE into :sname, :sage
FROM ONE.SAILOR
WHERE sid = :sid;
I like to see what statement allows me to INSERT into the database. I've tried something like the following, but it doesn't work.
EXEC SQL INSERT ....
See IBM's Embedded SQL manual.
Embedded SQL is largely the same no matter what the host language is.
The four dots aren't syntactically valid :-D
The reliable way is the same as with any other INSERT statement: list the columns and the values.
EXEC SQL INSERT INTO SomeTable(Col1, Col2, Col3) VALUES(:hv1, :hv2, :hv3);
Here, the :hv1, :hv2 and :hv3 represent three host variables of types appropriate to the columns in the table. Note that the table could contain other columns than these three as long as those columns have a default specified or accept NULL (which is really just a default default in this case). The unreliable way does not list the columns:
EXEC SQL INSERT INTO SomeTable VALUES(:hv1, :hv2, :hv3);
Now you are dependent on getting the sequence right, and you must provide a value for each column -- there cannot be extra columns in SomeTable.
I just started using sqllite. Besides the good documentation for C++, SQLlist might be a nice thing to have because you can unit-test your code without being dependent on DB2 and it's really easy add with your code.