I am trying to use a Teradata UDF (User Defined Function) in a SAS Implicit SQL which establishes the connection to Teradata using LIBNAME Statement.Assume that the function is called PTY_DECRYPT and is defined in a Database called TEST in Teradata. The Purpose of this function is to decrypt values in a Column of a View in Teradata.
What works is using the UDF in an Explicit Sql .Below I am using the function on a column called SSN_NBR in a view called V_TEST_PERS present in the Database called SAMPLE.
Explcit Sql:
Options debug=DBMS_TIMERS sastrace=',,,d'
sastraceloc=saslog no$stsuffix fullstimer;
Proc Sql;
Connect to TERADATA(User=XXXXX pwd=XXXXX server=XXXXX);
Create Table Final as
select * from connection to teradata
(
Select
sub_id,
SSN_NBR,
TEST.PTY_DECRYPT(SSN_NBR,'T_ssn_test',400,0,0 ) as SSN_NBR_Decrypt
from SAMPLE.V_TEST_PERS
);
disconnect from teradata;
Quit;
But I would like to use the same function in an Implicit SQL but it does not work. Any ideas as to how to make it work in an Implicit Sql with minimum changes to the Implicit SQL?
Implicit Sql
Options debug=DBMS_TIMERS sastrace=',,,d'
sastraceloc=saslog no$stsuffix fullstimer;
Libname Td Teradata User=XXXXX pwd=XXXXX server=XXXXX database=SAMPLE ;
Proc sql;
Create table Final as
select
sub_id,
SSN_NBR,
TEST.PTY_DECRYPT(SSN_NBR,'T_ssn_test',400,0,0 ) as SSN_NBR_Decrypt
from Td.V_TEST_PERS;
Quit;
In your implicit SQL you reference the view with the LIBNAME alias TD, however when you reference the UDF you are not aliasing the TEST database containing the UDF with the LIBNAME alias. Syntactically, you may not be able to do that in SAS. (e.g. TD.TEST.PTY_DECRYPT() - in fact I wouldn't expect this to work)
The UDF may need to be placed in SYSLIB or TD_SYSFNLIB so that it is in a default search path for the database optimizer to find the UDF without being fully qualified. (e.g. TD_WEEK_BEGIN()) Alternatively, the UDF could be placed in database SAMPLE but that likely violates how UDFs are maintained in your environment, as it would in my environment.
Otherwise, the UDF call could be embedded in a view on the database, but then you have other issues to consider with the security of that column if your environment is not granting security on a column level basis to views containing encrypted data elements. (e.g. PHI, PII, etc.) Without a row-column level security mechanism in place to dynamically filter a users ability to see the column you are decrypting in the view putting the UDF into the view isn't going to work.
I asked the same question the SAS Communities Forum and I am glad to say that i did find a Solution to this Problem.
Please see the link below :
https://communities.sas.com/t5/Base-SAS-Programming/Using-a-Teradata-UDF-in-SAS-Implicit-Sql-Pass-Thru/m-p/266850/highlight/false#M52685
Related
I am quite new to GBQ and any help is appreciated it.
I have a query below:
#Standard SQL
create or replace table `xxx.xxx.applications`
as select * from `yyy.yyy.applications`
What I need to do is to add today's date at the end of the table name so it is something like xxx.xxx.applications_<todays date>
basically create a filename with Application but add date at the end of the name applications.
I am writing a procedure to create a table every time it runs but need to add the date for audit purposes every time I create the table (as a backup).
I searched everywhere and can't get the exact answer, is this possible in Query Editor as I need to store this as a Proc.
Thanks in advance
BigQuery doesn't support dynamic SQL at the moment which means that this kind of construction is not possible.
Currently BigQuery supports Parameterized Queries but its not possible to use parameters to dynamically change the source table's name as you can see in the provided link.
BigQuery supports query parameters to help prevent SQL injection when
queries are constructed using user input. This feature is only
available with standard SQL syntax. Query parameters can be used as
substitutes for arbitrary expressions. Parameters cannot be used as
substitutes for identifiers, column names, table names, or other parts
of the query.
If you need to build a query based on some variable's value, I suggest that you use some script in SHELL, Python or any other programming language to create the SQL statement and then execute it using the bq command.
Another approach could be using the BigQuery client library in some of the supported languages instead of the bq command.
It may seem a little strange, but there are already tables with names for each date.
In my project, I have tables for each date to make statistics easier to handle.
Of course, I don't think this is always the best way, but this is the table structure for my project.
(It's a common technique in Google BigQuery and Amazon Athena. This question is about Google BigQuery)
So to get the data, I want to generate today's date. If I use TODAY, I can get the data of the latest day without rewriting the code even if it is the next day.
I tried, but the code didn't work.
Not work 1:
CONCAT in FROM
SELECT
*
FROM
CONCAT('foo_', FORMAT_TIMESTAMP('%Y%m%d', CURRENT_TIMESTAMP(), 'Asia/Tokyo'))
Error:
Table-valued function not found: CONCAT at [4:3]
Not work 2:
create temporary function:
create temporary function getTableName() as (CONCAT('foo_', FORMAT_TIMESTAMP('%Y%m%d', CURRENT_TIMESTAMP(), 'Asia/Tokyo')));
Error:
CREATE TEMPORARY FUNCTION statements must be followed by an actual query.
Question
How do I generate a table name that contains TODAY's date?
In this case, I would recommend you to use Wild tables in BigQuery, which allows you to use some features in Standard SQL.
With Wild Tables you can use _TABLE_SUFFIX, it grants you the ability to filter/scan tables containing this parameter. The syntax would be as follows:
SELECT *
FROM `test-proj-261014.sample.test_*`
where _TABLE_SUFFIX = FORMAT_DATE('%Y%m%d', CURRENT_DATE)
I hope it helps.
Your first query should go like this:
select CONCAT('foo_', FORMAT_TIMESTAMP('%Y%m%d', CURRENT_TIMESTAMP(), 'Asia/Tokyo'))
For creating temporary function, use the below code:
create temp function getTableName() as
((select CONCAT('foo_', FORMAT_TIMESTAMP('%Y%m%d', CURRENT_TIMESTAMP(), 'Asia/Tokyo'))
));
select getTableName()
The error "CREATE TEMPORARY FUNCTION statements must be followed by an actual query." is because once the temporary functions are defined then you have to use the actual query to use that function and then the validity of function dies out. To define persistent UDFs and use them in multiple queries please go through the link to define permanent functions.You can reuse persistent UDFs across multiple queries, whereas you can only use temporary UDFs in a single query.
The database setup at my organisation is SQL tables copied onto our SAS server. The SQL tables were setup to run pre-programmed SQL queries, now SAS is the tool used. This however creates an issue with some tables having variables that are too long for SAS, but work in SQL. The label for the source variable is correct and not shortened.
The source table (in SQL Server) names:
Consolidated_Arrears_Vs_Portfolio_Balance_Ltd
Consolidated_Arrears_Vs_Portfolio_Balance_Pure
In SAS:
Consolidated_Arrears_Vs_Portfoli
Consolidated_Arrears_Vs_Portfoli
SAS Labels:
Consolidated_Arrears_Vs_Portfolio_Balance_Ltd
Consolidated_Arrears_Vs_Portfolio_Balance_Pure
So, how do I tell the difference in code between these two?
Thanks in advance.
To use the data as native in SAS, one approach would be to write a macro to map the original SQL names (per label) to the corresponding new SAS names. If the original table names got mangled as well you have a lot more issues.
Original SQL
select Abracadabra_Magical_Unity_Formation_SequenceId from AMUF_Master
Replace with
select %nameFor(Abracadabra_Magical_Unity_Formation_SequenceId) from AMUF_Master
The macro %nameFor would either do a dynamic lookup against the tables in the library, or perhaps better, when a static table design, create a fixed mapping table from a one time lookup
* presume SQL data now in libref MIGRATED;
* do once to get the variable metadata that includes LABEL and NAME;
proc sql;
create table static.nameFor as
select * from sashelp.vcolumn
where libnames = 'MIGRATED';
* use as needed;
%macro nameFor(SQL_Name);
%sysfunc(dosubl(select NAME from static.nameFor where LABEL="&SQL_Name"))
%mend;
You could also use the static.nameFor to discover all the SQL names that got changed during migration. Those would be where name ne label.
An automated approach would be to create a search and replace program that makes changes to a copy of the original SQL queries on-hand.
The search and replace would be either
find <long-named column>, replace with %nameFor(<long-named column>) , or
find <long-named column>, replace with <migrated to SAS column name>
The first replacement way adds noise.
The second way loses some of the original queries 'true-flavor'
I am using SAS Enterprise Guide (EG) 6.1 and want to know what are the indexes of our Oracle tables. Is there a way to write a program to get this information?
I tried to do:
LIBNAME DW ORACLE USER='username' PASSWORD='password' PATH='path.world' SCHEMA='schema';
DATA _NULL_ ;
dsid = OPEN(DW.some_table) ;
isIndexed = ATTRN(dsid,"ISINDEX") ;
PUT isIndexed = ;
RUN ;
some_table is the name of (my table), but I get an error:
ERROR: DATA STEP Component Object failure. Aborted during the COMPILATION phase.
ERROR 557-185: Variable some_table is not an object.
Reference: https://communities.sas.com/t5/ODS-and-Base-Reporting/check-if-index-exists/td-p/1966
OPEN takes a string or a value that resolves to a string. So you need
dsid= OPEN('dw.some_dataset');
I don't know if you can use that with Oracle or not, and I don't know whether ATTRN will be useful for this particular purpose or not. These all work well with SAS datasets, but it's up to the libname engine (and whatever middleware it uses) to implement the functionality that ATTRN would use.
For example, I don't use Oracle but I do have SQL Server tables with indexes, and I can run the above code on them; the code appears to work (it doesn't show errors) but it shows the tables as being unindexed, when they clearly are.
Your best bet is to connect using pass-through (CONNECT TO ...) instead of libname, and then you can run native Oracle syntax rather than using SAS.
I have a SqlServer db that I would like to port to MySQL. What's the best way to to this. Things that need to be ported are:
Tables (and data)
FileStream → MySQL equivalent?
Stored Procedures
Functions
Data types are relatively similar.
There is no equivalent to FileStream in MySQL - the files must either be stored as BLOBs, or on the file system while the path is stored in the database.
Migrating away from TSQL means:
There's no WITH clause in MySQL - it will have to converted into a derived table/inline view
There's no TOP syntax - these have to be converted to use LIMIT
There's no ranking/analytic functionality in MySQL - can't use ROW_NUMBER, RANK, DENSE_RANK or NTILE. See this article for alternatives.
MySQL views have notoriously limited functionality:
The SELECT statement cannot contain a subquery in the FROM clause.
The SELECT statement cannot refer to system or user variables.
Within a stored program, the definition cannot refer to program parameters or local variables.
The SELECT statement cannot refer to prepared statement parameters.
Any table or view referred to in the definition must exist. However, after a view has been created, it is possible to drop a table or view that the definition refers to. In this case, use of the view results in an error. To check a view definition for problems of this kind, use the CHECK TABLE statement.
The definition cannot refer to a TEMPORARY table, and you cannot create a TEMPORARY view.
Any tables named in the view definition must exist at definition time.
You cannot associate a trigger with a view.
As of MySQL 5.0.52, aliases for column names in the SELECT statement are checked against the maximum column length of 64 characters (not the maximum alias length of 256 characters).
Dynamic SQL will have to be converted to use MySQL's Prepared Statement syntax
A guide/article with some useful tips is available on the official MySQL dev site.
This is not for the faint of heart. Here is an article that explains what you are in for:
http://searchenterpriselinux.techtarget.com/news/column/0,294698,sid39_gci1187176,00.html