It may seem a little strange, but there are already tables with names for each date.
In my project, I have tables for each date to make statistics easier to handle.
Of course, I don't think this is always the best way, but this is the table structure for my project.
(It's a common technique in Google BigQuery and Amazon Athena. This question is about Google BigQuery)
So to get the data, I want to generate today's date. If I use TODAY, I can get the data of the latest day without rewriting the code even if it is the next day.
I tried, but the code didn't work.
Not work 1:
CONCAT in FROM
SELECT
*
FROM
CONCAT('foo_', FORMAT_TIMESTAMP('%Y%m%d', CURRENT_TIMESTAMP(), 'Asia/Tokyo'))
Error:
Table-valued function not found: CONCAT at [4:3]
Not work 2:
create temporary function:
create temporary function getTableName() as (CONCAT('foo_', FORMAT_TIMESTAMP('%Y%m%d', CURRENT_TIMESTAMP(), 'Asia/Tokyo')));
Error:
CREATE TEMPORARY FUNCTION statements must be followed by an actual query.
Question
How do I generate a table name that contains TODAY's date?
In this case, I would recommend you to use Wild tables in BigQuery, which allows you to use some features in Standard SQL.
With Wild Tables you can use _TABLE_SUFFIX, it grants you the ability to filter/scan tables containing this parameter. The syntax would be as follows:
SELECT *
FROM `test-proj-261014.sample.test_*`
where _TABLE_SUFFIX = FORMAT_DATE('%Y%m%d', CURRENT_DATE)
I hope it helps.
Your first query should go like this:
select CONCAT('foo_', FORMAT_TIMESTAMP('%Y%m%d', CURRENT_TIMESTAMP(), 'Asia/Tokyo'))
For creating temporary function, use the below code:
create temp function getTableName() as
((select CONCAT('foo_', FORMAT_TIMESTAMP('%Y%m%d', CURRENT_TIMESTAMP(), 'Asia/Tokyo'))
));
select getTableName()
The error "CREATE TEMPORARY FUNCTION statements must be followed by an actual query." is because once the temporary functions are defined then you have to use the actual query to use that function and then the validity of function dies out. To define persistent UDFs and use them in multiple queries please go through the link to define permanent functions.You can reuse persistent UDFs across multiple queries, whereas you can only use temporary UDFs in a single query.
Related
I am trying to write a SQL Command to insert some data from one table to a new table without any insert statement in bigquery but I cannot find a way to do it. (something similar to select into)
Here is the table:
create table database1.table1
(
pdesc string,
num int64
);
And here is the insert statement. I also tried the select into but it is not supported in bigquery.
insert into database1.table1
select column1, count(column2) as num
from database1.table2
group by column1;
Above is a possible way to insert. but I am looking for a way that I do not need to use any select statement. I am looking for something similar to 'select into' statement.
I am thinking of declaring variables and then somehow feed the data into the tables but do not know how.
I am not a Google employee. However - I understand the reasoning for not supporting creating a copy of a table (or query) from the console.
The challenge is that each table needs to be created must have a number of features defined such as associated project and expiry time.
Looking through the documentation (briefly) - it is worth exploring using bq utility - specifically the cp command -
Explore the following operations :
cache the query results to a temporary table
get the name of said temporary table
pass to a copy table command perhaps?
Other methods are described in the google cloud doco https://cloud.google.com/bigquery/docs/managing-tables#copy-table
We use snowflake at work to store data, and for one of the tables, I dont have the SQL query used to create the table. Is there a way to see the query used to make that table?
I tried using the following
get_ddl('table', 'db.table', true)
but this gives me an output like-
This doesnt give me any information about the sql query that was used. How do I get that in snowflake?
If get_ddl() is not enough you may use INFORMATION_SCHEMA.
To get more information you have 2 options:
Use the QUERY_HISTORY() table functions: https://docs.snowflake.com/en/sql-reference/functions/query_history.html
Use the QUERY_HISTORY() view: https://docs.snowflake.com/en/sql-reference/account-usage/query_history.html
If you use the funtions/view above and filter all the records by QUERY_TEXT, maybe you get more information about the exact SQL that was used to create your table.
I am quite new to GBQ and any help is appreciated it.
I have a query below:
#Standard SQL
create or replace table `xxx.xxx.applications`
as select * from `yyy.yyy.applications`
What I need to do is to add today's date at the end of the table name so it is something like xxx.xxx.applications_<todays date>
basically create a filename with Application but add date at the end of the name applications.
I am writing a procedure to create a table every time it runs but need to add the date for audit purposes every time I create the table (as a backup).
I searched everywhere and can't get the exact answer, is this possible in Query Editor as I need to store this as a Proc.
Thanks in advance
BigQuery doesn't support dynamic SQL at the moment which means that this kind of construction is not possible.
Currently BigQuery supports Parameterized Queries but its not possible to use parameters to dynamically change the source table's name as you can see in the provided link.
BigQuery supports query parameters to help prevent SQL injection when
queries are constructed using user input. This feature is only
available with standard SQL syntax. Query parameters can be used as
substitutes for arbitrary expressions. Parameters cannot be used as
substitutes for identifiers, column names, table names, or other parts
of the query.
If you need to build a query based on some variable's value, I suggest that you use some script in SHELL, Python or any other programming language to create the SQL statement and then execute it using the bq command.
Another approach could be using the BigQuery client library in some of the supported languages instead of the bq command.
I'm trying to do pagination in DB2. I wouldn't like to do it with subquery, but OFFSET is not working with TIMESTAMP_FORMAT.
Use of function TIMESTAMP_FORMAT in QSYS2 not valid. Data mapping error on member
I've found this question, but seems here the problem is with content of column and it's not my case, as values are alright and TIMESTAMP_FORMAT works without OFFSET.
I didn't look for some other way to not use TIMESTAMP_FORMAT, as I need to create pagination on queries written not by me, but by client.
The query looks like this.
SELECT DATE(TIMESTAMP_FORMAT(CHAR("tablename"."date"),'YYMMDD'))
FROM tableName
OFFSET 10 ROWS
I get
"[SQL0583] Use of function TIMESTAMP_FORMAT in QSYS2 not valid."
I'm not sure how OFFSET can relate to TIMESTAMP_FORMAT, but when I replace the select with select * it works fine.
I wonder why there is a conflict between OFFSET and TIMESTAMP_FORMAT and is there a way to bypass this without subquery.
From Listing of SQL Messages:
SQL0583
Function &1 in &2 cannot be invoked where specified because it is
defined to be not deterministic or contains an external action.
Functions that are not deterministic cannot be specified in a GROUP BY
clause or in a JOIN clause, or in the default clause for a global
variable.
Functions that are not deterministic or contain an external
action cannot be specified in a PARTITION BY clause or an ORDER BY
clause for an OLAP function and cannot be specified in the select list
of a query that contains an OFFSET clause.
The RAISE_ERROR function
cannot be specified in a GROUP BY or HAVING clause.
I don't know how to check these properties for the QSYS2.TIMESTAMP_FORMAT function (there is no its definition in the QSYS2.SYSROUTINES table), but it looks like improper definition of this function - there is no reason to create it as not deterministic or external action.
You can "deceive" DB2 like this:
CREATE FUNCTION MYSCHEMA.TIMESTAMP_FORMAT(str VARCHAR(4000), fmt VARCHAR(128))
RETURNS TIMESTAMP
DETERMINISTIC
CONTAINS SQL
NO EXTERNAL ACTION
RETURN QSYS2.TIMESTAMP_FORMAT(str, fmt);
SELECT
DATE(MYSCHEMA.TIMESTAMP_FORMAT(CHAR(tablename.date), 'YYMMDD'))
--DATE(QSYS2.TIMESTAMP_FORMAT(CHAR(tablename.date), 'YYMMDD'))
FROM table(values '190412') tableName(date)
OFFSET 10 ROWS;
And use this function instead. It works on my 7.3 at least.
It's a harmless deception, and you may ask IBM support to clarify such a "feature" of QSYS2.TIMESTAMP_FORMAT...
I suspect your problem is bad data...
The default for the IBM interactive tools, STRSQL and ACS Run SQL Scripts, is OPTIMIZE(*FIRSTIO) meaning get the first few rows back as quickly as possible...
With the OFFSET 10 clause you're probably accessing rows initially that you didn't before.
Try the following
create table mytest as (
SELECT DATE(TIMESTAMP_FORMAT(CHAR("tablename"."date"),'YYMMDD')) as mydate
FROM tableName
) with data
If that doesn't error, then yes you've found a bug, open a PMR.
Otherwise, you can see how far along the DB got by looking at the rows in the new table and track down the record with bad data.
I am trying to use a Teradata UDF (User Defined Function) in a SAS Implicit SQL which establishes the connection to Teradata using LIBNAME Statement.Assume that the function is called PTY_DECRYPT and is defined in a Database called TEST in Teradata. The Purpose of this function is to decrypt values in a Column of a View in Teradata.
What works is using the UDF in an Explicit Sql .Below I am using the function on a column called SSN_NBR in a view called V_TEST_PERS present in the Database called SAMPLE.
Explcit Sql:
Options debug=DBMS_TIMERS sastrace=',,,d'
sastraceloc=saslog no$stsuffix fullstimer;
Proc Sql;
Connect to TERADATA(User=XXXXX pwd=XXXXX server=XXXXX);
Create Table Final as
select * from connection to teradata
(
Select
sub_id,
SSN_NBR,
TEST.PTY_DECRYPT(SSN_NBR,'T_ssn_test',400,0,0 ) as SSN_NBR_Decrypt
from SAMPLE.V_TEST_PERS
);
disconnect from teradata;
Quit;
But I would like to use the same function in an Implicit SQL but it does not work. Any ideas as to how to make it work in an Implicit Sql with minimum changes to the Implicit SQL?
Implicit Sql
Options debug=DBMS_TIMERS sastrace=',,,d'
sastraceloc=saslog no$stsuffix fullstimer;
Libname Td Teradata User=XXXXX pwd=XXXXX server=XXXXX database=SAMPLE ;
Proc sql;
Create table Final as
select
sub_id,
SSN_NBR,
TEST.PTY_DECRYPT(SSN_NBR,'T_ssn_test',400,0,0 ) as SSN_NBR_Decrypt
from Td.V_TEST_PERS;
Quit;
In your implicit SQL you reference the view with the LIBNAME alias TD, however when you reference the UDF you are not aliasing the TEST database containing the UDF with the LIBNAME alias. Syntactically, you may not be able to do that in SAS. (e.g. TD.TEST.PTY_DECRYPT() - in fact I wouldn't expect this to work)
The UDF may need to be placed in SYSLIB or TD_SYSFNLIB so that it is in a default search path for the database optimizer to find the UDF without being fully qualified. (e.g. TD_WEEK_BEGIN()) Alternatively, the UDF could be placed in database SAMPLE but that likely violates how UDFs are maintained in your environment, as it would in my environment.
Otherwise, the UDF call could be embedded in a view on the database, but then you have other issues to consider with the security of that column if your environment is not granting security on a column level basis to views containing encrypted data elements. (e.g. PHI, PII, etc.) Without a row-column level security mechanism in place to dynamically filter a users ability to see the column you are decrypting in the view putting the UDF into the view isn't going to work.
I asked the same question the SAS Communities Forum and I am glad to say that i did find a Solution to this Problem.
Please see the link below :
https://communities.sas.com/t5/Base-SAS-Programming/Using-a-Teradata-UDF-in-SAS-Implicit-Sql-Pass-Thru/m-p/266850/highlight/false#M52685