I tried to create a table (set returning) function in redshift but failed.
Because redshift only supports scalar function (documentation). Is it true? Can I create table functions?
The simple answer: no you can not create table functions ("set returning functions) as you can do in Postgres.
The section Unsupported PostgreSQL Features in the Redshift manual explicitly lists "Table functions".
Related
It may seem a little strange, but there are already tables with names for each date.
In my project, I have tables for each date to make statistics easier to handle.
Of course, I don't think this is always the best way, but this is the table structure for my project.
(It's a common technique in Google BigQuery and Amazon Athena. This question is about Google BigQuery)
So to get the data, I want to generate today's date. If I use TODAY, I can get the data of the latest day without rewriting the code even if it is the next day.
I tried, but the code didn't work.
Not work 1:
CONCAT in FROM
SELECT
*
FROM
CONCAT('foo_', FORMAT_TIMESTAMP('%Y%m%d', CURRENT_TIMESTAMP(), 'Asia/Tokyo'))
Error:
Table-valued function not found: CONCAT at [4:3]
Not work 2:
create temporary function:
create temporary function getTableName() as (CONCAT('foo_', FORMAT_TIMESTAMP('%Y%m%d', CURRENT_TIMESTAMP(), 'Asia/Tokyo')));
Error:
CREATE TEMPORARY FUNCTION statements must be followed by an actual query.
Question
How do I generate a table name that contains TODAY's date?
In this case, I would recommend you to use Wild tables in BigQuery, which allows you to use some features in Standard SQL.
With Wild Tables you can use _TABLE_SUFFIX, it grants you the ability to filter/scan tables containing this parameter. The syntax would be as follows:
SELECT *
FROM `test-proj-261014.sample.test_*`
where _TABLE_SUFFIX = FORMAT_DATE('%Y%m%d', CURRENT_DATE)
I hope it helps.
Your first query should go like this:
select CONCAT('foo_', FORMAT_TIMESTAMP('%Y%m%d', CURRENT_TIMESTAMP(), 'Asia/Tokyo'))
For creating temporary function, use the below code:
create temp function getTableName() as
((select CONCAT('foo_', FORMAT_TIMESTAMP('%Y%m%d', CURRENT_TIMESTAMP(), 'Asia/Tokyo'))
));
select getTableName()
The error "CREATE TEMPORARY FUNCTION statements must be followed by an actual query." is because once the temporary functions are defined then you have to use the actual query to use that function and then the validity of function dies out. To define persistent UDFs and use them in multiple queries please go through the link to define permanent functions.You can reuse persistent UDFs across multiple queries, whereas you can only use temporary UDFs in a single query.
I have a few questions around the create table syntax in standard and legacy sql
The new BigQueryUI doesn't show standard sql types and shows only legacy types. I understand they are mapped one to one with the legacy types but the examples in creating partitioned tables shows options which are not available in the UI
If I create a table using the JSON field schema can I still use the standard sql?
The BigQueryUI shows only partitioning the table by Ingestion time, but I want to create a table with date column and I don't see an option for it. If I have to create the DDL manually, I did not see the examples on how to use JSON field schema to construct a create table statement.
The new BigQueryUI doesn't show standard sql types
BigQuery standardSQL and LegacySQL are 2 options to write SQL syntax (See this link for more detail) and have nothing to do with the column Types in BigQuery, Details on table types can be found in this link, I also find this Link helpful
If I create a table using the JSON field schema can I still use the standard sql?
To create a table using JSON you need to run bq command line, If you need help how to write this syntax let us know
but I want to create a table with date column and I don't see an option for it
You can use this standardSQL syntax to do this:
#standardSQL
CREATE OR REPLACE TABLE `project.dataset.tableId`
PARTITION BY myDate
CLUSTER BY cluster_col AS
SELECT * from sourceTable
Note: myDate column is a column in the source table
I am trying to retrieve the DDL for a table in Redshift. I found this view where I can easily select the definition for any table. However I need this information in one line, and I know that there is this Listagg function, but if I try to do this:
select listagg(ddl, ' ')
from admin.v_generate_tbl_ddl
where schemaname = 'schema'
and tablename = 'orders'
It's giving me this error:
Query execution failed
Reason: SQL Error [XX000]: ERROR: One or more of the used functions
must be applied on at least one user created tables. Examples of user
table only functions are LISTAGG, MEDIAN, PERCENTILE_CONT, etc
Can You please help me on how can I achieve this?
listagg function is a compute-node only function.
But the query you run to get the table ddl runs only on leader because it only specifies pg_* tables.
According to AWS documentation
A query that references only catalog tables (tables with a PG prefix, such as PG_TABLE_DEF) or that does not reference any tables, runs exclusively on the leader node.
If a query that uses a compute-node function doesn't reference a user-defined table or Amazon Redshift system table returns the following error.
[Amazon] (500310) Invalid operation: One or more of the used functions must be applied on at least one user created table.
https://docs.aws.amazon.com/redshift/latest/dg/c_SQL_functions_compute_node_only.html
To sum up, since your query does not reference any user created table, you can not use listagg
How can you query a range of timestamped tables with the new syntax? Using TABLE_DATE_RANGE returns the error Unhandled node type in from clause: TVF.
The latest version of BigQuery supports an equivalent of table wildcards with Standard SQL. The documentation is available here: https://cloud.google.com/bigquery/docs/wildcard-tables.
Also please take a look at this post:
Is there an equivalent of table wildcard functions in BigQuery with standard SQL?
Does SQLite support common table expressions?
I'd like to run query like that:
with temp (ID, Path)
as (
select ID, Path from Messages
) select * from temp
As of Sqlite version 3.8.3 SQLite supports common table expressions.
Change log
Instructions
Another solution is to integrate a "CTE to SQLite" translation layer in your application :
"with w as (y) z" => "create temp view w as y;z"
"with w(x) as (y) z" => "create temp table w(x);insert into w y;z"
As an (ugly, desesperate, but working) example :
http://nbviewer.ipython.org/github/stonebig/baresql/blob/master/examples/baresql_with_cte_code_included.ipynb
SQLite doesn't support CTEs, window functions, or any of the like. You can, however, write your own user functions that you can call inside SQLite by registering them to the database with the SQLite API using sqlite_create_function(). You register them with the database, and then you can use them in your own application code. You can make an aggregate function that would perform the sum of a series of averages based on the individual column values. For each value, a step-type callback function is called that allows you to perform some calculation on the data, and a pointer for holding state data is also available.