SQLite code to convert date ( dd/mm/yyyy to yyyyQ1) not working in BigQuery SQL editor - sql

In order to visualize a bubble chart in Gapminder, all dates have to be converted from, dd/mm/yyyy to yyyyQ1. This code does the conversion on in SQLite.
I am able to convert the dates, using this code, on my local SQLite client, then load the outputted csv file into Gapminder to view the bubble chart.
However this 25 gb database has outgrown the SQLite client and needs to be queried using BigQuery.
There are two problems.
1: First, for BigQuery, this code needs to convert the csv date column from dd/mm/yyyy to mm/dd/yyyy.
Then for Gapminder it needs the final output to be yyyyQ1..
The problem is when I run the same code in the BigQuery web UI SQL editor, I receive an error, "unexpected pipe":
Input:
replace(substr(PCR.repdte,6),'/','')||'q'||CAST(1+ .
((substr(PCR.repdte,1,2)-1) / 3) AS INTEGER)
Output:
Syntax error: Unexpected "|" at [3:44]
Here is the entire statement I successfully run on the SQLite client, and attempted to run on the Bigquery SQL web ui editor:
SELECT
(SELECT
replace(substr(PCR.repdte,6),'/','')||'q'||CAST(1+ .
((substr(PCR.repdte,1,2)-1) / 3) AS INTEGER)
FROM All_Reports_19920331_Performance_and_Condition_Ratios as PCR) AS
Quarter,
(SELECT
Round(AVG(PCR.lnlsdepr))
FROM All_Reports_19920331_Performance_and_Condition_Ratios as PCR) AS
NetLoansAndLeasesToDeposits,
(SELECT sum(CAST(LD.IDdepsam as int))
FROM
'All_Reports_19920331_Deposits_Based_on_the_Dollars250,000_
Reporting_Threshold' AS LD) AS DepositAccountsWith$LessThan$250k
UNION ALL
SELECT
(SELECT
replace(substr(PCR.repdte,6),'/','')||'q'||CAST(1+ .
((substr(PCR.repdte,1,2)-1) / 3) AS INTEGER) --converts date format
from dd/mm/yyyy to yyyyq1 (financial quarters)
FROM All_Reports_19920630_Performance_and_Condition_Ratios as PCR) AS
Quarter,
(SELECT
Round(AVG(PCR.lnlsdepr))
FROM All_Reports_19920630_Performance_and_Condition_Ratios as PCR) AS
NetLoansAndLeasesToDeposits,
(SELECT sum(CAST(LD.IDdepsam as int))
FROM
'All_Reports_19920630_Deposits_Based_on_the_Dollars
250,000_Reporting_Threshold' AS LD) AS
DepositAccountsWith$LessThan$250k
The goal is to convert the date from dd/mm/yyyy to mm/dd/yyyy so BigQuery can read it. Then convert it again to, yyyyQ1, so Gapminder can read it.

all dates have to be converted from, dd/mm/yyyy to yyyyQ1
Below simplified example for BigQuery Standard SQL
#standardSQL
WITH `project.dataset.table` AS (
SELECT '31/12/2018' dt UNION ALL
SELECT '31/01/2019'
)
SELECT dt,
CONCAT(
FORMAT_DATE('%Y', PARSE_DATE('%d/%m/%Y', dt)),
'Q', CAST(EXTRACT(QUARTER FROM PARSE_DATE('%d/%m/%Y', dt)) AS STRING)
) date_yyyyQ1
FROM `project.dataset.table`
with result
Row dt date_yyyyQ1
1 31/12/2018 2018Q4
2 31/01/2019 2019Q1

Related

How to convert an int to DateTime in BigQuery

I have an INT64 column called "Date" which contains many different numbers like: "20210209" or "20200305". I want to turn those numbers into a date with this format: MM-YYYY (so in these cases, 02-2021 and 03-2020). Ultimately I want to sum all the data in each month together. The problem is that BigQuery can't convert INT64 to date, only to strings. I'm not sure if I should convert to a string and then to a date or if there is a better way.
Although converting to a string then a date both works and is very concise, over large enough numbers of rows (which may be the case in Big Query) you may be better off using integer maths and using DATE(year, month, day)...
https://cloud.google.com/bigquery/docs/reference/standard-sql/date_functions#date
SELECT
DATE(
DIV( 20210209 , 10000), -- Which gives 2021
DIV(MOD(20210209, 10000), 100), -- Which gives 02
MOD(20210209, 100) -- Which gives 09
)
You can convert the value to a string and use parse_date():
select parse_date('%Y%m%d', cast(20210209 as string))
Another option
select date,
regexp_replace('' || date, r'(\d{4})(\d{2})(\d{2})', r'\2-\1') as MM_YYYY
from your_table
if applied to sample data in your question - output is
Yet another option
select date,
format_date('%m-%Y', parse_date('%Y%m%d', '' || date)) as MM_YYYY
from your_table
with same output

How to Convert a date YYYY-MM-DD to a string YYYYMM, using Big Query SQL Standard?

Got a Export_date field with YYYY-MM-DD. I'd like to get YYYYMM as output.
input : 2020-06-02
Expected output : 202006
I'm using SQL Standard in GCP Bigquery
How to Convert a date YYYY-MM-DD to a string YYYYMM, using Big Query SQL Standard ?
I've tried this function
substr(cast(export_date as string), 0,7) as date,
But I got YYYY-DD as output (2020-06)
Thank you
You can use format_timestamp:
SELECT format_timestamp('%Y%m', '2020-06-02')
it gives back '202006'.
A simple method is:
select extract(year from export_date) * 100 + extract(month from export_date) as yyyymm
You could also format this as a string using date_format().
If your value is stored as a string and not a date, then just use string functions:
select concat(substr(export_date, 1, 4), substr(export_date, 6, 2))

Dynamic query using bigquery and data studio

I want to take out data for every date range in Data Studio without the need to change date range selectors in my BigQuery all the time. However, not sure if it is even possible to do so. The reasons I do this is to make sure that the queried data is only for 30 days, as later it do some kind of segmentation using that 30 days data.
Then I figured out that the Data Studio can use dynamic_date, however this way will never produce any datatable (datatable will be used to do other queries from it). Is it possible to do dynamic_date in BigQuery instead? like retrieving data from BigQuery using a date range not previously defined in the query.
From my point of view, code should be like :
SELECT
ID,
FROM `table`
WHERE DATE(Timestamp) between $DS_START_DATE and $DS_START_DATE + INTERVAL 30 DAY)
or
WHERE DATE(Timestamp) >= #DS_START_DATE
I believe in pure Bigquery you can use DECLARE clause for that purpose, defining variables of the specified type:
declare DS_START_DATE date default "2020-03-03";
declare DS_END_DATE date default "2020-03-04";
WITH sample AS (
SELECT '10001' AS id, cast('2020-03-01' AS timestamp) as date_id UNION ALL
SELECT '10002', cast('2020-03-02' AS timestamp) UNION ALL
SELECT '10003', cast('2020-03-03' AS timestamp) UNION ALL
SELECT '10004', cast('2020-03-04' AS timestamp) UNION ALL
SELECT '10005', cast('2020-03-05' AS timestamp) UNION ALL
SELECT '10006', cast('2020-03-06' AS timestamp)
)
select id, date_id from sample
where date(date_id) between DS_START_DATE and DS_END_DATE
Alternatively, you can take a look at parameterized queries, however as I mentioned in the comment, they are not supported in classic BigQuery web UI.

storing date in 'CCYYMMDD' format in Teradata

I would like to store dates in the format CCYYMMDD in Teradata, but I fail to do so. Find below what I tried so far:
query 1:
SEL CAST(CAST(CURRENT_DATE AS DATE FORMAT 'YYYYMMDD') AS VARCHAR(8))
-- Output: 20191230 ==> this works!
query 2:
SEL CAST(CAST(CURRENT_DATE AS DATE FORMAT 'CCYYMMDD') AS VARCHAR(8))
-- output: SELECT Failed. [3530] Invalid FORMAT string 'CCYYMMDD'.
It seems that the CCYYMMDD is not available in Teradata right away. Is there a workaround?
Tool used: Teradata SQL assistant
Internally, dates are stored as integers in Teradata. So when you say you want to store them in a different format, I don't think you can do that. But you can choose how to display / return the values.
I'm sure there's a cleaner way to get the format you want, but here's one way:
WITH cte (mydate) AS (
SELECT CAST(CAST(CURRENT_DATE AS DATE FORMAT 'YYYYMMDD') AS CHAR(8)) AS mydate
)
SELECT
CAST(
(CAST(SUBSTRING(mydate FROM 1 FOR 2) AS INTEGER) + 1) -- generate "century" value
AS CHAR(2) -- cast value as string
) || SUBSTRING(mydate FROM 3) AS new_date -- add remaining portion of date string
FROM cte
SQL Fiddle - Postgres
You'd have to add some extra logic to handle years before 1000 and after 9999. I don't have a TD system to test, but give it a try and let me know.

How to convert Timestamp to Date Data Type in Google Bigquery

I am trying to convert Timestamp data type columns to Date datatype using:
bq query -q --destination_table=NEW_DATE_TABLE --replace "SELECT DATE(CURR_DT) AS CURR_DT from TEST.DATE_TABLE"
The new table shows the column as STRING rather than date. Is there a way to convert timestamp to date data type.
Requested Screenshot
If you use Standard SQL, you can do the following:
SELECT * REPLACE(EXTRACT(DATE FROM curr_dt)) AS curr_dt FROM test.date_table
If curr_dt is repeated field, then the solution will look the following:
SELECT * REPLACE(
ARRAY(
SELECT EXTRACT(DATE FROM curr_dt) FROM t.curr_dt
) AS curr_dt)
FROM test.date_table t
Consider below!
Works in both Legacy and Standard SQL
SELECT CAST(DATE(CURR_DT) AS DATE) AS CURR_DT FROM TEST.DATE_TABLE
Added to address comment
Try below - as I mentioned above - it works for both Legacy and Standard
SELECT CAST(DATE(CURR_DT) AS DATE) AS CURR_DT
FROM (SELECT CURRENT_TIMESTAMP() AS CURR_DT)
if you are interested in making your case working with Legacy SQL - provide more details about CURR_DT field
Try this
SELECT TIMESTAMP_SECONDS(CAST(CURR_DT AS INT64)) AS CURR_DT FROM TEST.DATE_TABLE