I am trying to query data on JSON file using S3-Select. I am unable to filter based on the date field. I have tried using current_date, sysdate and a few CAST options too. I am planning to compute the curent_date and send it from API. Before that, I wanted to check if there was any other way to get the current date in the query.
Sample Json :
S3 Select Queries I have tried :
select * from S3Object[*].stations[*] as station where station.iataCd = 'NCL' and station.expiryDt >= sysdate();
select * from S3Object[*].stations[*] as station where station.iataCd = 'NCL' and station.expiryDt >= current_date;
select * from S3Object[*].stations[*] as station where station.iataCd = 'NCL' and station.expiryDt >= '2023-01-23'
Related
Table
Objective
I want to select only those record which matches a specific date (yyyy-MM-dd)
Like : SELECT * FROM WEATHER WHERE CREATED_AT = "2018-11-28"
For your request:
SELECT * FROM WEATHER WHERE FORMATDATETIME(CREATED_AT,'yyyy-MM-dd') = '2018-11-28'
It is a harder to find info about converting timestamp to date without formatting. Sample for finding all tomorrow scheduling tasks:
SELECT * FROM schedule WHERE CAST(date_time AS DATE)=DATEADD(DAY, 1, TODAY);
I want to take out data for every date range in Data Studio without the need to change date range selectors in my BigQuery all the time. However, not sure if it is even possible to do so. The reasons I do this is to make sure that the queried data is only for 30 days, as later it do some kind of segmentation using that 30 days data.
Then I figured out that the Data Studio can use dynamic_date, however this way will never produce any datatable (datatable will be used to do other queries from it). Is it possible to do dynamic_date in BigQuery instead? like retrieving data from BigQuery using a date range not previously defined in the query.
From my point of view, code should be like :
SELECT
ID,
FROM `table`
WHERE DATE(Timestamp) between $DS_START_DATE and $DS_START_DATE + INTERVAL 30 DAY)
or
WHERE DATE(Timestamp) >= #DS_START_DATE
I believe in pure Bigquery you can use DECLARE clause for that purpose, defining variables of the specified type:
declare DS_START_DATE date default "2020-03-03";
declare DS_END_DATE date default "2020-03-04";
WITH sample AS (
SELECT '10001' AS id, cast('2020-03-01' AS timestamp) as date_id UNION ALL
SELECT '10002', cast('2020-03-02' AS timestamp) UNION ALL
SELECT '10003', cast('2020-03-03' AS timestamp) UNION ALL
SELECT '10004', cast('2020-03-04' AS timestamp) UNION ALL
SELECT '10005', cast('2020-03-05' AS timestamp) UNION ALL
SELECT '10006', cast('2020-03-06' AS timestamp)
)
select id, date_id from sample
where date(date_id) between DS_START_DATE and DS_END_DATE
Alternatively, you can take a look at parameterized queries, however as I mentioned in the comment, they are not supported in classic BigQuery web UI.
In order to visualize a bubble chart in Gapminder, all dates have to be converted from, dd/mm/yyyy to yyyyQ1. This code does the conversion on in SQLite.
I am able to convert the dates, using this code, on my local SQLite client, then load the outputted csv file into Gapminder to view the bubble chart.
However this 25 gb database has outgrown the SQLite client and needs to be queried using BigQuery.
There are two problems.
1: First, for BigQuery, this code needs to convert the csv date column from dd/mm/yyyy to mm/dd/yyyy.
Then for Gapminder it needs the final output to be yyyyQ1..
The problem is when I run the same code in the BigQuery web UI SQL editor, I receive an error, "unexpected pipe":
Input:
replace(substr(PCR.repdte,6),'/','')||'q'||CAST(1+ .
((substr(PCR.repdte,1,2)-1) / 3) AS INTEGER)
Output:
Syntax error: Unexpected "|" at [3:44]
Here is the entire statement I successfully run on the SQLite client, and attempted to run on the Bigquery SQL web ui editor:
SELECT
(SELECT
replace(substr(PCR.repdte,6),'/','')||'q'||CAST(1+ .
((substr(PCR.repdte,1,2)-1) / 3) AS INTEGER)
FROM All_Reports_19920331_Performance_and_Condition_Ratios as PCR) AS
Quarter,
(SELECT
Round(AVG(PCR.lnlsdepr))
FROM All_Reports_19920331_Performance_and_Condition_Ratios as PCR) AS
NetLoansAndLeasesToDeposits,
(SELECT sum(CAST(LD.IDdepsam as int))
FROM
'All_Reports_19920331_Deposits_Based_on_the_Dollars250,000_
Reporting_Threshold' AS LD) AS DepositAccountsWith$LessThan$250k
UNION ALL
SELECT
(SELECT
replace(substr(PCR.repdte,6),'/','')||'q'||CAST(1+ .
((substr(PCR.repdte,1,2)-1) / 3) AS INTEGER) --converts date format
from dd/mm/yyyy to yyyyq1 (financial quarters)
FROM All_Reports_19920630_Performance_and_Condition_Ratios as PCR) AS
Quarter,
(SELECT
Round(AVG(PCR.lnlsdepr))
FROM All_Reports_19920630_Performance_and_Condition_Ratios as PCR) AS
NetLoansAndLeasesToDeposits,
(SELECT sum(CAST(LD.IDdepsam as int))
FROM
'All_Reports_19920630_Deposits_Based_on_the_Dollars
250,000_Reporting_Threshold' AS LD) AS
DepositAccountsWith$LessThan$250k
The goal is to convert the date from dd/mm/yyyy to mm/dd/yyyy so BigQuery can read it. Then convert it again to, yyyyQ1, so Gapminder can read it.
all dates have to be converted from, dd/mm/yyyy to yyyyQ1
Below simplified example for BigQuery Standard SQL
#standardSQL
WITH `project.dataset.table` AS (
SELECT '31/12/2018' dt UNION ALL
SELECT '31/01/2019'
)
SELECT dt,
CONCAT(
FORMAT_DATE('%Y', PARSE_DATE('%d/%m/%Y', dt)),
'Q', CAST(EXTRACT(QUARTER FROM PARSE_DATE('%d/%m/%Y', dt)) AS STRING)
) date_yyyyQ1
FROM `project.dataset.table`
with result
Row dt date_yyyyQ1
1 31/12/2018 2018Q4
2 31/01/2019 2019Q1
I have a date column (in YYYY-MM-DD format) in a big query table. I am unable to apply a where clause to the date column. I am using the following queries:
SELECT * FROM [dataSet_Id.TableName] where CR_DT=DATE("2016-01-01")
SELECT * FROM [dataSet_Id.TableName] where CR_DT=DATE("2016-01-01") where CR_DT=20160101
So how do I do it?
I got it work, If I use Standard SQL Dialect instead of Legacy SQL
Sample queries to handle date in where clause:
SELECT * from demoschema.demotable where dob = date('2016-08-10');
SELECT * from demoschema.demotable where dob = '2016-08-11';
If you want to use Standard SQL Dialect, just go to Show options then you will find SQL Version field which is use for enabling Standard SQL. .Dialect.
If the type of your CR_DT column is String then:
SELECT * FROM [dataSet_Id.TableName] where CR_DT = '2016-01-01'
If the type of your CR_DT column is TIMESTAMP then:
SELECT * FROM [dataSet_Id.TableName] where DATE(CR_DT) = DATE(timestamp('2016-01-01'))
I am having trouble with the following simple query
SELECT * FROM users WHERE Created = '28/02/2013'
The issue is the column CREATED is a datetime datatype, so the query executes fine as long as the timestamp is 0:00:0, if the time stamp is set to say 12:00, then the query does not return a result set.
Any idea why?
Thanks
Because you are not specifying the time, so it assumes that you are doing:
SELECT * FROM users WHERE Created = '28/02/2013 00:00:00'
If you want the whole day, then you need a range of times:
SELECT *
FROM users
WHERE Created >= '20130228' AND Created < '20130301'
Also, please use non ambiguous format for dates ('YYYYMMDD') instead of other formats.
SELECT * FROM users WHERE CAST(Created AS DATE) = '28/02/2013'
will fix it, but be careful, it disables indexes
SELECT * FROM users WHERE Created BETWEEN '28/02/2013 00:00' AND '28/02/2013 23:59'
And this will use index
If you don't need to consider time: try to convert created field to date and then compare as;
SELECT * FROM users WHERE convert(date,Created) = '28/02/2013'
--this would be even better with iso date format (not culture specific)
SELECT * FROM users WHERE convert(date,Created) = '20130228' --yyyymmdd format
You have to convert the column into date and then compare
SELECT * FROM users WHERE CONVERT(VARCHAR(11), Created, 106) = '28 Feb 2013'