I'm trying to simplify my databricks notebook by creating date parameters. I want my code to run with date filters, and my goal is to only have to modify one line to get my results by a period.
This is my query :
create or replace temp view deldet as
select
case when
dyh_date_event::date between '2022-09-01' and '2022-09-31'
then concat('2022-09-01','-','2022-09-31')
else ('2022-10-01','-','2022-10-31')
end as periode,
*
from
hivestore.f_delivery_detail
where
dyh_date_event::date between '2022-09-01' and '2022-10-31';
This is what i tried :
set p1d1 = '2022-09-01';
set p1d2 = '2022-09-31';
set p2d1 = '2022-10-01';
set p2d2 = '2022-10-31';
create or replace temp view deldet as
select
case when
dyh_date_event::date between p1d1 and p1d2
then concat(p1d1,'-',p1d2)
else (p2d1,'-',p2d2)
end as periode,
*
from
hivestore.f_delivery_detail
where
dyh_date_event::date between p1d1 and p2d2;
But it doesn't work :
AnalysisException: Column 'p1d1' does not exist
A view can not have parameters.
You can put the query into a SQL function and then pass the parameters to the function:
create or replace function deldet(p1d1 date, p1d2 date, p2d1 date, p2d2 date)
as
$$
select
case when
dyh_date_event::date between p1d1 and p1d2
then concat(p1d1,'-',p1d2)
else (p2d1,'-',p2d2)
end as periode,
*
from
hivestore.f_delivery_detail
where
dyh_date_event::date between p1d1 and p2d2;
$$
language sql
stable;
Then you can use it like this:
select *
from deldet('2022-09-01', '2022-09-30', '2022-10-01', '2022-10-31');
Related
I am developing a scheduled query where I am using the WITH statement to join and filtrate several tables from BigQuery. To filtrate the dates, I would like to declare the following variables:
DECLARE initial, final DATE;
SET initial = DATE_TRUNC(DATE_TRUNC(CURRENT_DATE(), MONTH)+7,ISOWEEK);
SET final = LAST_DAY(DATE_TRUNC(CURRENT_DATE(), MONTH)+7, ISOWEEK);
However, when executing this query, I am getting two results; one for the variables declared (which I am not interested in having them as output), and the WITH statement that is selected at the end (which as the results that I am interested in).
The principal problem is that, whenever I try t connect this scheduled query to a table in Google Data Studio I get the following error:
Invalid value: configuration.query.destinationTable cannot be set for scripts;
How can I declare a variable without getting it as a result at the end?
Here you have a sample of the code I am trying work in:
DECLARE initial, final DATE;
SET initial = DATE_TRUNC(DATE_TRUNC(CURRENT_DATE(), MONTH)+7,ISOWEEK);
SET final = LAST_DAY(DATE_TRUNC(CURRENT_DATE(), MONTH)+7, ISOWEEK);
WITH HelloWorld AS (
SELECT shop_date, revenue
FROM fulltable
WHERE shop_date >= initial
AND shop_date <= final
)
SELECT * from HelloWorld;
with initial1 as ( select DATE_TRUNC(DATE_TRUNC(CURRENT_DATE(), MONTH)+7,ISOWEEK) as initial2),
final1 as ( select LAST_DAY(DATE_TRUNC(CURRENT_DATE(), MONTH)+7, ISOWEEK) as final2),
HelloWorld AS (
SELECT shop_date, revenue
FROM fulltable
WHERE shop_date >= (select initial2 from initial1) AND shop_date <= (select final2 from final1)
)
SELECT * from HelloWorld;
With config table having just 1 row and cross-joining it with your table, your query can be written like below.
WITH config AS (
SELECT DATE_TRUNC(DATE_TRUNC(CURRENT_DATE(), MONTH)+7,ISOWEEK) AS initial,
LAST_DAY(DATE_TRUNC(CURRENT_DATE(), MONTH)+7, ISOWEEK) AS final
),
HelloWorld AS (
SELECT * FROM UNNEST([DATE '2022-06-06']) shop_date, config
WHERE shop_date >= config.initial AND shop_date <= config.final
)
SELECT * FROM HelloWorld;
A few patterns I've used:
If you have many that have the same return type (STRING)
CREATE TEMP FUNCTION config(key STRING)
RETURNS STRING AS (
CASE key
WHEN "timezone" THEN "America/Edmonton"
WHEN "something" THEN "Value"
END
);
Then use config(key) to retrieve the value.
Or,
Create a function for each constant
CREATE TEMP FUNCTION timezone()
RETURNS STRING AS ("America/Edmonton");
Then use timezone() to get the value.
It would execute the function each time, so don't do something expensive in there (like SELECT from another table).
Scripting/procedures for BigQuery just came out in beta - is it possible to invoke procedures using the BigQuery python client?
I tried:
query = """CALL `myproject.dataset.procedure`()...."""
job = client.query(query, location="US",)
print(job.results())
print(job.ddl_operation_performed)
print(job._properties) but that didn't give me the result set from the procedure. Is it possible to get the results?
Thank you!
Edited - stored procedure I am calling
CREATE OR REPLACE PROCEDURE `Project.Dataset.Table`(IN country STRING, IN accessDate DATE, IN accessId, OUT saleExists INT64)
BEGIN
IF EXISTS (SELECT 1 FROM dataset.table where purchaseCountry = country and purchaseDate=accessDate and customerId = accessId)
THEN
SET saleExists = (SELECT 1);
ELSE
INSERT Dataset.MissingSalesTable (purchaseCountry, purchaseDate, customerId) VALUES (country, accessDate, accessId);
SET saleExists = (SELECT 0);
END IF;
END;
If you follow the CALL command with a SELECT statement, you can get the return value of the function as a result set. For example, I created the following stored procedure:
BEGIN
-- Build an array of the top 100 names from the year 2017.
DECLARE
top_names ARRAY<STRING>;
SET
top_names = (
SELECT
ARRAY_AGG(name
ORDER BY
number DESC
LIMIT
100)
FROM
`bigquery-public-data.usa_names.usa_1910_current`
WHERE
year = 2017 );
-- Which names appear as words in Shakespeare's plays?
SET
top_shakespeare_names = (
SELECT
ARRAY_AGG(name)
FROM
UNNEST(top_names) AS name
WHERE
name IN (
SELECT
word
FROM
`bigquery-public-data.samples.shakespeare` ));
END
Running the following query will return the procedure's return as the top-level results set.
DECLARE top_shakespeare_names ARRAY<STRING> DEFAULT NULL;
CALL `my-project.test_dataset.top_names`(top_shakespeare_names);
SELECT top_shakespeare_names;
In Python:
from google.cloud import bigquery
client = bigquery.Client()
query_string = """
DECLARE top_shakespeare_names ARRAY<STRING> DEFAULT NULL;
CALL `swast-scratch.test_dataset.top_names`(top_shakespeare_names);
SELECT top_shakespeare_names;
"""
query_job = client.query(query_string)
rows = list(query_job.result())
print(rows)
Related: If you have SELECT statements within a stored procedure, you can walk the job to fetch the results, even if the SELECT statement isn't the last statement in the procedure.
# TODO(developer): Import the client library.
# from google.cloud import bigquery
# TODO(developer): Construct a BigQuery client object.
# client = bigquery.Client()
# Run a SQL script.
sql_script = """
-- Declare a variable to hold names as an array.
DECLARE top_names ARRAY<STRING>;
-- Build an array of the top 100 names from the year 2017.
SET top_names = (
SELECT ARRAY_AGG(name ORDER BY number DESC LIMIT 100)
FROM `bigquery-public-data.usa_names.usa_1910_2013`
WHERE year = 2000
);
-- Which names appear as words in Shakespeare's plays?
SELECT
name AS shakespeare_name
FROM UNNEST(top_names) AS name
WHERE name IN (
SELECT word
FROM `bigquery-public-data.samples.shakespeare`
);
"""
parent_job = client.query(sql_script)
# Wait for the whole script to finish.
rows_iterable = parent_job.result()
print("Script created {} child jobs.".format(parent_job.num_child_jobs))
# Fetch result rows for the final sub-job in the script.
rows = list(rows_iterable)
print("{} of the top 100 names from year 2000 also appear in Shakespeare's works.".format(len(rows)))
# Fetch jobs created by the SQL script.
child_jobs_iterable = client.list_jobs(parent_job=parent_job)
for child_job in child_jobs_iterable:
child_rows = list(child_job.result())
print("Child job with ID {} produced {} rows.".format(child_job.job_id, len(child_rows)))
It works if you have SELECT inside your procedure, given the procedure being:
create or replace procedure dataset.proc_output() BEGIN
SELECT t FROM UNNEST(['1','2','3']) t;
END;
Code:
from google.cloud import bigquery
client = bigquery.Client()
query = """CALL dataset.proc_output()"""
job = client.query(query, location="US")
for result in job.result():
print result
will output:
Row((u'1',), {u't': 0})
Row((u'2',), {u't': 0})
Row((u'3',), {u't': 0})
However, if there are multiple SELECT inside a procedure, only the last result set can be fetched this way.
Update
See below example:
CREATE OR REPLACE PROCEDURE zyun.exists(IN country STRING, IN accessDate DATE, OUT saleExists INT64)
BEGIN
SET saleExists = (WITH data AS (SELECT "US" purchaseCountry, DATE "2019-1-1" purchaseDate)
SELECT Count(*) FROM data where purchaseCountry = country and purchaseDate=accessDate);
IF saleExists = 0 THEN
INSERT Dataset.MissingSalesTable (purchaseCountry, purchaseDate, customerId) VALUES (country, accessDate, accessId);
END IF;
END;
BEGIN
DECLARE saleExists INT64;
CALL zyun.exists("US", DATE "2019-2-1", saleExists);
SELECT saleExists;
END
BTW, your example is much better served with a single MERGE statement instead of a script.
I am trying to create a procedure in oracle, which upon calling from PL SQL block will create a view in database from which i will query data for a report. I am new to Oracle and need help with this code.
CREATE OR REPLACE PROCEDURE CREATE_VIEW
(
TO_DT IN Date
) AS
BEGIN
Create or Replace view BORR_DUR As
SELECT e."Deal_No", (Select "DeskName" From MM_S_DESK Where e."DeskCode" = MM_S_DESK."DeskCode") Facility, e."Remarks" Counterparty,
m."MaturityDate", m."PriRedem" Principal,
(select MAX("INTEREST_RATE") from MM_BOR_PLA_PAR d
WHERE e."Deal_No" = d."DEAL_NO" and "INTERESTINPUTDATE" <= to_dt)/100 yield, (m."MaturityDate" - To_date(to_dt,'dd/mm/yyyy')) Days_to_Mat,
Round(((m."MaturityDate" - To_date(to_dt,'dd/mm/yyyy'))/365)/ (1+((select MAX("INTEREST_RATE") from MM_BOR_PLA_PAR d
WHERE e."Deal_No" = d."DEAL_NO" and "INTERESTINPUTDATE" <= to_dt)/100)),4) MDURATION
FROM MM_T_BORROWING e, MM_T_BORROWING_PM_DETAIL m
Where e."DeskCode" in ('10','11','12','13') and e."Value_Date" <= to_dt and e."Maturity_Date" > to_dt and e."Status" not in ('C', 'D', 'Z', '0','X')
and e."Deal_No" = m."Deal_No" and "PriRedem" > '0' and m."MaturityDate" > to_dt;
END CREATE_VIEW;
On Compilation, i get PLS00103 error which says
encountered the symbol "Create" when expecting one of the
following....
Any help in solving this issue will be greatly appreciated.
When you want execute SQL statement which is dynamic you have to use EXECUTE IMMEDIATE statement
First , you don't need double quotes in fields name , after that you can try the query of the view and check if it runs without errors .
Put the create replace view... statement in an variable and in your procedure call :
BEGIN
EXECUTE IMMEDIATE view_string_variable ;
END;
/
I have a problem, I want to pass an array to a postgres function and use that array so returns values in a SELECT IN clause.
But It shows me this error:
An error occurred executing the SQL command :
SELECT
*
FROM
get_invtransferences_porders_fporders (300001300 , array [ 300093753 , 300094126 , 300093349 , 300093838 , 300094128 ] ...
ERROR: operator does not exist : integer = integer [ ]
Hint : No operator matches the name and type of arguments. You may need to add explicit type conversions .
Where : PL / pgSQL get_invtransferences_porders_fporders (numeric , integer []) function on line 8 FOR loop around rows of a SELECT
This is my function:
CREATE OR REPLACE FUNCTION public.get_invtransferences_porders_fporders(p_product_id numeric, P_INVTRANSFERENCES_IDS integer[])
RETURNS SETOF record
LANGUAGE plpgsql
AS
$body$
DECLARE
PORDER_PRODUCT RECORD;
COMPONENT RECORD;
COMPONENT2 RECORD;
COMPONENT3 RECORD;
BEGIN
FOR PORDER_PRODUCT IN (
SELECT
'porder' AS "operation_type"
,porders.id AS "porder_id"
,porders.user_id AS "porder_user_id"
,(SELECT name FROM users WHERE users.id = porders.user_id) AS "porder_user_name"
,porders.delivery_datetime AS "porder_delivery_datetime"
,porders_products.requested AS "product_requested"
,porders_products.produced AS "product_produced"
,products.code AS "product_code"
,products.NAME AS "product_name"
,(
SELECT products.name
FROM products
WHERE id = product_components.component_product_id
) AS "component_product_name"
,product_components.quantity AS "component_quantity"
,(
SELECT products.um_id
FROM products
WHERE id = product_components.component_product_id
) AS "component_um_id"
,(product_components.quantity / products.production_base) * porders_products.requested AS "total"
FROM porders
,porders_products
,products
,product_components
WHERE porders.id = porders_products.porder_id
AND porders_products.product_id = products.id
AND porders_products.product_id = product_components.product_id
AND porders.id IN (
SELECT rawm_audit_porders.porder_id
FROM rawm_audit_invtransferences
,rawm_audits
,rawm_audit_porders
WHERE rawm_audit_invtransferences.rawm_audits_id = rawm_audits.id
AND rawm_audit_porders.rawm_audits_id = rawm_audits.id
AND rawm_audit_invtransferences.invtransference_id IN
(
SELECT
invtransferences.id
FROM invtransferences
,invtransferences_products
,products
WHERE invtransferences.id = invtransferences_products.invtransference_id
AND products.id = invtransferences_products.product_id
AND invtransferences.id IN (P_INVTRANSFERENCES_IDS)
)
)
AND product_components.component_product_id = p_product_id
) LOOP
IF(PORDER_PRODUCT.porder_id IS NOT NULL)THEN
RETURN NEXT PORDER_PRODUCT;
END IF;
END LOOP;
RETURN;
END;
$body$
VOLATILE
COST 100
ROWS 1000
I think the error it here `invtransferences.id IN (P_INVTRANSFERENCES_IDS)
This is the select that calls the function:
SELECT
*
FROM
get_invtransferences_porders_fporders(300001300 , array[300093753, 300094126, 300093349, 300093838, 300094128] )
AS
(
"operation_type" varchar,
"porder_id" numeric,
"porder_user_id" numeric,
"porder_user_name" varchar,
"porder_delivery_datetime" date,
"product_requested" numeric,
"product_produced" numeric,
"product_code" varchar,
"product_name" varchar,
"component_product_name" varchar,
"component_quantity" numeric,
"component_um_id" varchar,
"total" numeric
)
ORDER BY
"porder_id";
EDIT: I remove the VARIADIC words that were in the function and in the select that calls the function
Can you hep me Please.
You don't need to declare your function as VARIADIC to pass array to it.
Try this
CREATE OR REPLACE FUNCTION xxx(
p_product_id integer,
P_INVTRANSFERENCES_IDS integer[])
RETURNS SETOF record
LANGUAGE sql
AS
$body$
select p_product_id = ANY(P_INVTRANSFERENCES_IDS)
$body$;
Note there is no VARIADIC before P_INVTRANSFERENCES_IDS.
You also need to use ANY instead of IN to check membership in array.
SqlFiddle
I don't know why this function always returns 0
CREATE OR REPLACE FUNCTION QTYDEPOT(
p_org_id IN NUMBER,
p_product_id IN NUMBER,
p_datefrom IN DATE,
p_dateto IN DATE)
RETURN NUMBER
AS
qty NUMBER;
BEGIN
SELECT COALESCE(SUM(C_InvoiceLine.qtyinvoiced), 0)
INTO qty
FROM C_InvoiceLine
INNER JOIN C_invoice
ON (c_invoiceline.C_INVOICE_ID = c_invoice.C_INVOICE_ID)
INNER JOIN C_BPartner
ON (c_invoice.C_BPARTNER_ID = c_bpartner.C_BPARTNER_ID)
WHERE C_BPartner.ISSALESREP = 'N'
AND C_BPartner.ISEMPLOYEE = 'N'
AND c_bpartner.ISCUSTOMER = 'Y'
AND c_invoiceline.AD_org_id = p_org_id
AND c_invoiceline.m_product_id= p_product_id
AND c_invoice.DateInvoiced BETWEEN p_datefrom AND p_dateto;
RETURN qty ;
END;
P.S : if I remove the date part of the close where
c_invoice.DateInvoiced BETWEEN p_datefrom AND p_dateto;
The function returns the real values.
I call it like this
SELECT
..
QTYDEPOT( 1000000, p.m_product_id,'7/7/2014','24/7/2014') as qtyDepot
try this:
SELECT
..
QTYDEPOT( 1000000, p.m_product_id,to_date('7/7/2014','dd/mm/yyyy'),to_date('24/7/2014','dd/mm/yyyy')) as qtyDepot
you have to specify the date format you are passing to the function,
hope this helps!
You have problems with DATE manipulation. I would suggest you to try calling your function like this :
SELECT
..
QTYDEPOT( 1000000, p.m_product_id,DATE('2014-07-07'),DATE('2014-07-24')) as qtyDepot
AS you can see, Oracle standard format is 'yyyy-mm-dd' I don't know if DATE(...) is needed, but I use to manipulate DATE like this: code is clearer.