Hi we have a SP in Bigquery with parameters and can expect length more than 100k
CREATE OR REPLACE PROCEDURE `Procname`(Criteria ARRAY<STRING>, Criteria2 STRING)
BEGIN
DECLARE Total INT64 DEFAULT 0;
DECLARE Invalid STRING ;
DECLARE InvalidCount INT64 ;
--#1 Get Total
SET Total = (SELECT SUM(numbers) FROM `table` WHERE Number IN UNNEST (Criteria));
--#2 Get List of numbers not found and return with comma seperated
execute immediate concat("SELECT STRING_AGG(Invalid) AS Invalid, count(Invalid)
as InvalidCount FROM (SELECT Invalid FROM(SELECT '' as Invalid union all select ",Criteria2," )v
EXCEPT DISTINCT SELECT Postcode FROM `table`)") INTO Invalid,InvalidCount;
SELECT Total, Invalid, InvalidCount;
END;
where my Criteria is an array of string and Criteria2 is a string'number1' union all select 'number2' union all select 'invalidnumber' union all select 'number3' union all select 'anotherinvalidnumber'
so as per above SP say with Criteria as ['number1', 'number2','invalidnumber','number3' ,'anotherinvalidnumber'] and Criteria2 as above the result should be
The problem is on Criteria2 parameter as soon as I increase this string length I got this error as
Message[Resources exceeded during query execution: Not enough
resources for query planning - too many subqueries or query is too
complex..
which is believe is due to a lot of select '' union all.
two questions
Any better way to write query #2 taking performance into account, we tried with temp tables but performance is an issue. If we go with this approach then Bigquery API is not liking it
How to get rid of this error
Related
I'm using Azure Synapse Analytics SQL Database. I'm aware I can't use selects in a scalar function (hence the error The SELECT statement is not allowed in user-defined functions). I'm looking for a work-around since this function does not rely on any tables. The goal is a scalar function that takes two delimited lists parameters, a delimiter parameter and returns 1 if the lists have one or more matching items, and returns 0 if no matches are found.
--The SELECT statement is not allowed in user-defined functions
CREATE FUNCTION util.get_lsts_have_mtch
(
#p_lst_1 VARCHAR(8000),
#p_lst_2 VARCHAR(8000),
#p_dlmtr CHAR(1)
)
RETURNS BIT
/***********************************************************************************************************
Description: This function returns 1 if two delimited lists have an item that exists in both lists.
--Example run:
SELECT util.get_lsts_have_mtch('AB|CD|EF|GH|IJ','UV|WX|CD|IJ|YZ','|') -- returns 1, there's a match
SELECT util.get_lsts_have_mtch('AB|CD|EF|GH|IJ','ST|UV|WX|YZ','|') -- returns 0, there's no match
**********************************************************************************************************/
AS
BEGIN
DECLARE #v_result BIT;
-- *** CAN THIS BE ACCOMPLISHED EFFICIENTLY WITHOUT ANY SELECTS? ***
SET #v_result = (SELECT CAST(CASE WHEN EXISTS (SELECT 1
FROM STRING_SPLIT(#p_lst_1, #p_dlmtr) AS tokens_1
INNER JOIN STRING_SPLIT(#p_lst_2, #p_dlmtr) AS tokens_2
ON tokens_1.value = tokens_2.value)
THEN 1
ELSE 0
END) AS BIT);
RETURN #v_result;
END;
I ditched the function and used this CASE statement. I wanted a function to join on that would be reusable. If anyone can find a function to do this, I will make that the accepted answer.
SELECT ...
FROM tbl_1
JOIN tbl_2
ON
-- wanted: util.get_lsts_have_mtch(tbl_1.my_lst, tbl_2.my_lst, '|') = 1
-- but settled for:
CASE WHEN EXISTS
(SELECT [value]
FROM STRING_SPLIT(tbl_1.my_lst, '|')
INTERSECT
SELECT [value]
FROM STRING_SPLIT(tbl_2.my_lst, '|'))
THEN 1
ELSE 0
END = 1
I am working on a sql database which will provide with data some grid. The grid will enable filtering, sorting and paging but also there is a strict requirement that users can enter free text to a text input above the grid for example
'Engine 1001 Requi' and that the result will contain only rows which in some columns contain all the pieces of the text. So one column may contain Engine, other column may contain 1001 and some other will contain Requi.
I created a technical column (let's call it myTechnicalColumn) in the table (let's call it myTable) which will be updated each time someone inserts or updates a row and it will contain all the values of all the columns combined and separated with space.
Now to use it with entity framework I decided to use a table valued function which accepts one parameter #searchQuery and it will handle it like this:
CREATE FUNCTION myFunctionName(#searchText NVARCHAR(MAX))
RETURNS #Result TABLE
( ... here come columns )
AS
BEGIN
DECLARE #searchToken TokenType
INSERT INTO #searchToken(token) SELECT value FROM STRING_SPLIT(#searchText,' ')
DECLARE #searchTextLength INT
SET #searchTextLength = (SELECT COUNT(*) FROM #searchToken)
INSERT INTO #Result
SELECT
... here come columns
FROM myTable
WHERE (SELECT COUNT(*) FROM #searchToken WHERE CHARINDEX(token, myTechnicalColumn) > 0) = #searchTextLength
RETURN;
END
Of course the solution works fine but it's kinda slow. Any hints how to improve its efficiency?
You can use an inline Table Valued Function, which should be quite a lot faster.
This would be a direct translation of your current code
CREATE FUNCTION myFunctionName(#searchText NVARCHAR(MAX))
RETURNS TABLE
AS RETURN
(
WITH searchText AS (
SELECT value token
FROM STRING_SPLIT(#searchText,' ') s(token)
)
SELECT
... here come columns
FROM myTable t
WHERE (
SELECT COUNT(*)
FROM searchText
WHERE CHARINDEX(s.token, t.myTechnicalColumn) > 0
) = (SELECT COUNT(*) FROM searchText)
);
GO
You are using a form of query called Relational Division Without Remainder and there are other ways to cut this cake:
CREATE FUNCTION myFunctionName(#searchText NVARCHAR(MAX))
RETURNS TABLE
AS RETURN
(
WITH searchText AS (
SELECT value token
FROM STRING_SPLIT(#searchText,' ') s(token)
)
SELECT
... here come columns
FROM myTable t
WHERE NOT EXISTS (
SELECT 1
FROM searchText
WHERE CHARINDEX(s.token, t.myTechnicalColumn) = 0
)
);
GO
This may be faster or slower depending on a number of factors, you need to test.
Since there is no data to test, i am not sure if the following will solve your issue:
-- Replace the last INSERT portion
INSERT INTO #Result
SELECT
... here come columns
FROM myTable T
JOIN #searchToken S ON CHARINDEX(S.token, T.myTechnicalColumn) > 0
In SQL HANA, I need to find how many times a given word is repeated in a string column whose values are delimited by "," and output it as a separate column.
Example, the string column contains:
ZN,ZN,ZS,ZQ
Expected result for "ZN":
2
You might find it acceptable to search only the string ZN by ignoring the fact that there's a comma.
You may count the number of occurrences of any substring by using the string function OCCURRENCES_REGEXPR:
SELECT OCCURRENCES_REGEXPR('(ZN)' IN STRINGCOLUMN) "occurrences_zn" FROM TABLE;
If you really want to clearly specify that ZN is to be searched as an entire word between commas or at the edges, then you may find a better regular expression (the question is then more about regular expressions and not SQL HANA, and you may find existing answers in Stack Overflow).
I can't remember where I found the trick, but in SQL Server, the following works like a charm:
DECLARE #myStringToSearch nvarchar(250) = 'ZN,ZN,ZS,ZQ'
DECLARE #searchValue nvarchar(5) = 'ZN'
SELECT (LEN(#myStringToSearch) - LEN(REPLACE(#myStringToSearch, #searchValue, ''))) / LEN(#searchValue)
The last line compares the length of the original string with the length of the same string, but this time replacing your search value (ZN) with a blank string. In our case, this would result in 4, because ZN is 2 characters, and it was removed twice. However, we're not interested in how many characters were removed, but in how many times the value was encountered, so we divide that result by the length of your search string (2).
Output of the query:
2
You could easily implement this as a DEFAULT constraint in your table, provided your search string is the same across every row.
I wrote one anonymous block in sql , which can be converted to HANA Table function and can be used to achieve expected result.
DO
BEGIN
DECLARE FULL_STRING VARCHAR(100);
DECLARE TRIM_STRING VARCHAR(100);
DECLARE VAL_STRING VARCHAR(100);
FULL_STRING ='ZN,ZN,ZS,ZQ';
FULL_STRING=CONCAT(FULL_STRING,',');
--SELECT :FULL_STRING FROM DUMMY;
VAL_STRING=SUBSTRING(:FULL_STRING,1,LOCATE(:FULL_STRING,',',1)-1);
VAR_TABLE=SELECT :VAL_STRING STRINGVAL FROM DUMMY;
TRIM_STRING=SUBSTRING(:FULL_STRING,LOCATE(:FULL_STRING,',',1)+1 ,LENGTH(:FULL_STRING));
--SELECT * FROM :VAR_TABLE;
--SELECT :TRIM_STRING FROM DUMMY;
WHILE :TRIM_STRING IS NOT NULL AND LENGTH(:TRIM_STRING)>0
DO
VAL_STRING=SUBSTRING(:TRIM_STRING,1,LOCATE(:TRIM_STRING,',',1)-1);
--SELECT :VAL_STRING FROM DUMMY;
VAR_TABLE=SELECT STRINGVAL FROM :VAR_TABLE
UNION ALL
SELECT :VAL_STRING FROM DUMMY;
TRIM_STRING=SUBSTRING(:TRIM_STRING,LOCATE(:TRIM_STRING,',',1)+1 ,LENGTH(:TRIM_STRING));
--i=i+1;
--SELECT :TRIM_STRING FROM DUMMY;
END WHILE ;
SELECT STRINGVAL,COUNT(STRINGVAL) FROM :VAR_TABLE GROUP BY STRINGVAL;
--SELECT :TRIM_STRING FROM DUMMY;
I have been mulling over a problem for the last few hours with a query which executes a decode statement as part of my WHERE CLAUSE.
The best Example I can come up with here is as follows:
--This variable is normally populated by a query.
--Debugger confirms this is set to 'US' with no leading or tailing characters
Str_MyVar varchar2(10) := 'US';
-- Query 1 returns 16 rows (incorrect)
Select *
From myTable, table2
Where myTable.value = decode(Str_MyVar, 'US', table2.value, 0)
-- Query 2 returns 1 rows (correct)
Select *
From myTable, table2
Where myTable.value = decode('US', 'US', table2.value, 0)
Now then, if I change the query which populates the Str_MyVar variable to instead convert the value 'US' to a 1 and all other values to a 0 and store it in a numeric variable things begin to work.
--This variable is normally populated by a query.
--Debugger confirms this is set to 1 with no leading or tailing characters
nbr_Myvar number := 1;
-- Query 1 returns 1 rows (correct)
Select *
From myTable, table2
Where myTable.value = decode(nbr_MyVar, 1, table2.value, 0);
This second example using a numeric data type for the variable in the decode works properly. The question is Why? Sure I've coded around the issue but I'd like to understand if this is an oracle bug or just a quirk of the DECODE function. Database is oracle 10.2.0.3
I have a function that accepts one parameter and returns a table/resultset. I want to set a field in a table to the first result of that recordset, passing in one of the table's other fields as the parameter. If that's too complicated in words, the query looks something like this:
UPDATE myTable
SET myField = (SELECT TOP 1 myFunctionField
FROM fn_doSomething(myOtherField)
WHERE someCondition = 'something')
WHERE someOtherCondition = 'somethingElse'
In this example, myField and myOtherField are fields in myTable, and myFunctionField is a field return by fn_doSomething. This seems logical to me, but I'm getting the following strange error:
'myOtherField' is not a recognized OPTIMIZER LOCK HINTS option.
Any idea what I'm doing wrong, and how I can accomplish this?
UPDATE:
Based on Anil Soman's answer, I realized that the function is expecting a string parameter and the field being passed is an integer. I'm not sure if this should be a problem as an explicit call to the function using an integer value works - e.g. fn_doSomething(12345) seems to automatically cast the number to an string. However, I tried to do an explicit cast:
UPDATE myTable
SET myField = (SELECT TOP 1 myFunctionField
FROM fn_doSomething(CAST(myOtherField AS varchar(1000)))
WHERE someCondition = 'something')
WHERE someOtherCondition = 'somethingElse'
Now I'm getting the following error:
Line 5: Incorrect syntax near '('.
I have never done anything like this so .... all the code I have seen uses a schema on the function name - so something like:
FROM dbo.fn_doSomething(myOtherField)
seems like a compiler bug in SQL 2000
try in Server 2005 and join the table-valued function using CROSS APPLY or OUTER APPLY
also try this, guru huys
CREATE FUNCTION FCN_pruebaChicaBorrame(#numerito int)
RETURNS #returnTable TABLE (numerito int)
AS
BEGIN
insert into #returnTable values(#numerito)
return
END
Select * from FCN_pruebaChicaBorrame(20)
Select col_1
from ( select 1 as col_1
union select 2
union select 3) as tablita
Select col_1, (select * from dbo.FCN_pruebaChicaBorrame(20) as fcnTable)
from ( select 1 as col_1
union select 2
union select 3) as tablita
Select col_1, (select * from dbo.FCN_pruebaChicaBorrame(col_1) as fcnTable)
from ( select 1 as col_1
union select 2
union select 3) as tablita
Select col_1, (select * from dbo.FCN_pruebaChicaBorrame(case when 1=1 then 20 else 21) as fcnTable)
from ( select 1 as col_1
union select 2
union select 3) as tablita
I searched on google for this error and one person talks about missing single quotes in search condition. Is that the case with your function code? link to related blog
It seems that (at least in SQL Server 2000) you can't pass a column value to a table valued function. I had to set up a scalar function to get around this.