MSSQL query with too many paramers in IN clause is very slow - sql

I'm having a query with IN clause with 2000 parameters of varchar2 type. I have applied required index as per execution plan.
It is working very slowly. I'm looking for an alternative solution. One way I found is creating a temporary table with these value and fetch using join. Is there any way other than this? I'm using spring data JPA/Criteria for queries in Java. Thanks in advance.


Bulk Load the values into a temprary table, either using SqlServerBulkCopy directly or using the useBulkCopyForBatchInsert.
Use a Table-Valued Parameter
Or (SQL 2016+) send the values as a JSON array. Just create a long string of the form
["Value1","Value2","Value3"]
and pass it as a parameter to a query like:
select *
from SomeTable
where SomeColumn in ( select value from openjson(#jsonValues) with (value varchar(200) '$') )
Or do the same thing with XML.
You can also use string_split, but that's also only available on SQL 2016+ and JSON is more robust.

Related

Is there any SQL query character limit while executing it by using the JDBC driver [duplicate]

I'm using the following code:
SELECT * FROM table
WHERE Col IN (123,123,222,....)
However, if I put more than ~3000 numbers in the IN clause, SQL throws an error.
Does anyone know if there's a size limit or anything similar?!!

Depending on the database engine you are using, there can be limits on the length of an instruction.
SQL Server has a very large limit:
http://msdn.microsoft.com/en-us/library/ms143432.aspx
ORACLE has a very easy to reach limit on the other side.
So, for large IN clauses, it's better to create a temp table, insert the values and do a JOIN. It works faster also.

There is a limit, but you can split your values into separate blocks of in()
Select *
From table
Where Col IN (123,123,222,....)
or Col IN (456,878,888,....)

Parameterize the query and pass the ids in using a Table Valued Parameter.
For example, define the following type:
CREATE TYPE IdTable AS TABLE (Id INT NOT NULL PRIMARY KEY)
Along with the following stored procedure:
CREATE PROCEDURE sp__Procedure_Name
#OrderIDs IdTable READONLY,
AS
SELECT *
FROM table
WHERE Col IN (SELECT Id FROM #OrderIDs)

Why not do a where IN a sub-select...
Pre-query into a temp table or something...
CREATE TABLE SomeTempTable AS
SELECT YourColumn
FROM SomeTable
WHERE UserPickedMultipleRecordsFromSomeListOrSomething
then...
SELECT * FROM OtherTable
WHERE YourColumn IN ( SELECT YourColumn FROM SomeTempTable )

Depending on your version, use a table valued parameter in 2008, or some approach described here:
Arrays and Lists in SQL Server 2005

For MS SQL 2016, passing ints into the in, it looks like it can handle close to 38,000 records.
select * from user where userId in (1,2,3,etc)

I solved this by simply using ranges
WHERE Col >= 123 AND Col <= 10000
then removed unwanted records in the specified range by looping in the application code. It worked well for me because I was looping the record anyway and ignoring couple of thousand records didn't make any difference.
Of course, this is not a universal solution but it could work for situation if most values within min and max are required.

You did not specify the database engine in question; in Oracle, an option is to use tuples like this:
SELECT * FROM table
WHERE (Col, 1) IN ((123,1),(123,1),(222,1),....)
This ugly hack only works in Oracle SQL, see https://asktom.oracle.com/pls/asktom/asktom.search?tag=limit-and-conversion-very-long-in-list-where-x-in#9538075800346844400
However, a much better option is to use stored procedures and pass the values as an array.

You can use tuples like this:
SELECT * FROM table
WHERE (Col, 1) IN ((123,1),(123,1),(222,1),....)
There are no restrictions on number of these. It compares pairs.

How to get a exact number from a String in sql server?

How to get Order Numbers(189,190) from following TrigParams field in SQL server 2014 Database.
TrigParams
{"OWLSObjKey":{"key":"OWLSObjKey","value":"189","type":null},"OWLSObjType":{"key":"OWLSObjType","value":"17","type":null},"ObjKey":{"key":"ObjKey","value":"189","type":null},"ObjType":{"key":"ObjType","value":"17","type":null}}
{"OWLSObjKey":{"key":"OWLSObjKey","value":"190","type":null},"OWLSObjType":{"key":"OWLSObjType","value":"17","type":null},"ObjKey":{"key":"ObjKey","value":"190","type":null},"ObjType":{"key":"ObjType","value":"17","type":null}}

You see from the other answers, that SQL-Server 2016+ with broad JSON support would be of great help. But without you are not lost. You can use string methods:
credits to Panagiotis Kanavos for the MCVE
declare #table table (trigparams nvarchar(2000))
insert into #table
values
('{"OWLSObjKey":{"key":"OWLSObjKey","value":"189","type":null},"OWLSObjType":{"key":"OWLSObjType","value":"17","type":null},"ObjKey":{"key":"ObjKey","value":"189","type":null},"ObjType":{"key":"ObjType","value":"17","type":null}}'),
('{"OWLSObjKey":{"key":"OWLSObjKey","value":"190","type":null},"OWLSObjType":{"key":"OWLSObjType","value":"17","type":null},"ObjKey":{"key":"ObjKey","value":"190","type":null},"ObjType":{"key":"ObjType","value":"17","type":null}}')
--This is the query
select LEFT(CutOff,CHARINDEX('"',CutOff)-1)
from #table t
CROSS APPLY(SELECT STUFF(t.trigparams,1,CHARINDEX('"value":"',t.trigparams)+8,'')) A(CutOff);
The idea in short:
Within the APPLY we will use STUFF() to write nothing over the first characters up to the number you are looking for (after the first occurance of "value":". This is returned as Column CutOff. We can now use LEFT() to pick the number only.

The question doesn't specify the server version, the table schema or whether the string represents a single value or the contents of two separate rows.
I'll just assume it's SQL Server 2016 which supports JSON and the text comes from two separate rows. I'll also assume the query only needs to return the data as single value.
In this case a simple call to JSON_VALUE('$.OWLSObjKey.value') will return the data. JSON_VALUE returns a single value from a well-formed JSON string :
declare #table table (trigparams nvarchar(2000))
insert into #table
values
('{"OWLSObjKey":{"key":"OWLSObjKey","value":"189","type":null},"OWLSObjType":{"key":"OWLSObjType","value":"17","type":null},"ObjKey":{"key":"ObjKey","value":"189","type":null},"ObjType":{"key":"ObjType","value":"17","type":null}}'),
('{"OWLSObjKey":{"key":"OWLSObjKey","value":"190","type":null},"OWLSObjType":{"key":"OWLSObjType","value":"17","type":null},"ObjKey":{"key":"ObjKey","value":"190","type":null},"ObjType":{"key":"ObjType","value":"17","type":null}}')
select JSON_VALUE(trigparams,'$.OWLSObjKey.value') As SomeKey
from #table
This returns :
SomeKey
189
190

Assuming you're using SQL Server 2016+, use OPENJSON:
SELECT O.OrderNumber
FROM (VALUES('{"OWLSObjKey":{"key":"OWLSObjKey","value":"189","type":null},"OWLSObjType":{"key":"OWLSObjType","value":"17","type":null},"ObjKey":{"key":"ObjKey","value":"189","type":null},"ObjType":{"key":"ObjType","value":"17","type":null}}'),
('{"OWLSObjKey":{"key":"OWLSObjKey","value":"190","type":null},"OWLSObjType":{"key":"OWLSObjType","value":"17","type":null},"ObjKey":{"key":"ObjKey","value":"190","type":null},"ObjType":{"key":"ObjType","value":"17","type":null}}')) V(TrigParams)
CROSS APPLY OPENJSON(V.TrigParams) WITH (OWLSObjKey nvarchar(MAX) AS JSON) OK
CROSS APPLY OPENJSON(OK.OWLSObjKey) WITH (OrderNumber int '$.value') O;
If not, then SQL Server is not your friend for this without using a CLR Function. If you can't use CLR, then I would suggest using a different tool to read the JSON value.

SQL Where clause from a file on Oracle

I have a requirement to construct an SQL that has a where clause which is expected to look into a file entries to be used in that clause.
SELECT DISTINCT '/',t_05.puid, ',', t_01.puid,'/', t_01.poriginal_file_name
FROM PWORKSPACEOBJECT t_02, PREF_LIST_0 t_03, PPOM_APPLICATION_OBJECT t_04, PDATASET t_05, PIMANFILE t_01
WHERE t_03.pvalu_0 = t_01.puid AND t_02.puid = t_03.puid AND t_03.puid = t_04.puid AND t_04.puid = t_05.puid AND t_02.puid IN ( 'izeVNXjf44e$yB',
'gWYRvN9044e$yB' );
The above is the SQL query. As you can see the IN clause has two different strings ( puids ) that are to be considered. But in my case, this list is like 50k entries long and would come from splunk and will be in a text file.
Sample output of the text file looks as belows:
'gWYRvN9044e$yB',
'DOZVpdOQ44e$yB',
'TlfVpdOQ44e$yB',
'wOWRehUc44e$yB',
'wyeRehUc44e$yB',
'w6URehUc44e$yB',
'wScRehUc44e$yB',
'yzXVNXjf44e$yB',
'guWRvN9044e$yB',
'QiYRehUc44e$yB',
'gycRvN9044e$yB'
I am not an SQL guru, but a quick google on this gave me a reference to OPENROWSET construct, which is not available on Oracle.
Can you please suggest some pointers on what can be done to circumvent the problem.
Thanks,
Pavan.

Consider using an external table, SQL Loader or perhaps loading the file into a table in the application layer and querying it normally.

I would recommend creating a Global Temporary table, adding the rows to that table, and then joining to your temp table.
How to create a temporary table in Oracle
Other options:
You could also use pipelined functions:
https://oracle-base.com/articles/misc/pipelined-table-functions
Or use the with as... construct to fold the data into the SQL. But that would create a long SQL statement.

dbms_lob.instr Takes a very very long time to fetch the data

We store clob data in the database in XML format and the data type is CLOB.
So to get this data we try the use the following query:
select message_xml
from table
where dbms_lob.instr(message_xml, 'A12345678') > 0
order by message_date;
But this query takes a very long time around an hour or more to give me the result.
Is there a better way to do this.
Kindly note that I am firing the query in Oracle 11g database.

Yes or No your question is to general. The answer depends from many factors.
You have two easy solution and one more complicated.
1) Trivial function-based index. Works only for one searching value.
create fb_trivial index on table( dbms_lob.instr(message_xml, 'A12345678'))
and query to utilize this index
select message_xml
from table
where dbms_lob.instr(message_xml, 'A12345678') > 0
order by message_date;
2) Extract function base-index.
create index fb_extract_index on table(extractvalue(xmltype(message_xml),'/path/to/your/value'))
or version with namaspaces
create index fb_extract_index on table(extractvalue(xmltype(message_xml),'/ns:path/ns:to/ns:your/ns:value', 'xmlns:ns="namespace:from:xml" '))
and query to utilize
select message_xml from table where extractvalue(xmltype(message_xml),'/path/to/your/value')) = 'A12345678'
3) For more sophisticated solution refer this article oracle-11g-xmlindex-part-1
and oracle documentation about xml indexing xmlIndex

what is the maximum value we can use with IN operator in sql [duplicate]

I'm using the following code:
SELECT * FROM table
WHERE Col IN (123,123,222,....)
However, if I put more than ~3000 numbers in the IN clause, SQL throws an error.
Does anyone know if there's a size limit or anything similar?!!

Depending on the database engine you are using, there can be limits on the length of an instruction.
SQL Server has a very large limit:
http://msdn.microsoft.com/en-us/library/ms143432.aspx
ORACLE has a very easy to reach limit on the other side.
So, for large IN clauses, it's better to create a temp table, insert the values and do a JOIN. It works faster also.

There is a limit, but you can split your values into separate blocks of in()
Select *
From table
Where Col IN (123,123,222,....)
or Col IN (456,878,888,....)

Parameterize the query and pass the ids in using a Table Valued Parameter.
For example, define the following type:
CREATE TYPE IdTable AS TABLE (Id INT NOT NULL PRIMARY KEY)
Along with the following stored procedure:
CREATE PROCEDURE sp__Procedure_Name
#OrderIDs IdTable READONLY,
AS
SELECT *
FROM table
WHERE Col IN (SELECT Id FROM #OrderIDs)

Why not do a where IN a sub-select...
Pre-query into a temp table or something...
CREATE TABLE SomeTempTable AS
SELECT YourColumn
FROM SomeTable
WHERE UserPickedMultipleRecordsFromSomeListOrSomething
then...
SELECT * FROM OtherTable
WHERE YourColumn IN ( SELECT YourColumn FROM SomeTempTable )

Depending on your version, use a table valued parameter in 2008, or some approach described here:
Arrays and Lists in SQL Server 2005

For MS SQL 2016, passing ints into the in, it looks like it can handle close to 38,000 records.
select * from user where userId in (1,2,3,etc)

I solved this by simply using ranges
WHERE Col >= 123 AND Col <= 10000
then removed unwanted records in the specified range by looping in the application code. It worked well for me because I was looping the record anyway and ignoring couple of thousand records didn't make any difference.
Of course, this is not a universal solution but it could work for situation if most values within min and max are required.

You did not specify the database engine in question; in Oracle, an option is to use tuples like this:
SELECT * FROM table
WHERE (Col, 1) IN ((123,1),(123,1),(222,1),....)
This ugly hack only works in Oracle SQL, see https://asktom.oracle.com/pls/asktom/asktom.search?tag=limit-and-conversion-very-long-in-list-where-x-in#9538075800346844400
However, a much better option is to use stored procedures and pass the values as an array.

You can use tuples like this:
SELECT * FROM table
WHERE (Col, 1) IN ((123,1),(123,1),(222,1),....)
There are no restrictions on number of these. It compares pairs.