Can CONTAINS Handle Table Columns that have NULL values? - sql

I'm trying to do some query performance and I've been looking into applying a FULLTEXT INDEX as compared to using LIKE.
I have a database w/person information, i.e. FirstName, LastName, etc. where some of those columns are null.
When I try to use CONTAINS object, I'm getting this error message;
Null or empty full-text predicate
I've put in code to ensure the passed in #FirstName field is not null, and if a null was passed in then I've set it to ' "" ' like I've read elsewhere. I still get the error message. I'm starting to wonder if it's a problem because the index has columns that are null.
Does anyone know if that is the case, I've looked everywhere and I cannot find any information on this.
Here is the query. The #FirstNameLocal and #LastNameLocal are passed into this stored procedure. In this example, FirstName and/or LastName field data in the PersonTable could be null. I'm using SQL Server 2012 w/SP1. "Try to use" means when I try to run the query in SQL Management Studio.
SELECT DISTINCT LastName, FirstName, MiddleName, PersonId
FROM PersonTable
Where IsDeleted = 0
AND CONTAINS(FirstName, #FirstNameLocal)
AND CONTAINS(LastName, #LastNameLocal)

You could check whether your variable IS NULL or equals an empty string first. I think though that this will partially negate the benefit of full-text indexes and you should consider handling this differently (for instance with dynamic SQL).
SELECT DISTINCT LastName, FirstName, MiddleName, PersonId
FROM PersonTable
Where IsDeleted = 0
AND (ISNULL(#FirstNameLocal,'')='' OR CONTAINS(FirstName, #FirstNameLocal))
AND (ISNULL(#LastNameLocal,'')='' OR CONTAINS(LastName, #LastNameLocal));
If you would do this with dynamic SQL it would look something like this:
DECLARE #stmt NVARCHAR(4000);
SET #stmt=
N'SELECT DISTINCT LastName, FirstName, MiddleName, PersonId ' +
N'FROM PersonTable ' +
N'Where IsDeleted = 0' +
CASE WHEN ISNULL(#FirstNameLocal,'')=''
THEN N''
ELSE N' AND CONTAINS(FirstName, #FirstNameLocal)'
END +
CASE WHEN ISNULL(#LastNameLocal,'')=''
THEN N''
ELSE N' AND CONTAINS(LastName, #LastNameLocal)'
END + ';'
EXEC sp_executesql
#stmt,
N'#FirstNameLocal VARCHAR(256),#LastNameLocal VARCHAR(256)',
#FirstNameLocal,
#LastNameLocal;
The downside is that the statement is compiled again each time this script is run, but this will be small compared to the benefit of full-text indexes that can be used in this statement.
Always compare the Actual Execution Plans to choose which version is best.

Related

Why does MS SQL Server return results with 'LIKE' query but returns empty set using '='

I'm using utilizing MS SQL Server 2017. In this example, I have a table [myTable] with one entity full name (varchar(255).
Inside of this table, I have one record bulk imported with the following parameters
(FIELDTERMINATOR = '~',
ROWTERMINATOR = '0x0A' )
The following query returns the single record:
SELECT [full name]
FROM myTable
WHERE [full name] LIKE '%LastName%'
However, the following query does not:
SELECT [full name]
FROM myTable
WHERE [full name] = 'Firstname Lastname'
I have verified the record exists, I have copied the value of the entity to make sure there was no funny business going on with the data. It comes up clean, I even tried this:
SELECT [full name]
FROM myTable
WHERE [full name] = 'Firstname Lastname '
to make sure there wasn't a random space at the end of the column. What would cause an issue like this?
I have a handy script I use to identify invisible characters; it gives you every character in the string but as a separate row per character.
DECLARE #nstring NVARCHAR(1000)
select top 1 #nstring = [full name]
FROM myTable
WHERE [full name] LIKE '%LastName%'
print #nstring
DECLARE #position INT
SET #position = 1
DECLARE #CharList TABLE (
Position INT,
UnicodeChar NVARCHAR(1),
UnicodeValue INT
)
WHILE #position <= DATALENGTH(#nstring)
BEGIN
INSERT #CharList
SELECT #position as Position
,CONVERT(nchar(1),SUBSTRING(#nstring, #position, 1)) as UnicodeChar
,UNICODE(SUBSTRING(#nstring, #position, 1)) as UnicodeValue
SET #position = #position + 1
END
SELECT * FROM #CharList
The = operator works only when it got exact match
but The LIKE operator is used in a to search for a specified pattern in of a data.
so in your case your compare value is not directly matched with the column values that's why it does not returned any rows but in times like it matched pattern so it returned rows
LIKE as the name itself suggests, is mainly used to compare data of some form that form is mostly regular expressions
whereas,
the = finds only the exact match not that it should be of some form rather the data on left of = should be a crystal clear copy of whats on the right of it.
Eg.
As per your above input sample
..where fullname LIKE %FirstName%. This would mean in general, select data Where fullname has firstname in it
If I replace this %
...where fullname LIKE FirstName
then it would be same as
...where fullname = FirstName coz here it wants exact match not data of some form
There is a tip given on mssqltips that says you can use SSIS. The article also explains this issue occurring with the ROWTERMINATOR. With that said, I would create an SSIS project that contains a package that does the bulk data insert from the data source. If this is a repeated process it can be scheduled or quickly tested in the SSMS.
If you are new to SSIS or are not familiar with SSIS, refere here: SQL Server Integration Services

sql full text search with nullable parameter

We would like to work with full-text search but keep the incoming parameters in our procedure null-able. In this example #Street can have a value like Heuvel or can be NULL.
Working this out in a query is something that doesn't work.
heuvel* gives all the rows that are expected.
* gives zero rows while other threads on forums say that it will give all rows
street = isnull(#street, street) gives all rows
Knowing all the above I would think this query would work. It does but it's VERY slow.
When I execute the or clause separately then the query is fast.
Declare #Street nvarchar(50)= '"heuvel*"'
Declare #InnerStreet nvarchar(50) = '"'+ isnull(#Street, '') +'*"';
SELECT *
FROM Address
WHERE street = isnull(#street, street) or CONTAINS(street, #InnerStreet)
Now the question is, how to work with full-text search and null-able parameters?
Try this. #street is either NULL or has a value.
declare #streetWildcard nvarchar(50) = '"' + #street + '*"'
SELECT *
FROM Address
WHERE (#street is NULL) or CONTAINS(street, #streetWildcard)
If the performance is still bad then the execution plan may not be short circuiting the WHERE condition. In that case, try this instead.
if (#street is NULL) begin
SELECT *
FROM Address
end
else begin
declare #streetWildcard nvarchar(50) = '"' + #street + '*"'
SELECT *
FROM Address
WHERE CONTAINS(street, #streetWildcard)
end

Conditional where clause?

I want to apply the conditional where clause That is if my barcode parameter comes null then i want to fetch all the records and if it comes with value then i want to fetch only matching records for the second part i am able to fetch the matching records but i am stuck at fetching the all records in case of null value i have tried as below ,
SELECT item
FROM tempTable
WHERE
((ISNULL(#barcode,0)=1)
// but this is not fetching all the records if barcode is null
OR
ISNULL(#barcode,0!= 1 AND tempTable.barcode LIKE #barcode+'%'))
//THis is working perfect
so any help will be great
I might have misunderstood what you ask, but the logic OR operator might help:
SELECT item
FROM tempTable
WHERE
#barcode IS NULL OR tempTable.barcode LIKE #barcode+'%'
If #barcode is NULL, it returns all the records, and when it is not NULL, it returns all of the records that fulfill the condition LIKE #barcode+'%'
Important
Also, bear in mind that using the OR operator can seemingly cause funny results when used with several complex conditions AND-ed together, and not enclosed properly in braces:
<A> AND <B> AND <C> OR <D> AND <E> AND <F>
Should most likely actually be formulated as:
(<A> AND <B> AND <C>) OR (<D> AND <E> AND <F>)
Remember, the parser does not know what you want to achieve, you have to describe your intents properly...
I think you could simplify it to:
SELECT item
FROM tempTable
WHERE #barcode IS NULL OR tempTable.barcode LIKE #barcode+'%'
so when #barcode is null you'll get everything - i.e. the Like part of the where won't need to execute. If #barcode has a value then the Like will be executed.
If the barcode field is non-null, then this is the method I would use -
SELECT item
FROM tempTable
WHERE barcode like isnull(#barcode, barcode) + '%'
If #barcode is null all records are returned and if it is non null then only matching records are returned.
If the barcode field is nullable then -
SELECT item
FROM tempTable
WHERE isnull(barcode, '') like isnull(#barcode, isnull(barcode, '')) + '%'
Same as the first but here we convert the null values in the barcode field to blank strings before doing the compare.
An alternate answer and an attempt at the bounty
declare #barcode nvarchar(10) -- chose nvarchar not necessarily should be nvarchar
select #barcode= NULL
--select #barcode='XYZ'
if #barcode is null
select item from temptable;
else
select item from temptable where temptable.barcode like #barcode+'%';
If I have to do this, I would have done like
SELECT item
FROM tempTable
WHERE
( ( ISNULL(#barcode,'') <> '') AND ( tempTable.barcode LIKE #barcode+'%' ) )
( ISNULL(#barcode,'') <> '') would also check if the variable is blank then it should not return anything. But if you just check for null, then in case when the #barcode is blank, you will be getting all item selected from the tempTable.
If column barcode would be non-nullable, you could greatly simplify the query.
This is based on the fact that the pattern '%' matches any string; even an empty (i.e. zero-length) string.
Consequently, the following WHERE clause matches all records:
WHERE barcode LIKE '%'
You may notice that this has a very close resemblance to the WHERE clause you are using to filter records on a specific barcode:
WHERE barcode LIKE #barcode + '%'
In fact, they are so similar that we may as well use a single WHERE clause for both cases; after all, '' + '%' equals '%'!
IF #barcode IS NULL SET #barcode = ''
SELECT item FROM tempTable WHERE barcode LIKE #barcode + '%'
There is an even shorter version, which preserves the original value of #barcode:
SELECT item FROM tempTable WHERE barcode LIKE ISNULL(#barcode, '') + '%'
As mentioned earlier, this works only if column barcode is non-nullable.
If column barcode is nullable (and you are genuinely interested in records where barcode IS NULL), then the following query might work for you:
SELECT item FROM tempTable
WHERE ISNULL(barcode, '') LIKE ISNULL(#barcode, '') + '%'
However, this version has two disadvantages:
It may perform much slower, because the query optimizer may not benefit from an index on column barcode.
If #barcode = '', then it will match not only the non-null barcodes, but also the records with barcode IS NULL; whether this is acceptable, is up to you.
One last simplification: you may want to reach consensus with the outside world that they should set #barcode = '' instead of NULL to retrieve all records. Then you could replace ISNULL(#barcode, '') by #barcode.
Apart of ppeterka's solution (which will causes an Index/Table Scan) there are at least three other solutions. These solutions could use an Index Seek if #barcode isn't NULL and, also, if there is an index on barcode column:
Solution #2: The execution plan isn't cached and reused:
SELECT item
FROM tempTable
WHERE #barcode IS NULL OR tempTable.barcode LIKE #barcode+'%'
OPTION(RECOMPILE);
Solution #3: The execution plan is cached (it can be used if the num. of optional parameters is small):
IF #barcode IS NULL
SELECT item
FROM tempTable;
ELSE
SELECT item
FROM tempTable
WHERE tempTable.barcode LIKE #barcode+'%';
Solution #4: The execution plans are cached (it can be used if the num. of optional parameters is high):
DECLARE #SqlStatement NVARCHAR(MAX);
SET #SqlStatement = N'
SELECT item
FROM tempTable
WHERE 1=1 '
+ CASE WHEN #barcode IS NULL THEN N'' ELSE N'AND tempTable.barcode LIKE #pBarcode+''%''; ' END;
-- + CASE WHEN #anotherparam IS NULL THEN N'' ELSE 'AND ...' END ;
EXEC sp_executesql #SqlStatement, N'#pBarcode VARCHAR(10)', #pBarcode = #barcode;
Note: Use the proper type and max. lenght/precision & scale for #pBarcode parameter.
IF #barcode is null
begin
SELECT item FROM tempTable
end
else
SELECT item FROM tempTable where tempTable.barcode LIKE #barcode+'%'

Build dynamic query according to variable value

I am using SQL Server 2008 R2.
I am facing problem while fetching records from database in Role Based Access.
I have a Table let us say TableA. I have 5 Columns in it i.e, ID (primary key), FirstName, LastName, RegistrationNumber, EmployeeIdent.
All the columns except ID are of Varchar type.
I have a stored procedure to search the records. I am passing string to it and finding matching records from all five columns.
Now, the problem I am facing is in Role Based Access.
The requirement is when user is of "Admin" type then find matching records from all five columns but when is of "naive" user type then search matching records from only ID column.
I have passed the variable #userType to stored procedure from which I can determine the user type.
One way to resolve this problem is If Condition. Like if (#userType) = 'Admin' then some query Else Some other.
But I don't want to write query in If Condition.
Another way to resolve this is to store query in varchar type of variable and then execute is using EXEC (), but I have heard that it creates more overhead on server (I am not sure about it).
So any other way to fulfill this requirement?
The query to search records is
DECLARE #SearchText VARCHAR(MAX)= ''
SELECT *
FROM TableA
WHERE
Convert(varchar,ID) LIKE CASE WHEN LEN(#SearchText) = 0 THEN Convert(varchar,ID) ELSE '%'+ ISNULL(#SearchText,'0') + '%' END OR
FirstName LIKE CASE WHEN LEN(#SearchText) = 0 THEN FirstName ELSE '%'+ ISNULL(#SearchText,'') + '%' END OR
LastName LIKE CASE WHEN LEN(#SearchText) = 0 THEN LastName ELSE '%'+ ISNULL(#SearchText,'') + '%' END OR
RegistrationNumber LIKE CASE WHEN LEN(#SearchText) = 0 THEN RegistrationNumber ELSE '%'+ ISNULL(#SearchText,'') + '%' END OR
EmployeeIdent LIKE CASE WHEN LEN(#SearchText) = 0 THEN EmployeeIdent ELSE '%'+ ISNULL(#SearchText,'') + '%' END
SELECT
*
FROM TABLEA
WHERE
(#userType = 'admin'
--add any other criteria for admin type here
) or (
#userType = 'naive'
--add any other criteria for naive type here
)

Dynamic SQL - Search Query - Variable Number of Keywords

We are trying to update our classic asp search engine to protect it from SQL injection. We have a VB 6 function which builds a query dynamically by concatenating a query together based on the various search parameters. We have converted this to a stored procedure using dynamic sql for all parameters except for the keywords.
The problem with keywords is that there are a variable number words supplied by the user and we want to search several columns for each keyword. Since we cannot create a separate parameter for each keyword, how can we build a safe query?
Example:
#CustomerId AS INT
#Keywords AS NVARCHAR(MAX)
#sql = 'SELECT event_name FROM calendar WHERE customer_id = #CustomerId '
--(loop through each keyword passed in and concatenate)
#sql = #sql + 'AND (event_name LIKE ''%' + #Keywords + '%'' OR event_details LIKE ''%' + #Keywords + '%'')'
EXEC sp_executesql #sql N'#CustomerId INT, #CustomerId = #CustomerId
What is the best way to handle this and maintaining protection from SQL injection?
You may not like to hear this, but it might be better for you to go back to dynamically constructing your SQL query in code before issuing against the database. If you use parameter placeholders in the SQL string you get the protection against SQL injection attacks.
Example:
string sql = "SELECT Name, Title FROM Staff WHERE UserName=#UserId";
using (SqlCommand cmd = new SqlCommand(sql))
{
cmd.Parameters.Add("#UserId", SqlType.VarChar).Value = "smithj";
You can build the SQL string depending on the set of columns you need to query and then add the parameter values once the string is complete. This is a bit of a pain to do, but I think it is much easier than having really complicated TSQL which unpicks lots of possible permutations of possible inputs.
You have 3 options here.
Use a function that converts lists tables and join into it. So you will have something like this.
SELECT *
FROM calendar c
JOIN dbo.fnListToTable(#Keywords) k
ON c.keyword = k.keyword
Have a fixed set of params, and only allow the maximum of N keywords to be searched on
CREATE PROC spTest
#Keyword1 varchar(100),
#Keyword2 varchar(100),
....
Write an escaping string function in TSQL and escape your keywords.
Unless you need it, you could simply strip out any character that's not in [a-zA-Z ] - most of those things won't be in searches and you should not be able to be injected that way, nor do you have to worry about keywords or anything like that. If you allow quotes, however, you will need to be more careful.
Similar to sambo99's #1, you can insert the keywords into a temporary table or table variable and join to it (even using wildcards) without danger of injection:
This isn't really dynamic:
SELECT DISTINCT event_name
FROM calendar
INNER JOIN #keywords
ON event_name LIKE '%' + #keywords.keyword + '%'
OR event_description LIKE '%' + #keywords.keyword + '%'
You can actually generate an SP with a large number of parameters instead of coding it by hand (set the defaults to '' or NULL depending on your preference in coding your searches). If you found you needed more parameters, it would be simple to increase the number of parameters it generated.
You can move the search to a full-text index outside the database like Lucene and then use the Lucene results to pull the matching database rows.
You can try this:
SELECT * FROM [tablename] WHERE LIKE % +keyword%