Handling Ignoring Empty Values in Porting SQL Data - sql

I am in the process of porting over some data from a SQL environment to my MongoDB backend. I'm familiar with using a NULL check with your SELECT statement, like so:
SELECT * FROM clients WHERE note is not NULL ORDER BY id_number
... but in this old SQL database table I'm noticing a lot of rows where the value is not null, it's simply empty. It would be pointless to port these across. So what would it look like to prevent pulling those over -- in terms of the SELECT statement syntax?
To clarify "note" values are of type varchar. In JavaScript I would just guard against an empty string " ". Is there a way to do this with a SQL statement?

Something like that :
SELECT * FROM clients
WHERE note is not NULL
AND TRIM(note) <> ''
ORDER BY id_number;

Related

USE WHERE 1=1 SQL [duplicate]

Why would someone use WHERE 1=1 AND <conditions> in a SQL clause (Either SQL obtained through concatenated strings, either view definition)
I've seen somewhere that this would be used to protect against SQL Injection, but it seems very weird.
If there is injection WHERE 1 = 1 AND injected OR 1=1 would have the same result as injected OR 1=1.
Later edit: What about the usage in a view definition?
Thank you for your answers.
Still,
I don't understand why would someone use this construction for defining a view, or use it inside a stored procedure.
Take this for example:
CREATE VIEW vTest AS
SELECT FROM Table WHERE 1=1 AND table.Field=Value
If the list of conditions is not known at compile time and is instead built at run time, you don't have to worry about whether you have one or more than one condition. You can generate them all like:
and <condition>
and concatenate them all together. With the 1=1 at the start, the initial and has something to associate with.
I've never seen this used for any kind of injection protection, as you say it doesn't seem like it would help much. I have seen it used as an implementation convenience. The SQL query engine will end up ignoring the 1=1 so it should have no performance impact.
Just adding a example code to Greg's answer:
dim sqlstmt as new StringBuilder
sqlstmt.add("SELECT * FROM Products")
sqlstmt.add(" WHERE 1=1")
''// From now on you don't have to worry if you must
''// append AND or WHERE because you know the WHERE is there
If ProductCategoryID <> 0 then
sqlstmt.AppendFormat(" AND ProductCategoryID = {0}", trim(ProductCategoryID))
end if
If MinimunPrice > 0 then
sqlstmt.AppendFormat(" AND Price >= {0}", trim(MinimunPrice))
end if
I've seen it used when the number of conditions can be variable.
You can concatenate conditions using an " AND " string. Then, instead of counting the number of conditions you're passing in, you place a "WHERE 1=1" at the end of your stock SQL statement and throw on the concatenated conditions.
Basically, it saves you having to do a test for conditions and then add a "WHERE" string before them.
Seems like a lazy way to always know that your WHERE clause is already defined and allow you to keep adding conditions without having to check if it is the first one.
Indirectly Relevant: when 1=2 is used:
CREATE TABLE New_table_name
as
select *
FROM Old_table_name
WHERE 1 = 2;
this will create a new table with same schema as old table. (Very handy if you want to load some data for compares)
I found this pattern useful when I'm testing or double checking things on the database, so I can very quickly comment other conditions:
CREATE VIEW vTest AS
SELECT FROM Table WHERE 1=1
AND Table.Field=Value
AND Table.IsValid=true
turns into:
CREATE VIEW vTest AS
SELECT FROM Table WHERE 1=1
--AND Table.Field=Value
--AND Table.IsValid=true
1 = 1 expression is commonly used in generated sql code. This expression can simplify sql generating code reducing number of conditional statements.
Actually, I've seen this sort of thing used in BIRT reports. The query passed to the BIRT runtime is of the form:
select a,b,c from t where a = ?
and the '?' is replaced at runtime by an actual parameter value selected from a drop-down box. The choices in the drop-down are given by:
select distinct a from t
union all
select '*' from sysibm.sysdummy1
so that you get all possible values plus "*". If the user selects "*" from the drop down box (meaning all values of a should be selected), the query has to be modified (by Javascript) before being run.
Since the "?" is a positional parameter and MUST remain there for other things to work, the Javascript modifies the query to be:
select a,b,c from t where ((a = ?) or (1==1))
That basically removes the effect of the where clause while still leaving the positional parameter in place.
I've also seen the AND case used by lazy coders whilst dynamically creating an SQL query.
Say you have to dynamically create a query that starts with select * from t and checks:
the name is Bob; and
the salary is > $20,000
some people would add the first with a WHERE and subsequent ones with an AND thus:
select * from t where name = 'Bob' and salary > 20000
Lazy programmers (and that's not necessarily a bad trait) wouldn't distinguish between the added conditions, they'd start with select * from t where 1=1 and just add AND clauses after that.
select * from t where 1=1 and name = 'Bob' and salary > 20000
where 1=0, This is done to check if the table exists. Don't know why 1=1 is used.
While I can see that 1=1 would be useful for generated SQL, a technique I use in PHP is to create an array of clauses and then do
implode (" AND ", $clauses);
thus avoiding the problem of having a leading or trailing AND. Obviously this is only useful if you know that you are going to have at least one clause!
Here's a closely related example: using a SQL MERGE statement to update the target tabled using all values from the source table where there is no common attribute on which to join on e.g.
MERGE INTO Circles
USING
(
SELECT pi
FROM Constants
) AS SourceTable
ON 1 = 1
WHEN MATCHED THEN
UPDATE
SET circumference = 2 * SourceTable.pi * radius;
If you came here searching for WHERE 1, note that WHERE 1 and WHERE 1=1 are identical. WHERE 1 is used rarely because some database systems reject it considering WHERE 1 not really being boolean.
Why would someone use WHERE 1=1 AND <proper conditions>
I've seen homespun frameworks do stuff like this (blush), as this allows lazy parsing practices to be applied to both the WHERE and AND Sql keywords.
For example (I'm using C# as an example here), consider the conditional parsing of the following predicates in a Sql query string builder:
var sqlQuery = "SELECT * FROM FOOS WHERE 1 = 1"
if (shouldFilterForBars)
{
sqlQuery = sqlQuery + " AND Bars > 3";
}
if (shouldFilterForBaz)
{
sqlQuery = sqlQuery + " AND Baz < 12";
}
The "benefit" of WHERE 1 = 1 means that no special code is needed:
For AND - whether zero, one or both predicates (Bars and Baz's) should be applied, which would determine whether the first AND is required. Since we already have at least one predicate with the 1 = 1, it means AND is always OK.
For no predicates at all - In the case where there are ZERO predicates, then the WHERE must be dropped. But again, we can be lazy, because we are again guarantee of at least one predicate.
This is obviously a bad idea and would recommend using an established data access framework or ORM for parsing optional and conditional predicates in this way.
Having review all the answers i decided to perform some experiment like
SELECT
*
FROM MyTable
WHERE 1=1
Then i checked with other numbers
WHERE 2=2
WHERE 10=10
WHERE 99=99
ect
Having done all the checks, the query run town is the same. even without the where clause. I am not a fan of the syntax
This is useful in a case where you have to use dynamic query in which in where
clause you have to append some filter options. Like if you include options 0 for status is inactive, 1 for active. Based from the options, there is only two available options(0 and 1) but if you want to display All records, it is handy to include in where close 1=1.
See below sample:
Declare #SearchValue varchar(8)
Declare #SQLQuery varchar(max) = '
Select [FirstName]
,[LastName]
,[MiddleName]
,[BirthDate]
,Case
when [Status] = 0 then ''Inactive''
when [Status] = 1 then ''Active''
end as [Status]'
Declare #SearchOption nvarchar(100)
If (#SearchValue = 'Active')
Begin
Set #SearchOption = ' Where a.[Status] = 1'
End
If (#SearchValue = 'Inactive')
Begin
Set #SearchOption = ' Where a.[Status] = 0'
End
If (#SearchValue = 'All')
Begin
Set #SearchOption = ' Where 1=1'
End
Set #SQLQuery = #SQLQuery + #SearchOption
Exec(#SQLQuery);
Saw this in production code and asked seniors for help.
Their answer:
-We use 1=1 so when we have to add a new condition we can just type
and <condition>
and get on with it.
I do this usually when I am building dynamic SQL for a report which has many dropdown values a user can select. Since the user may or may not select the values from each dropdown, we end up getting a hard time figuring out which condition was the first where clause. So we pad up the query with a where 1=1 in the end and add all where clauses after that.
Something like
select column1, column2 from my table where 1=1 {name} {age};
Then we would build the where clause like this and pass it as a parameter value
string name_whereClause= ddlName.SelectedIndex > 0 ? "AND name ='"+ ddlName.SelectedValue+ "'" : "";
As the where clause selection are unknown to us at runtime, so this helps us a great deal in finding whether to include an 'AND' or 'WHERE'.
Making "where 1=1" the standard for all your queries also makes it trivially easy to validate the sql by replacing it with where 1 = 0, handy when you have batches of commands/files.
Also makes it trivially easy to find the end of the end of the from/join section of any query. Even queries with sub-queries if properly indented.
I first came across this back with ADO and classic asp, the answer i got was: performance.
if you do a straight
Select * from tablename
and pass that in as an sql command/text you will get a noticeable performance increase with the
Where 1=1
added, it was a visible difference. something to do with table headers being returned as soon as the first condition is met, or some other craziness, anyway, it did speed things up.
Using a predicate like 1=1 is a normal hint sometimes used to force the access plan to use or not use an index scan. The reason why this is used is when you are using a multi-nested joined query with many predicates in the where clause where sometimes even using all of the indexes causes the access plan to read each table - a full table scan. This is just 1 of many hints used by DBAs to trick a dbms into using a more efficient path. Just don't throw one in; you need a dba to analyze the query since it doesn't always work.
Here is a use case... however I am not too concerned with the technicalities of why I should or not use 1 = 1.
I am writing a function, using pyodbc to retrieve some data from SQL Server. I was looking for a way to force a filler after the where keyword in my code. This was a great suggestion indeed:
if _where == '': _where = '1=1'
...
...
...
cur.execute(f'select {predicate} from {table_name} where {_where}')
The reason is because I could not implement the keyword 'where' together inside the _where clause variable. So, I think using any dummy condition that evaluates to true would do as a filler.

What is SQL Server 2005 expected behavior of insert into table select query where one of the columns attempts to convert a null value

We have a statement in some legacy SQL Server 2005 code like
insert into myTable
select distinct
wherefield1,
wherefield2,
anotherfield,
convert(numeric(10,2), varcharfield1),
convert(numeric(10,2), varcharfield2),
convert(numeric(10,2), varcharfield3),
convert(datetime, varcharfield4),
otherfields
from myStagingTable
where insertflag='true'
and wherefield1 = #wherevalue1
and wherefield2 = #wherevalue2
Earlier in the code, a variable is set to determine whether varcharfield1 or varcharfield2 is null, and the insert is programmed to execute as long as one of them is not null.
We know that if varcharfield1, varcharfield2, or varcharfield3 is a nonnumeric character string, an exception will be thrown and the insert will not occur. But I am perplexed by the behavior when one of these variables is null, as it often is. Actually, it is always the case that one of these values is null. But it seems that the insertion does take place. It looks like the legacy code relies on this to prevent only insertion of nonnumeric character data, while allowing insertion of null or empty values (in an earlier step, all empty strings in these fields of myStagingTable are replaced with null values).
This has been running on a Production SQL Server 2005 instance with all default settings for a number of years. Is this behavior we can rely on if we upgrade to a newer version of SQL Server?
Thanks,
Rebeccah
conversion of NULL to anything is still NULL. If the column allows NULL, that's what you'll get. If the column is not nullable, it will fail.
You can see this yourself without even doing an INSERT. Just run this:
SELECT CONVERT(numeric(10,2), NULL)
and note how it produces a NULL result. Then run this:
SELECT CONVERT(numeric(10,2), 'x')
and note how it throws an error message instead of returning anything.

assigning IS NULL to a string before querying DB

I have an excel spreadsheet with 15 or so fields. Wat I'm doing is, i open it, grab a row of data, then check each row for a value. Then using a few criteria I go look up to see if this Client value is already in the DB. As Example
Some of the fields mind be empty. Basically after checking some of the fields, I use them to check if that record exists already in DB. the problem arises when some fields are empty in which case when I query sql server it looks something like...
Select * from TblA where Company='Apple' and CompanyAdd ='Cupertino' and City=''
Because City = '' - it doesnt not find anything in SQL. The only thing that works is
and City is NULL
How am I able to programmatically assign that to a variable like CITY?
it is a string and the field in SQL is varchar
EDIT:
I want to be able to do something like this..... (as example)
if city = "" then
'I need this here to be so that....
city IS NULL
End if
So that when I query db it looks something like...
Select count(*) from TblA where City is Null
Is somethng like that possible?
You can use COALESCE for this purpose.
SELECT *
FROM TblA
WHERE COALESCE(Company, '')='Apple'
AND COALESCE(CompanyAdd, '') = 'Cupertino'
AND COALESCE(City, '') = ''
Keep in mind that the performance of this query will most likely not be stellar.

returning emptystring if columnn name not exists

I have a very simple select statement like
SELECT Column-Name
FROM Object
WHERE ID = 123
where the Column-Name is dynamically generated. Is there a possibility to get an empty string if column not exists?
This is a huge hack for SQL Server
SQL is batch based: when submitted to the database engine, the whole batch is parse and a plan compiled. If a column or object does not exist, you'll get an error.
You can test the metadata to see if it exists (COLUMNPROPERTY) then use EXEC to have the SELECT in an separate batch
IF COLUMNPROPERTY(OBJECT_ID('Object'), 'Column-Name', 'ColumnID') IS NOT NULL
EXEC ('SELECT Column-Name FROM Object WHERE ID = 123')
ELSE
SELECT '' AS Column-Name;
Personally, I'd never expect this to be in production code or running on my database servers.
Why not do a desc object or something similar and validate if the column name exists before firing the query?
Otherwise you will end up doing funny things to work around the sql error

Considering spaces in a SQL row as null

I was suppose to get all data from the table where the column "Address" is not null
so I made a statement that look like this...
Select * from Table where Address is not null
Unfortunately, there are rows in "Address" column that has spaces so SQL cannot consider it as Null
How can I display rows where Address is not null?
Thanks :)
Most database systems have a NULLIF() function. It was defined together with COALESCE() in the ANSI SQL-99 standard if not earlier. It is implemented in at least SQL Server, Oracle, PostgreSQL, MySQL, SQLite, DB2, Firebird.
Select * from Table where NULLIF(Address,'') is not null
But for me, I like this more
Select * from Table where Address > ''
It kills nulls and empty strings in one go. It will even exclude strings that are made up entirely of spaces ('', ' ', etc). It also retains SARGability.