I'm working on a vb.net application which imports from an Excel spreadsheet.
If rdr.HasRows Then
Do While rdr.Read()
If rdr.GetValue(0).Equals(System.DBNull.Value) Then
Return Nothing
Else
Return rdr.GetValue(0)
End If
Loop
Else
I was using string value to store the double values and when preparing the database statement I'd use this code:
If (LastDayAverage = Nothing) Then
command.Parameters.AddWithValue("#WF_LAST_DAY_TAG", System.DBNull.Value)
Else
command.Parameters.AddWithValue("#WF_LAST_DAY_TAG", Convert.ToDecimal(LastDayAverage))
End If
I now have some data with quite a few decimal places and the data was put into the string variable in scientific notation, so this seems to be the wrong approach. It didn't seem right using the string variable to begin with.
If I use a double or decimal type variable, the blank excel values come across as 0.0.
How can I preserve the blank values?
Note: I have tried
Variable as Nullabe(Of Double)
But when passing the value to the SQL insert I get: "Nullable object must have a value."
Solution:
Fixed by changing the datatype of the parameter in the sub I was calling and then using Variable.HasValue to do the conditional DBNull insert.
I don't know which API you're using to do database inserts, but with many of them, including ADO.NET, the proper way to insert nulls is to use DBNull.Value. So my recommendation is that you use Nullable(Of Double) in your VB code, but then when it comes time to do the insert, you'd substitute any null values with DBNull.Value.
You need the question mark ? to let Double (or any value type) can store null (or Nothing). E.g.:
Dim num as Double? = Nothing
Note the ? mark.
To store in the db:
If num Is Nothing Then
... System.DBNull.Value ...
Else
... num ...
End If
or better:
If num.HasValue Then
... System.DBNull.Value ...
Else
... num.Value ...
End If
I am posting an article HERE while I keep looking for how to acheive a solution to your situation, but the article might have another solution which is to remove the null values, and add a default value. If I find anything else I will post it.
When you set up a database (at least
in MS SQL Server) you can flag a field
as allowing NULL values and which
default values to take. If you look
through people's DB structures, you'll
see that a lot of people allow NULL
values in their database. This is a
very bad idea. I would recommend never
allowing NULL values unless the field
can logically have a NULL value (and
even this I find this only really
happens in DATE/TIME fields).
NULL values cause several problems. For starters, NULL values
are not the same as data values. A
NULL value is basically an undefined
values. On the ColdFusion end, this is
not terrible as NULL values come
across as empty strings (for the most
part). But in SQL, NULL and empty
string are very different and act very
differently. Take the following data
table for example:
id name
---------------
1 Ben
2 Jim
3 Simon
4 <NULL>
5 <NULL>
6 Ye
7
8
9 Dave
10
This table has some empty strings (id: 7, 8, 10) and some NULL values
(id: 4, 5). To see how these behave
differently, look at the following
query where we are trying to find the
number of fields that do not have
values:
Launch code in new window » Download code as text file »
* SELECT
* (
* SELECT
* COUNT( * )
* FROM
* test t
* WHERE
* LEN( t.name ) = 0
* ) AS len_count,
* (
* SELECT
* COUNT( * )
* FROM
* test t
* WHERE
* t.name IS NULL
* ) AS null_count,
* (
* SELECT
* COUNT( * )
* FROM
* test t
* WHERE
* t.name NOT LIKE '_%'
* ) AS like_count,
* (
* SELECT
* COUNT( * )
* FROM
* test t
* WHERE
* t.name IS NULL
* OR
* t.name NOT LIKE '_%'
* ) AS combo_count
This returns the following record:
LEN Count: 3
NULL Count: 2
LIKE Count: 3
Combo Count: 5
We were looking for 5 as records 4, 5, 7, 8, and 10 do not have values
in them. However, you can see that
only one attempt returned 5. This is
because while a NULL value does NOT
have a length, it is not a data type
that makes sense with length. How can
nothing have or not have a length?
It's like asking "What does that math
equation smell like?" You can't make
comparisons like that.
So, allowing NULL values makes you work extra hard to get the kind of
data you are looking for. From a
related angle, allowing NULL values
reduces your convictions about the
data in your database. You can never
quite be sure if a value exists or
not. Does that make you feel safe and
comfortable when programming?
Furthermore, while running LEN() on a NULL value doesn't act as you
might think it to, it also does NOT
throw an error. This will make
debugging your code even harder if you
do not understand the difference
between NULL values and data values.
Bottom line: DO NOT ALLOW NULL VALUES unless absolutely necessary.
You will only be making things harder
for yourself.
Related
Is there a sane way of storing int, float and boolean values in the same column in Postgres?
If have something like that:
rid
time
value
2d9c5bdc-dfc5-4ce5-888f-59d06b5065d0
2021-01-01 00:00:10.000000 +00:00
true
039264ad-af42-43a0-806b-294c878827fe
2020-01-03 10:00:00.000000 +00:00
2
b3b1f808-d3c3-4b6a-8fe6-c9f5af61d517
2021-01-01 00:00:10.000000 +00:00
43.2
Currently I'm using jsonb to store it, the problem however now is, that I can't filter in the table with for instance the greater operator.
The query
SELECT *
FROM points
WHERE value > 0;
gives back the error:
ERROR: operator does not exist: jsonb > integer: No operator matches the given name and argument types. You might need to add explicit type casts.
For me it's okay to handle boolean as 1 or 0 in case of true or false. Is there any possibility to achieve that with jsonb or is there maybe another super type which lets me use a column that is able to use all three types?
Performance is not so much of a concern here, as I'm going to have very few records inside of that table, max 5k I guess.
If you were just storing integers and floats, normally you'd use a float or numeric column.
But there's that pesky true.
You could cast the JSON...
select *
from test
where value::float > 1;
...but there's that pesky true.
You have to convert the boolean to a number to make it work.
select *
from test
where
(case when value = 'true' then 1.0 when value = 'false' then 0.0 else value::float end) >= 1;
Or ignore it.
This having to work around the type system suggests that value is actually two or even three different fields crammed into one. Consider separating them into multiple columns.
You should skip the rows where value is not number and cast the value to numeric, e.g.:
with points(id, value) as (
values
(1, 'true'::jsonb),
(2, '2'),
(3, '43.2')
)
select *
from points
where jsonb_typeof(value) = 'number'
and value::text::numeric > 0;
id | value
----+-------
2 | 2
3 | 43.2
(2 rows)
I actually found out, regardless of the jsonb fields value, that you can compare it to other jsonb in postgres. That means, I can for instance do the following:
SELECT *
FROM points
WHERE val > '5'
This correctly gives me back only the third row. It just ignores the bool value. To filter for a certain bool I can achieve that with the following query:
SELECT *
FROM points
WHERE val = 'true'
This is good enough for me. I even could hold timestamps in the json column and compare them using this methodology.
Another way of solving the problem after all your comments seem to be to make the column a numeric. This would work as well, but requires more client side conversion, as I would have to have a second type column, remembering what the actual type is. This type should than be used on the client side to convert the value back into its og value. For integers its trivial, for booleans like #schwern suggested, one can use 1 and 0, for dates, one could use the unix timestamp representation.
When I now want to search for a certain value, the type has to be contained in the where clause as well.
I have customer data with mobile phone numbers where '1' has been entered 10 times or more in a cell to bypass the customer onboarding system validation. For example '1111111111'
I used below condition in my where clause but that didn't really help.
AND p.mobile_no LIKE '%[1111111111]%'
It is possible that users might enter 1 multiple number of times in the new customer form to bypass validation. To find only 0 values in the cell I used %[^0]% in the WHERE clause and I was hoping to use something similar to find 1s where regardless of how many times it has been entered in the field, as long as it only has 1 in it it will skim out the data for me.
How can I find these instances in my data using a SQL query?
The goal is to find these anomalies and remove them.
Using: Microsoft SQL Server 2016 (SP2).
I think you are looking for the following, which tests if at least 1 '1' exists, and that no other characters exist.
select Number
from (values ('111'),('121'),('1-2'),('22')) x (Number)
-- Test that at least 1 '1' exists
where Number like '%1%'
-- And that no other allowable characters exist - expand to cover all options
and Number not like '%[0,2-9,-]%'
Using a table to define invalid phone numbers:
Declare #invalidPhoneNumbers Table (PhoneNumber char(10));
Insert Into #testData (PhoneNumber)
Values ('0000000000'), ('1111111111'), ('2222222222'), ('3333333333'), ('4444444444')
, ('5555555555'), ('6666666666'), ('7777777777'), ('8888888888'), ('9999999999');
Select ...
From ...
Where ...
And p.mobile_no Not In (Select i.PhoneNumber From #invalidPhoneNumbers i)
Or - using NOT EXISTS which may perform better:
Declare #invalidPhoneNumbers Table (PhoneNumber char(10));
Insert Into #testData (PhoneNumber)
Values ('0000000000'), ('1111111111'), ('2222222222'), ('3333333333'), ('4444444444')
, ('5555555555'), ('6666666666'), ('7777777777'), ('8888888888'), ('9999999999');
Select ...
From ...
Where ...
And Not Exists (Select * From #invalidPhoneNumbers i Where i.PhoneNumber = p.mobile_no)
When declaring the table - make sure the data type defined matches exactly the defined data type of p.mobile_no. This will make sure there are no implicit conversions that can cause issues.
Assume a sqlite database with an integer column.
Now, it tends to happen that the integer field contains NULL values (=unset) as well.
I would like to interpret NULL values as zero (0) when doing queries on that field.
Is there a way to tell sqlite that I like NULL handled like 0 in this case, especially when I do comparisons using a SELECT statement?
Since I construct the queries dynamically in C-like code, I like to be able to write something like this:
query = "SELECT * FROM tbl WHERE intField=" + IntToStr(anIntValue)
Currently, I work around this with code as follow, which I like to avoid:
if (anIntValue == 0) {
query = "SELECT * FROM tbl WHERE intField IS NULL OR intField=0"
} else {
query = "SELECT * FROM tbl WHERE intField=" + IntToStr(anIntValue)
}
Maybe there's an operator I can use in the query that converts NULLs to other values I specify?
Of course, another way would be to make sure one never ends up with NULL values in that field, but one might want to be able to tell when the value hasn't been set at all yet, so I like to be able to keep the NULL values in the database.
In Standard SQL this is called COALESCE:
COALESCE(col, 0)
Why using a proprietary extension like IFNULL/NVL if there's a Standard which is supported by every DBMS?
please, try ifnull function, see doc at http://www.sqlite.org/lang_corefunc.html#ifnull
You can use IFNULL
or
CASE WHEN fieldname IS NULL THEN 0 ELSE fieldname END
This works the same as isnull(fieldname, 0)
You can read more here
Although you could use null coalesce functionality of your RDBMS, a universal approach exists that lets you treat NULLs as zeros in a condition, like this:
String val = IntToStr(anIntValue)
"SELECT * FROM tbl WHERE intField=" + val + " OR (0="+ val+" AND intField IS NULL)"
I found a great resource here (
Comparing two bitmasks in SQL to see if any of the bits match
) for doing searches in a SQL database, where you're storing data with multiple properties using bit masks. In the example, though, all the data is stored as ints and the where clause seems to only work with ints.
Is there an easy way to convert a very similar test case to use full bitstrings instead? So instead of an example like:
with test (id, username, roles)
AS
(
SELECT 1,'Dave',1
UNION SELECT 2,'Charlie',3
UNION SELECT 3,'Susan',5
UNION SELECT 4,'Nick',2
)
select * from test where (roles & 7) != 0
instead having something like:
with test (id, username, roles)
AS
(
SELECT 1,'Dave',B'001'
UNION SELECT 2,'Charlie',B'011'
UNION SELECT 3,'Susan',B'101'
UNION SELECT 4,'Nick',B'110'
)
select * from test where (roles & B'001') != 0
I can convert back and forth, but it's easier to visualize with the actual bitstrings. For my simple conversion (above) I get an error that the operator doesn't work for bitstrings. Is there another way to set this up that would work?
One way would be to just use a bit string on the right side of the expression, too:
WITH test (id, username, roles) AS (
VALUES
(1,'Dave',B'001')
,(2,'Charlie',B'011')
,(3,'Susan',B'101')
,(4,'Nick',B'110')
)
SELECT *, (roles & B'001') AS intersection
FROM test
WHERE (roles & B'001') <> B'000';
Or you can cast an integer 0 to bit(3)
...
WHERE (roles & B'001') <> 0::bit(3);
You may be interested in this related answer that demonstrates a number of ways to convert between boolean, bit string and integer:
Can I convert a bunch of boolean columns to a single bitmap in PostgreSQL?
Be aware that storing the data as integer can save some space. integer needs 4 bytes for up to 32 bit of information, while - I quote the manual at said location:
A bit string value requires 1 byte for each group of 8 bits, plus 5 or
8 bytes overhead depending on the length of the string [...]
I'm selecting a value out of a table that can either be an integer or a nvarchar. It's stored as nvarchar. I want to conditionally call a function that will convert this value if it is an integer (that is, if it can be converted into an integer), otherwise I want to select the nvarchar with no conversion.
This is hitting a SQL Server 2005 database.
select case
when T.Value (is integer) then SomeConversionFunction(T.Value)
else T.Value
end as SomeAlias
from SomeTable T
Note that it is the "(is integer)" part that I'm having trouble with. Thanks in advance.
UPDATE
Check the comment on Ian's answer. It explains the why and the what a little better. Thanks to everyone for their thoughts.
select case
when ISNUMERIC(T.Value) then T.Value
else SomeConversionFunction(T.Value)
end as SomeAlias
Also, have you considered using the sql_variant data type?
The result set can only have one type associated with it for each column, you will get an error if the first row converts to an integer and there are strings that follow:
Msg 245, Level 16, State 1, Line 1
Conversion failed when converting the nvarchar value 'word' to data type int.
try this to see:
create table testing
(
strangevalue nvarchar(10)
)
insert into testing values (1)
insert into testing values ('word')
select * from testing
select
case
when ISNUMERIC(strangevalue)=1 THEN CONVERT(int,strangevalue)
ELSE strangevalue
END
FROM testing
best bet is to return two columns:
select
case
when ISNUMERIC(strangevalue)=1 THEN CONVERT(int,strangevalue)
ELSE NULL
END AS StrangvalueINT
,case
when ISNUMERIC(strangevalue)=1 THEN NULL
ELSE strangevalue
END AS StrangvalueString
FROM testing
or your application can test for numeric and do your special processing.
You can't have a column that is sometimes an integer and sometimes a string. Return the string and check it using int.TryParse() in the client code.
ISNUMERIC. However, this accepts +, - and decimals so more work is needed.
However, you can't have the columns as both datatypes in one go: you'll need 2 columns.
I'd suggest that you deal with this in your client or use an ISNUMERIC replacement
IsNumeric will get you part of the way there. You can then add some further code to check whether it is an integer
for example:
select top 10
case
when isnumeric(mycolumn) = 1 then
case
when convert(int, mycolumn) = mycolumn then
'integer'
else
'number but not an integer'
end
else
'not a number'
end
from mytable
To clarify some other answers, your SQL statement can't return different data types in one column (it looks like the other answers are saying you can't store different data types in one column - yours are all strign represenations).
Therefore, if you use ISNUMERIC or another function, the value will be cast as a string in the table that is returned anyway if there are other strigns being selected.
If you are selecting only one value then it could return a string or a number, however your front end code will need to be able to return the different data types.
Just to add to some of the other comments about not being able to return different data types in the same column... Database columns should know what datatype they are holding. If they don't then that should be a BIG red flag that you have a design problem somewhere, which almost guarantees future headaches (like this one).