SQL Server float unstable last digits - sql

In SQL Server I create an aggregated column (a combination of other columns that I add, multiple, sum etc) which is of SQL datatype float.
However, when I run the same query multiple times, the last 2 digits of my float are unstable and keep changing.
Below the floats I get with the random last two digits - I try to convert to decimal and then chop off the last two digits.
select round(convert(decimal(20,19), 0.0020042890676442646), 17,1)
select round(convert(decimal(20,19), 0.0020042890676442654), 17,1)
In SSMS the result for both is: 0.0020042890676442600 as expected.
Mind you, the input constants here i took from python, so they might have been modified already. I can't take them from sql directly, as it is incredibly rare to get the calculation anomaly and i don't know how to reproduce it.
But running this via pypyodbc to python, sometimes the result is a python decimal.Decimal type with value 0.0020042890676442700 for the second statement, so it does seem to do rounding rather than truncation.
I have also noticed that the result of the calculation in sql is not always the same, and there is instability there in the last digit of the float - not sure how to test this sytematically though.
The constants casted to floats give:
select convert(float,0.0020042890676442646)
select convert(float,0.0020042890676442654)
Result: 0.00200428906764427.
Wrapped in decimals and rounded:
select round(convert(decimal(20,19), convert(float,0.0020042890676442646)), 17,1)
select round(convert(decimal(20,19), convert(float,0.0020042890676442654)), 17,1)
The result in SSMS is: 0.0020042890676442700 in both cases.
I tried sending back the floats directly instead of casting to decimal, but it seems the two unstable digits are always added at the end when they reach python. Even truncating doesn't help, other random numbers are then added.
It almost seems as if python modifies both float and Decimal during transport in a random manner, or that the instability is in sql already or both.
I tried truncating the np.float64 on the python side like this: Truncating decimal digits numpy array of floats
but as the last float digit in sql can be between e15 and e19 I can't set a consistent truncate level unless i floor everything at e15.

The order of processing of an aggregate is undefined, in the same way that the order of the results of any query are undefined unless you use an ORDER BY clause. In the case of floats, order matters. Order of aggregate processing can be forced using an OVER clause. Here's some code to demonstrate:
-- demonstrate that order matters when adding floats
declare #a float
declare #b float
declare #c float
declare #d float
declare #e float
set #a = 1
set #b = 1
set #c = 9024055778268167
-- add A to B, and then add C
-- result is 9024055778268170
set #d = #a + #b
set #e = #d + #c
select cast( #e as decimal(38,0) )
-- add C to B, and then add A
-- result is 9024055778268168
set #d = #c + #b
set #e = #d + #a
select cast( #e as decimal(38,0) )
-- put these values into a table
create table OrderMatters ( x float )
insert into OrderMatters ( x ) values ( #a )
insert into OrderMatters ( x ) values ( #b )
insert into OrderMatters ( x ) values ( #c )
declare #x float
-- add them in ascending order
-- result is 9024055778268170
select #x = sum(x) over (order by x asc ) from OrderMatters
select cast(#x as decimal(38,0))
-- add them in descending order
-- result is 9024055778268168
select #x = sum(x) over (order by x desc ) from OrderMatters
select cast(#x as decimal(38,0))

Related

SQL Server 2012 string functions

I have a field that can vary in length of the format CxxRyyy where x and y are numeric. I want to choose xx and yyy. For instance, if the field value is C1R12, then I want to get 1 and 12. if I use substring and charindex then I have to use a length, but I would like to use a position like
SUBSTRING(WPLocationNew, CHARINDEX('C',WPLocationNew,1)+1, CHARINDEX('R',WPLocationNew,1)-1)
or
SUBSTRING(WPLocationNew, CHARINDEX('C',WPLocationNew,1)+1, LEN(WPLocationNew) - CHARINDEX('R',WPLocationNew,1))
to get x, but I know that doesn't work. I feel like there is a fairly simple solution, but I am not coming up with it yet. Any suggestions
If these are cell references and will always be in the form C{1-5 digits}R{1-5 digits} you can do this:
DECLARE #t TABLE(Original varchar(32));
INSERT #t(Original) VALUES ('C14R4535'),('C1R12'),('C57R123');
;WITH src AS
(
SELECT Original, c = REPLACE(REPLACE(Original,'C',''),'R','.')
FROM #t
)
SELECT Original, C = PARSENAME(c,2), R = PARSENAME(c,1)
FROM src;
Output
Original
C
R
C14R4535
14
4535
C1R12
1
12
C57R123
57
123
Example db<>fiddle
If you need to protect against other formats, you can add
FROM #t WHERE Original LIKE 'C%[0-9]%R%[0-9]%'
AND PATINDEX('%[^C^R^0-9]%', Original) = 0
Updated db<>fiddle
It appears that you are attempting to parse an Excel cell reference. Those are predictably structured or I wouldn't suggest such an embarrassing hack as this.
Basically, take advantage of the fact that a try_cast in SQL ignores spaces when converting strings to numbers.
declare #val as varchar(20) = 'C1R12'
declare #newval as varchar(20)
declare #c as smallint
declare #r as smallint
--replace the C with 5 spaces
set #newval = replace(#val,'C',' ')
--replace the R with 5 spaces
set #newval = replace(#newval,'R',' ')
--take a look at the intermediate result, which is ' 1 14'
select #newval
set #c = try_cast(left(#newval,11) as smallint)
set #r = try_cast(right(#newval,6) as smallint)
--take a look at the results... two smallint, 1 and 14
select #c, #r
That can all be accomplished in one line for each element (a line for column and a line for row) but I wanted you to be able to understand what was happening so this example goes through the steps individually.
Here's yet another way:
declare #val as varchar(20) = 'C12R345'
declare #c as varchar(5)
declare #r as varchar(5)
set #c = SUBSTRING(#val, patindex('C%', #val)+1,(patindex('%R%', #val)-1)-patindex('C%', #val) )
set #r = SUBSTRING(#val, patindex('%R%', #val)+1, LEN(#val) -patindex('%R%', #val))
select cast(#c as int) as 'C', cast(#r as int) as 'R'
dbfiddle
There are lots of different ways to approach string parsing. Here's just one possible idea:
declare #s varchar(10) = 'C01R002';
select
rtrim( left(replace(stuff(#s, 1, 1, ''), 'R', ' '), 10)) as c,
ltrim(right(replace(substring(#s, 2, 10), 'R', ' '), 10)) as r
Strip out the 'C' and then replace the 'R' with enough spaces so that the left and right sides can be extracted using a fixed length and then easily trimmed back.
stuff() and substring() as used above are just different ways accomplish exactly the same thing. One advantage here is that it does use fairly portable string functions and it's conceivable that this is somewhat faster. This is also done inline and without multiple steps.

SQL 2017 rounds of decimal number using "FLOOR" function

I have been calculating different integer percentages with different numbers but each time I get floor rounded number. select 13*100/60 gives me 21 and the actual number is 21.66 which using a round function should give us 22 but it can only give me 21 for all different decimal numbers.
I am using SQL 2017. please help
This is due to the fact that you are dividing ints and not floating-point numbers. Integer division returns an integer.
Try the following instead (noting the .0 on the end of the 60):
SELECT 13 * 100 / 60.0
Making one of the components a floating-point number will automatically output the result as a floating-point number.
Output:
21.666666
Incidentally, if you are working with variables and one of them is a FLOAT, it will automatically produce the output you expect:
DECLARE #A FLOAT
DECLARE #B INT
DECLARE #C INT
SET #A = 13
SET #B = 100
SET #C = 60
SELECT #A * #B / #C
Output:
21.6666666666667

Wrong calculation of values after applying the round off in SQL Server

have three values a,b,c with datatype Decimal(15,7).
a is a summation of values b and c.
I didn't get what is wrong. What is possible way to get right result after rounding?
For example:
Declare #a decimal(15,7),
#b decimal(15,7),
#c decimal(15,7)
SET #a = '15212.82856500000'
SET #b = '15207.52909500000'
SET #c = '5.29947000000'
In above condition it returns the right result
select #a, #b + #c
But I used the round off 3 to all values it returns incorrect result.
select ROUND(#a, 3), ROUND(#b, 3) + ROUND(#c, 3)
Expected result is
ROUND(#a, 3) = ROUND(#b, 3) + ROUND(#c, 3)
You should round the result, not each variable individually:
SELECT ROUND(#a + #b + #c, 3)
The exact query should be
select ROUND(#a,3),ROUND(#b+#c,3)
This can gives you the desired result.
Each time you ROUND(), you deviating from the actual by a certain percentage, however tiny. Now the more you use the ROUND(), the further away you are deviating. It is always advisable to ROUND the final output.
I'm assuming the example numbers that you gave are not the actual numbers you're having issues with ?

SQL Real number convert to Ft & In

enter code hereI asked a question back in May about how to convert a number from a table that inches, such as 300.9 to a Ft' In" display. I got two very good answers...
CONVERT(VARCHAR(20),finlength /12) + '''' + CONVERT(VARCHAR(20),finlength %12)+'"' as FinishLen
replace(replace('<feet>'' <inches>', '<feet>', FinLength / 12), '<inches>', FinLength % 12) as FinishLen
Both worked well until I ran into a table that the inches are declared as "REAL" numbers. Now I ran into this error...
"The data types real and int are incompatible in the modulo operator."
How can I display that? I can't change the table declarations. Other users need that data as well.
Thanks and Kuddos for the great site.
Guess the full query might help, sorry.
SELECT TOP 1000 ProdWkYr
,Product
,Grade
,CONVERT(VARCHAR(20),finlength /12) + '''' + CONVERT(VARCHAR(20),finlength %12)+'"' as FinishLen
,BlmWeight
,BlmsNeeded
,BlmFootWgt
FROM NYS2MiscOrderInfo
where ProdWkYr = 3215
order by product, Grade
Just include a floor() in your expression like
-- -------------------------------------------------------------------
-- set-up some test data using a CTE:
WITH tst as ( SELECT 13.7 finlength UNION ALL SELECT 123 )
-- alternatively: generate a table [tst] with a single column [finlength]
-- -------------------------------------------------------------------
SELECT CONVERT(VARCHAR(20),FLOOR(finlength / 12)) + ''''
+ CONVERT(VARCHAR(20),finlength % 12)+'"' as FinishLen
FROM tst
-- results:
FinishLen
1'1.70"
10'3."
This will turn the first (ft) value into an integer while the second one (in) will still show all the digits after the decimal point.
UPDATE
When I ran the select from a #tmp table I got the same error as OP. I then modified and ended up with this:
It is as ugly as hell now, but at least it works now, see here SQL Demo:
create table #tst (finlength float);
INSERT INTO #tst VALUES (13.7),(123.),(300.9);
SELECT CONVERT(VARCHAR(20),FLOOR(finlength / 12)) + '''' -- ft
+CONVERT(VARCHAR(20),finlength-FLOOR(finlength) -- in: fractional part
+CAST(FLOOR(finlength) as int) %12)+'"' -- in: integer part
as FinishLen
FROM #tst
Please note: The formula will return reasonable results for positive values. For "negative distances" further changes are necessary. If similar output is required in different places then a UDF makes sense here. Something like:
CREATE FUNCTION ftinstr(#v float) RETURNS varchar(32) BEGIN
DECLARE #l int;
SELECT #l=FLOOR(ABS(#v));
RETURN CAST(SIGN(#v)*(#l/12) AS varchar(6))+''''
+CAST( ABS(#v)-#l+#l%12 AS varchar(20))+'"'
END
would do the trick, To be called like dbo.ftinstr( floatval ).
Maybe I can beautify it a little still ...

Remove only zero after decimal sql server 2012

Consider the following numbers.
7870.2
8220.0
I need to remove decimal points if the value ends with .0. If it ends with .2 then it should keep the value as it is.
I have used ceiling but it removes all the values after decimal.
How can I write a select query in which I can add some condition for this?
Generally speaking you should not do this in your dB. This is an app or reporting side operation. The dB is made to store and query information. It is not made to format/string manipulate information.
use right within a case statement and:
DECLARE #val decimal(5,1)
SET #val = 7870.0
Select
Case
When right(#val,1)<> '0' then
cast(#val as varchar)
else
cast(cast(#val as int) as varchar)
End
output: 7870
EDIT: I could write :
Case
When right(#val,1)<> '0' then
#val
else
cast(#val as int) -- or floor(#val)
End
but because return type of case statement is the highest precedence type from the set of given types, so the output for second version is: 7870.0 not 7870, that's why I convert it to i.e varchar in when clauses, and it can be converted outside of case statement, I mean cast ((case when...then...else... end) as datatype)
Cast the number as a float, using float(24) to increase precision:
DECLARE #t table(number decimal(10,1))
INSERT #t values(7870.2),(8220.0)
SELECT cast(number as float(24))
FROM #t
Result:
7870,2
8220
Here below goes a sample:
declare #1 decimal(4,3)
select #1 = 2.9
select case when SUBSTRING (PARSENAME(#1,1), 1, 1) = 0 then FLOOR(#1) else #1 end
Change the #1 in the select statement with your database field name.
sqlfiddle
The solution seems to be simple:
SELECT CONVERT (FLOAT, PAYLOAD)