Debugging some finance-related SQL code found a strange issue with numeric(24,8) mathematics precision.
Running the following query on your MSSQL you would get A + B * C expression result to be 0.123457
SELECT A,
B,
C,
A + B * C
FROM
(
SELECT CAST(0.12345678 AS NUMERIC(24,8)) AS A,
CAST(0 AS NUMERIC(24,8)) AS B,
CAST(500 AS NUMERIC(24,8)) AS C
) T
So we have lost 2 significant symbols. Trying to get this fixed in different ways i got that conversion of the intermediate multiplication result (which is Zero!) to numeric (24,8) would work fine.
And finally a have a solution. But still I hace a question - why MSSQL behaves in this way and which type conversions actually occured in my sample?
Just as addition of the float type is inaccurate, multiplication of the decimal types can be inaccurate (or cause inaccuracy) if you exceed the precision. See Data Type Conversion and decimal and numeric.
Since you multiplied NUMERIC(24,8) and NUMERIC(24,8), and SQL Server will only check the type not the content, it probably will try to save the potential 16 non-decimal digits (24 - 8) when it can't save all 48 digits of precision (max is 38). Combine two of them, you get 32 non-decimal digits, which leaves you with only 6 decimal digits (38 - 32).
Thus the original query
SELECT A, B, C, A + B * C
FROM ( SELECT CAST(0.12345678 AS NUMERIC(24,8)) AS A,
CAST(0 AS NUMERIC(24,8)) AS B,
CAST(500 AS NUMERIC(24,8)) AS C ) T
reduces to
SELECT A, B, C, A + D
FROM ( SELECT CAST(0.12345678 AS NUMERIC(24,8)) AS A,
CAST(0 AS NUMERIC(24,8)) AS B,
CAST(500 AS NUMERIC(24,8)) AS C,
CAST(0 AS NUMERIC(38,6)) AS D ) T
Again, between NUMERIC(24,8) and NUMERIC(38,6), SQL Server will try to save the potential 32 digits of non-decimals, so A + D reduces to
SELECT CAST(0.12345678 AS NUMERIC(38,6))
which gives you 0.123457 after rounding.
Following the logic pointed out by eed3si9n and what you said in your question it seems that the best approach when doing mathematics operations is to extract them into a function and additionally to specify precision after each operation,
It this case the function could look something like:
create function dbo.myMath(#a as numeric(24,8), #b as numeric(24,8), #c as numeric(24,8))
returns numeric(24,8)
as
begin
declare #d as numeric(24,8)
set #d = #b* #c
return #a + #d
end
Despite what it says on Precision, Scale, and Length (Transact-SQL). I believe it is also applying a minimum 'scale' (number of decimal places) of 6 to the resulting NUMERIC type for multiplication the same as it does for division etc.
Related
Case A: When you are trying to round the result yourself to the nearest decimal
SELECT ROUND (3.833333333333333) -- 4
Case B: When you let SQL do the math and then round to the nearest decimal
SELECT ROUND (23/6) -- 3 (OR CEIL)
In this case according to the order of operations:
SQL will divide what’s between the parenthesis, first = 3.833333333333333
And then (This is the problem) it will erase everything in the decimal places. (Converting it to int, automatically) =3.0
Now, let's round the decimals (Which are already erased in the previous step! And now it’s = 0)
So, the last result will be (3). Not (4).!
Even with conversions:
SELECT CAST (DIV (23,6) AS NUMERIC (10,5)) AS tst -- 3.00000
SELECT CAST ((23/6) AS DECIMAL (5,2)) AS tst -- 3.00
SELECT CAST (23/6 AS FLOAT) AS tst -- 3
SELECT CAST (23/6 AS REAL) AS tst -- 3
Is there a solution to this problem ?
Because it performs integer division. When then engine evaluates:
ROUND (23/6)
the expression 23/6 is evaluated first as 3. Then:
ROUND (3)
is evaluated as 3.
If you want the float precision you can multiply by 1.0. For example by doing:
ROUND ( 1.0 * 23/6)
In SQL Server I create an aggregated column (a combination of other columns that I add, multiple, sum etc) which is of SQL datatype float.
However, when I run the same query multiple times, the last 2 digits of my float are unstable and keep changing.
Below the floats I get with the random last two digits - I try to convert to decimal and then chop off the last two digits.
select round(convert(decimal(20,19), 0.0020042890676442646), 17,1)
select round(convert(decimal(20,19), 0.0020042890676442654), 17,1)
In SSMS the result for both is: 0.0020042890676442600 as expected.
Mind you, the input constants here i took from python, so they might have been modified already. I can't take them from sql directly, as it is incredibly rare to get the calculation anomaly and i don't know how to reproduce it.
But running this via pypyodbc to python, sometimes the result is a python decimal.Decimal type with value 0.0020042890676442700 for the second statement, so it does seem to do rounding rather than truncation.
I have also noticed that the result of the calculation in sql is not always the same, and there is instability there in the last digit of the float - not sure how to test this sytematically though.
The constants casted to floats give:
select convert(float,0.0020042890676442646)
select convert(float,0.0020042890676442654)
Result: 0.00200428906764427.
Wrapped in decimals and rounded:
select round(convert(decimal(20,19), convert(float,0.0020042890676442646)), 17,1)
select round(convert(decimal(20,19), convert(float,0.0020042890676442654)), 17,1)
The result in SSMS is: 0.0020042890676442700 in both cases.
I tried sending back the floats directly instead of casting to decimal, but it seems the two unstable digits are always added at the end when they reach python. Even truncating doesn't help, other random numbers are then added.
It almost seems as if python modifies both float and Decimal during transport in a random manner, or that the instability is in sql already or both.
I tried truncating the np.float64 on the python side like this: Truncating decimal digits numpy array of floats
but as the last float digit in sql can be between e15 and e19 I can't set a consistent truncate level unless i floor everything at e15.
The order of processing of an aggregate is undefined, in the same way that the order of the results of any query are undefined unless you use an ORDER BY clause. In the case of floats, order matters. Order of aggregate processing can be forced using an OVER clause. Here's some code to demonstrate:
-- demonstrate that order matters when adding floats
declare #a float
declare #b float
declare #c float
declare #d float
declare #e float
set #a = 1
set #b = 1
set #c = 9024055778268167
-- add A to B, and then add C
-- result is 9024055778268170
set #d = #a + #b
set #e = #d + #c
select cast( #e as decimal(38,0) )
-- add C to B, and then add A
-- result is 9024055778268168
set #d = #c + #b
set #e = #d + #a
select cast( #e as decimal(38,0) )
-- put these values into a table
create table OrderMatters ( x float )
insert into OrderMatters ( x ) values ( #a )
insert into OrderMatters ( x ) values ( #b )
insert into OrderMatters ( x ) values ( #c )
declare #x float
-- add them in ascending order
-- result is 9024055778268170
select #x = sum(x) over (order by x asc ) from OrderMatters
select cast(#x as decimal(38,0))
-- add them in descending order
-- result is 9024055778268168
select #x = sum(x) over (order by x desc ) from OrderMatters
select cast(#x as decimal(38,0))
This post has the following code:
DECLARE #A DECIMAL(3, 0), #B DECIMAL(18, 0), #F FLOAT
SET #A = 3
SET #B = 3
SET #F = 3
SELECT 1 / #A * 3.0, 1 / #B * 3.0, 1 / #F * 3.0
SELECT 1 / #A * 3 , 1 / #B * 3 , 1 / #F * 3
Using float, the expression evaluates to 1. Using Decimal, the expression evaluates to some collection of 9s after the decimal point. Why does float yield the more accurate answer in this case? I thought that Decimal is more accurate / exact per Difference between numeric, float and decimal in SQL Server and Use Float or Decimal for Accounting Application Dollar Amount?
The decimal values that you have declared are fixed width, and there are no points after the decimal place. This affects the calculations.
SQL Server has a rather complex formula for how to calculate the precision of arithmetical expressions containing decimal numbers. The details are in the documentation. You also need to take into account that numeric constants are in decimal format, rather than numeric.
Also, in the end, you need to convert back to a decimal format with the precision that you want. In that case, you might discover that float and decimal are equivalent.
The database I am using is SQL Server 2005. I am trying to round values DOWN to the nearest .05 (nickel).
So far I have:
SELECT ROUND(numberToBeRounded / 5, 2) * 5
which almost works - what I need is for the expression, when numberToBeRounded is 1.99, to evaluate to 1.95, not 2.
Specify a non-zero value for a third parameter to truncate instead of round:
SELECT ROUND(numberToBeRounded / 5, 2, 1) * 5
Note: Truncating rounds toward zero, rather than down, but that only makes a difference if you have negative values. To round down even for negative values you can use the floor function, but then you can't specify number of decimals so you need to multiply instead of dividing:
SELECT FLOOR(numberToBeRounded * 20) / 20
If your data type is numeric (ISO decimal) or `money, you can round towards zero quite easily, to any particular "unit", thus:
declare #value money = 123.3499
declare #unit money = 0.05
select value = value ,
rounded_towards_zero = value - ( value % #unit )
from #foo
And it works regardless of the sign of the value itself, though the unit to which you're rounding should be positive.
123.3499 -> 123.3000
-123.3499 -> -123.3000
I have a stored procedure which calculates the distance between two coordinate pairs as a float. I'm trying to use this to filter a list of values but getting an arithmetic overflow error. The query is:
SELECT * FROM Housing h WHERE convert(float, dbo.CalculateDistance(35, -94, h.Latitude, h.Longitude)) <= 30.0
Which errors with:
Msg 8115, Level 16, State 6, Line 1 Arithmetic overflow error
converting float to data type numeric.
The stored procedure for reference:
CREATE FUNCTION [dbo].[CalculateDistance]
(#Longitude1 DECIMAL(8,5),
#Latitude1 DECIMAL(8,5),
#Longitude2 DECIMAL(8,5),
#Latitude2 DECIMAL(8,5))
RETURNS FLOAT
AS
BEGIN
DECLARE #Temp FLOAT
SET #Temp = SIN(#Latitude1/57.2957795130823) * SIN(#Latitude2/57.2957795130823) + COS(#Latitude1/57.2957795130823) * COS(#Latitude2/57.2957795130823) * COS(#Longitude2/57.2957795130823 - #Longitude1/57.2957795130823)
IF #Temp > 1
SET #Temp = 1
ELSE IF #Temp < -1
SET #Temp = -1
RETURN (3958.75586574 * ACOS(#Temp) )
END
've also tried converting the result to decimal with no effect.
Your inputs are DECIMAL(8,5). This means that the equations consist of, for example, SIN(DECIMAL(8,5) / 57.2957795130823). Where 57.2957795130823 can not be represented as a DECIMAL(8,5).
This means that you have an implicat CAST operation due to the different data type. In this case, it would seem that the 57.2957795130823 is being cast to DECIMAL(8,5) [a numeric], and causing the overflow.
I would recommend any of these:
- Altering your function to take the inputs as FLOATS. Even if the function is called with numerics
- Changing 57.2957795130823 to 57.29577
- Explicitly casting the DECIMALs to FLOATs
I would try converting some of my arithmetic just in case
convert(float,(SIN(#Latitude1/57.2957795130823)) * convert(float,(SIN(#Latitude2/57.2957795130823)) + convert(float,(COS(#Latitude1/57.2957795130823)) * convert(float,(COS(#Latitude2/57.2957795130823)) * convert(float,COS(#Longitude2/57.2957795130823 - #Longitude1/57.2957795130823))
another thing you could use is the
IFNULL(convert(float,(SIN(#Latitude1/57.2957795130823)),0.00)
your results may be returning nulls
It's your comparison to <= 30.0
30.0 is decimal(3,2) (Constants with decimal points are decimal in SQL Server) and the float output won't cast. See:
SELECT 30.0 AS What INTO dbo.DataType
Go
SELECT t.name, c.*
FROM sys.columns c JOIN sys.types t ON c.system_type_id = t.system_type_id
WHERE object_id = OBJECT_ID('dbo.DataType')
GO
DROP TABLE dbo.DataType
GO
Try
... <= CAST(30.0 As float)
You're returning a float. Shouldn't you be using floats for the latitude and longitude variables as well?