This post has the following code:
DECLARE #A DECIMAL(3, 0), #B DECIMAL(18, 0), #F FLOAT
SET #A = 3
SET #B = 3
SET #F = 3
SELECT 1 / #A * 3.0, 1 / #B * 3.0, 1 / #F * 3.0
SELECT 1 / #A * 3 , 1 / #B * 3 , 1 / #F * 3
Using float, the expression evaluates to 1. Using Decimal, the expression evaluates to some collection of 9s after the decimal point. Why does float yield the more accurate answer in this case? I thought that Decimal is more accurate / exact per Difference between numeric, float and decimal in SQL Server and Use Float or Decimal for Accounting Application Dollar Amount?
The decimal values that you have declared are fixed width, and there are no points after the decimal place. This affects the calculations.
SQL Server has a rather complex formula for how to calculate the precision of arithmetical expressions containing decimal numbers. The details are in the documentation. You also need to take into account that numeric constants are in decimal format, rather than numeric.
Also, in the end, you need to convert back to a decimal format with the precision that you want. In that case, you might discover that float and decimal are equivalent.
Related
I have been calculating different integer percentages with different numbers but each time I get floor rounded number. select 13*100/60 gives me 21 and the actual number is 21.66 which using a round function should give us 22 but it can only give me 21 for all different decimal numbers.
I am using SQL 2017. please help
This is due to the fact that you are dividing ints and not floating-point numbers. Integer division returns an integer.
Try the following instead (noting the .0 on the end of the 60):
SELECT 13 * 100 / 60.0
Making one of the components a floating-point number will automatically output the result as a floating-point number.
Output:
21.666666
Incidentally, if you are working with variables and one of them is a FLOAT, it will automatically produce the output you expect:
DECLARE #A FLOAT
DECLARE #B INT
DECLARE #C INT
SET #A = 13
SET #B = 100
SET #C = 60
SELECT #A * #B / #C
Output:
21.6666666666667
I have a column of decimal type and i need to use the sum function on it like this:
declare #credit decimal = (select
( select ISNULL(SUM(Convert(DECIMAL(13,2), Amount)),0)
from TransactionDetail as t1
where t1.AccountFrom = #fromAccount and t1.AccountTo = #toAccount
) -
( select ISNULL(SUM(Convert(DECIMAL(13,2),Amount)),0)
from TransactionDetail as t1
where t1.AccountFrom = #toAccount and t1.AccountTo = #fromAccount
)
)
select #credit
The output should be a decimal number like :
13.56
However, the result is always int, Any suggestions?
The default scale is 0. If you want the result as a particular format, try explicitly adding precision and scale to the variable:
declare #credit decimal(13, 2) = (select . . .
This behavior is well documented:
The number of decimal digits that will be stored to the right of the
decimal point. This number is substracted from p to determine the
maximum number of digits to the left of the decimal point. The maximum
number of decimal digits that can be stored to the right of the
decimal point. Scale must be a value from 0 through p. cSale can be
specified only if precision is specified. The default scale is 0;
The database I am using is SQL Server 2005. I am trying to round values DOWN to the nearest .05 (nickel).
So far I have:
SELECT ROUND(numberToBeRounded / 5, 2) * 5
which almost works - what I need is for the expression, when numberToBeRounded is 1.99, to evaluate to 1.95, not 2.
Specify a non-zero value for a third parameter to truncate instead of round:
SELECT ROUND(numberToBeRounded / 5, 2, 1) * 5
Note: Truncating rounds toward zero, rather than down, but that only makes a difference if you have negative values. To round down even for negative values you can use the floor function, but then you can't specify number of decimals so you need to multiply instead of dividing:
SELECT FLOOR(numberToBeRounded * 20) / 20
If your data type is numeric (ISO decimal) or `money, you can round towards zero quite easily, to any particular "unit", thus:
declare #value money = 123.3499
declare #unit money = 0.05
select value = value ,
rounded_towards_zero = value - ( value % #unit )
from #foo
And it works regardless of the sign of the value itself, though the unit to which you're rounding should be positive.
123.3499 -> 123.3000
-123.3499 -> -123.3000
In SQL Server I declare a decimal variable, then set it from an equation. The decimal variable loses precision. However, if I just select the equation, the precision is intact. How do I set the decimal variable without losing precision?
SQL:
DECLARE #_oDiscount decimal(10,2)
SET #_oDiscount = CAST(9.99 AS decimal(10,2)) * CAST(.5 AS float)
SELECT #_oDiscount AS DecimalVariable, CAST(9.99 AS decimal(10,2)) * CAST(.5 AS float) AS Equation
OUTPUT:
DecimalVariable | Equation
-------------------------------
5.00 | 4.995
Well, YOU'VE defined the decimal to contain only 2 digits after the comma!
Therefore, the result of the calculation being 4.995, it will be rounded to 5.00.
If you would be using DECIMAL(12,4) instead, then there is no loss of precision!
DECLARE #_oDiscount decimal(10,4)
SET #_oDiscount = CAST(9.99 AS decimal(10,4)) * CAST(0.5 AS DECIMAL(10,4))
SELECT #_oDiscount
--> return 4.9950
Also: I would recommend against using FLOAT whenever possible!
And furthermore: there's really no need for all those casts..... just use
SET #_oDiscount = 9.99 * 0.5
and you'll get just the same results.
The DECIMAL(p, s) defines how precise the decimal value will be: p (precision) stands for the total number of digits, while s (scale) stands for the number of digits after the decimal point.
I have the following set of calculations in excel that I want to be able to use in a stored procedure.
Excel
CellA: 45/448.2 = 0.100401606425703
CellB: 1-CellA = 0.899598393574297
CellC: 1-CellB = 0.100401606425703
CellD: CellC * 448.2 = 45.000000000000000
In SQL I am doing the following:
declare #a decimal(18,15) = 45/448.2
declare #b decimal(18,15) = 1-#a
declare #c decimal(18,15) = 1-#b
declare #d decimal(18,15) = #c * 448.2
I have also tried running the calculation in one line
declare #e decimal(18,15) = (1-(1-(45/448.2)))*448.2
when I return the values SQL gives me the following:
#a: 0.100401000000000
#b: 0.899599000000000
#c: 0.100401000000000
#d: 44.999728200000000
#e: 44.999728200000000
I've tried adjusting the precision of the decimals in SQL but I nothing makes a difference, it only returns the first 6 digits of the decimal.
Does Excel do any optimization when running the formula?
Any ideas?
Even just your first line is enough to show the problem:
declare #a decimal(18,15) = 45/448.2
print #a
gives
---------------------------------------
0.100401000000000
This is because of data types. When you say
448.2
it is (per the documentation) interpreted as a constant of type decimal, and also per the documentation,
In Transact-SQL statements, a constant with a decimal point is
automatically converted into a numeric data value, using the minimum
precision and scale necessary. For example, the constant 12.345 is
converted into a numeric value with a precision of 5 and a scale of 3.
So 448.2 is decimal(4,3). 45 is integer, which when combined with a decimal is treated as having precision of 10 and scale 0 . When we divide, the rules say
Operation Result precision Result scale
e1 / e2 p1 - s1 + s2 + max(6, s1 + p2 + 1) max(6, s1 + p2 + 1)
which in this case gives a result precision of 10 - 3 + 0 + max(6, 0 + 3 + 1) and scale of max(6, 0 + 3 + 1), which comes out to 13 and 6.
That result scale of 6 is why the result only has those six decimal places.
The way to fix it is to get your operands into an appropriate type before acting on them; for example, here are two ways:
Force a number to be treated as floating-point:
declare #a decimal(18,15) = 45/448.2e0
select #a
---------------------------------------
0.100401606425703
Explicitly supply a decimal scale:
declare #a decimal(18,15) = 45/cast(448.2 as decimal(18,10))
select #a
---------------------------------------
0.100401606425703