This post has the following code:
DECLARE #A DECIMAL(3, 0), #B DECIMAL(18, 0), #F FLOAT
SET #A = 3
SET #B = 3
SET #F = 3
SELECT 1 / #A * 3.0, 1 / #B * 3.0, 1 / #F * 3.0
SELECT 1 / #A * 3 , 1 / #B * 3 , 1 / #F * 3
Using float, the expression evaluates to 1. Using Decimal, the expression evaluates to some collection of 9s after the decimal point. Why does float yield the more accurate answer in this case? I thought that Decimal is more accurate / exact per Difference between numeric, float and decimal in SQL Server and Use Float or Decimal for Accounting Application Dollar Amount?
The decimal values that you have declared are fixed width, and there are no points after the decimal place. This affects the calculations.
SQL Server has a rather complex formula for how to calculate the precision of arithmetical expressions containing decimal numbers. The details are in the documentation. You also need to take into account that numeric constants are in decimal format, rather than numeric.
Also, in the end, you need to convert back to a decimal format with the precision that you want. In that case, you might discover that float and decimal are equivalent.
Debugging some finance-related SQL code found a strange issue with numeric(24,8) mathematics precision.
Running the following query on your MSSQL you would get A + B * C expression result to be 0.123457
SELECT A,
B,
C,
A + B * C
FROM
(
SELECT CAST(0.12345678 AS NUMERIC(24,8)) AS A,
CAST(0 AS NUMERIC(24,8)) AS B,
CAST(500 AS NUMERIC(24,8)) AS C
) T
So we have lost 2 significant symbols. Trying to get this fixed in different ways i got that conversion of the intermediate multiplication result (which is Zero!) to numeric (24,8) would work fine.
And finally a have a solution. But still I hace a question - why MSSQL behaves in this way and which type conversions actually occured in my sample?
Just as addition of the float type is inaccurate, multiplication of the decimal types can be inaccurate (or cause inaccuracy) if you exceed the precision. See Data Type Conversion and decimal and numeric.
Since you multiplied NUMERIC(24,8) and NUMERIC(24,8), and SQL Server will only check the type not the content, it probably will try to save the potential 16 non-decimal digits (24 - 8) when it can't save all 48 digits of precision (max is 38). Combine two of them, you get 32 non-decimal digits, which leaves you with only 6 decimal digits (38 - 32).
Thus the original query
SELECT A, B, C, A + B * C
FROM ( SELECT CAST(0.12345678 AS NUMERIC(24,8)) AS A,
CAST(0 AS NUMERIC(24,8)) AS B,
CAST(500 AS NUMERIC(24,8)) AS C ) T
reduces to
SELECT A, B, C, A + D
FROM ( SELECT CAST(0.12345678 AS NUMERIC(24,8)) AS A,
CAST(0 AS NUMERIC(24,8)) AS B,
CAST(500 AS NUMERIC(24,8)) AS C,
CAST(0 AS NUMERIC(38,6)) AS D ) T
Again, between NUMERIC(24,8) and NUMERIC(38,6), SQL Server will try to save the potential 32 digits of non-decimals, so A + D reduces to
SELECT CAST(0.12345678 AS NUMERIC(38,6))
which gives you 0.123457 after rounding.
Following the logic pointed out by eed3si9n and what you said in your question it seems that the best approach when doing mathematics operations is to extract them into a function and additionally to specify precision after each operation,
It this case the function could look something like:
create function dbo.myMath(#a as numeric(24,8), #b as numeric(24,8), #c as numeric(24,8))
returns numeric(24,8)
as
begin
declare #d as numeric(24,8)
set #d = #b* #c
return #a + #d
end
Despite what it says on Precision, Scale, and Length (Transact-SQL). I believe it is also applying a minimum 'scale' (number of decimal places) of 6 to the resulting NUMERIC type for multiplication the same as it does for division etc.
I am trying to do the following calculation:
DECLARE #X Decimal(18,6);
DECLARE #Y Decimal(20,16);
SET #X = 0.002700;
SET #Y = 0.0027770000000000;
SELECT 200 * ( POWER(#Y, ( -1 / ( 2 * #X ))) - 1 );
However this results in:
Msg 8115, Level 16, State 2, Line 7
Arithmetic overflow error converting expression to data type float.
I have tried casting to float but to no avail.
Any ideas what I am missing here?
A float data type can store a maximum value of about 1.79E+308, but the result of this expression is approximately 5.16E+475. See http://tinyurl.com/oy7deu8
May be there is a way to do this. You are trying to calculate #y power (-1/2*#x). This is equivalent to calculating e power (log(y)*(-1/2x)).
You could calculate it as
select 200 * POWER(2.718281828, log(#Y) * (-1/(2*#X)))
Note that LOG by default calculates the natural logarithm of the number. And e = 2.718281828
I try to make IP to country query.
Simplest way is make query like this:
select Country
from dbip
where 8 * power(255,3) +
8 * power(255,2) +
8 * power(255,1) + 8
between FromIPInt and ToIPInt
It works for small Google's IP, but for greater IP:
Arithmetic overflow error converting expression to data type int.
Which other way I can compute it?
CAST is the solution, but you have do put it in the right place :
select Country
from dbip
where cast(255 as bigint) * power(255,3)
+ 255 * power(255,2)
+ 255 * power(255,1)
+ 255
between FromIPInt and ToIPInt
The error Arithmetic overflow error converting expression to data type int occurs if you CAST the whole calculation:
select cast(255 * power(255,3) as bigint)
But not if you cast your first number
select cast(255 as bigint) * power(255,3)
More details about that: https://msdn.microsoft.com/en-us/library/ms187745.aspx
Integer constants greater than 2,147,483,647 are converted to the decimal data type, not the bigint data type.
SQL Server does not automatically promote other integer data types (tinyint, smallint, and int) to bigint.
I'm trying to determine the best way to truncate or drop extra decimal places in SQL without rounding. For example:
declare #value decimal(18,2)
set #value = 123.456
This will automatically round #value to be 123.46, which is good in most cases. However, for this project, I don't need that. Is there a simple way to truncate the decimals I don't need? I know I can use the left() function and convert back to a decimal. Are there any other ways?
ROUND ( 123.456 , 2 , 1 )
When the third parameter != 0 it truncates rather than rounds.
Syntax
ROUND ( numeric_expression , length [ ,function ] )
Arguments
numeric_expression
Is an expression of the exact numeric or approximate numeric data
type category, except for the bit data type.
length
Is the precision to which numeric_expression is to be rounded. length must be an expression of type tinyint, smallint, or int. When length is a positive number, numeric_expression is rounded to the number of decimal positions specified by length. When length is a negative number, numeric_expression is rounded on the left side of the decimal point, as specified by length.
function
Is the type of operation to perform. function must be tinyint, smallint, or int. When function is omitted or has a value of 0 (default), numeric_expression is rounded. When a value other than 0 is specified, numeric_expression is truncated.
select round(123.456, 2, 1)
SELECT Cast(Round(123.456,2,1) as decimal(18,2))
Here's the way I was able to truncate and not round:
select 100.0019-(100.0019%.001)
returns 100.0010
And your example:
select 123.456-(123.456%.001)
returns 123.450
Now if you want to get rid of the ending zero, simply cast it:
select cast((123.456-(123.456%.001)) as decimal (18,2))
returns 123.45
Actually whatever the third parameter is, 0 or 1 or 2, it will not round your value.
CAST(ROUND(10.0055,2,0) AS NUMERIC(10,2))
Do you want the decimal or not?
If not, use
select ceiling(#value),floor(#value)
If you do it with 0 then do a round:
select round(#value,2)
Another truncate with no rounding solution and example.
Convert 71.950005666 to a single decimal place number (71.9)
1) 71.950005666 * 10.0 = 719.50005666
2) Floor(719.50005666) = 719.0
3) 719.0 / 10.0 = 71.9
select Floor(71.950005666 * 10.0) / 10.0
Round has an optional parameter
Select round(123.456, 2, 1) will = 123.45
Select round(123.456, 2, 0) will = 123.46
ROUND(number, decimals, operation)
number => Required. The number to be rounded
decimals => Required. The number of decimal places to round number to
operation => Optional. If 0, it rounds the result to the number of decimal. If another value than 0, it truncates the result to the number of decimals. Default value is 0
SELECT ROUND(235.415, 2, 1)
will give you 235.410
SELECT ROUND(235.415, 0, 1)
will give you 235.000
But now trimming0 you can use cast
SELECT CAST(ROUND(235.415, 0, 1) AS INT)
will give you 235
This will remove the decimal part of any number
SELECT ROUND(#val,0,1)
SELECT CAST(Value as Decimal(10,2)) FROM TABLE_NAME;
Would give you 2 values after the decimal point. (MS SQL SERVER)
Another way is ODBC TRUNCATE function:
DECLARE #value DECIMAL(18,3) =123.456;
SELECT #value AS val, {fn TRUNCATE(#value, 2)} AS result
LiveDemo
Output:
╔═════════╦═════════╗
║ val ║ result ║
╠═════════╬═════════╣
║ 123,456 ║ 123,450 ║
╚═════════╩═════════╝
Remark:
I recommend using built-in ROUND function with 3rd parameter set to 1.
I know this is pretty late but I don't see it as an answer and have been using this trick for years.
Simply subtract .005 from your value and use Round(#num,2).
Your example:
declare #num decimal(9,5) = 123.456
select round(#num-.005,2)
returns 123.45
It will automatically adjust the rounding to the correct value you are looking for.
By the way, are you recreating the program from the movie Office Space?
Try like this:
SELECT cast(round(123.456,2,1) as decimal(18,2))
If you desire to take some number like 89.0904987 and turn it into 89.09 by simply omitting the undesired decimal places, simply use the following:
select cast(yourColumnName as decimal(18,2))
The following screenshot is from W3Schools SQL Data Types section, which describes what decimal(18,2) is doing:
Therefore,
select cast(89.0904987 as decimal(18,2))
gives you: 89.09
Please try to use this code for converting 3 decimal values after a point into 2 decimal places:
declare #val decimal (8, 2)
select #val = 123.456
select #val = #val
select #val
The output is 123.46
I think you want only the decimal value,
in this case you can use the following:
declare #val decimal (8, 3)
SET #val = 123.456
SELECT #val - ROUND(#val,0,1)
I know this question is really old but nobody used sub-strings to round. This as advantage the ability to round really long numbers (limit of your string in SQL server which is usually 8000 characters):
SUBSTRING('123.456', 1, CHARINDEX('.', '123.456') + 2)
I think we can go much easier with simpler example solution found in Hackerrank:
Problem statement: Query the greatest value of the Northern Latitudes
(LAT_N) from STATION that is less than 137.2345. Truncate your answer
to 4 decimal places.
SELECT TRUNCATE(MAX(LAT_N),4)
FROM STATION
WHERE LAT_N < 137.23453;
Solution Above gives you idea how to simply make value limited to 4 decimal points. If you want to lower or upper the numbers after decimal, just change 4 to whatever you want.
Mod(x,1) is the easiest way I think.
select convert(int,#value)