This question already has answers here:
Why is casting from float to varchar being rounded in SQL Server?
(3 answers)
Closed 3 years ago.
The following query gives the output as 123.5
SELECT STR(123.45, 6, 1);
GO
But.. The following query gives the output as 123.3
SELECT STR(123.35, 6, 1);
GO
Why it is not giving the result as 123.4 ?
SELECT STR(123.45, 6, 1);
GO
SELECT STR(123.35, 6, 1);
GO
The following query gives the output as 123.3
SELECT STR(123.35, 6, 1);
GO
Why it is not giving the result as 123.4 ?
From the documentation for SQL Server's STR function:
[the first parameter] Is an expression of approximate numeric (float) data type with a decimal point.
From my local testing, even if I pass in a DECIMAL value, the imprecision you are seeing continues, and the input parameter still gets treated as a float.
That is:
SELECT STR(CAST(123.45 AS DECIMAL(10,2)), 6, 1)
still returns 123.5.
If you want to truncate a numerical value exactly in SQL Server, then just try casting to a DECIMAL type, e.g.
SELECT CAST(123.45 AS DECIMAL(10,1))
returns 123.4 as you would expect.
Long story short, use FORMAT instead of STR, only to format strings. There is an inconsistency here indeed.
STR is rounding half to odd which is an .... odd behavior. The more common approach is round half to even, also known as Banker's rounding. Both strategies minimize the aggregated rounding error.
That's why
SELECT STR(123.45, 6, 1),STR(123.35, 6, 1);
Returns
123.5 123.3
STR is not a rounding function though, it's a string formatting function. The strings it produces aren't meant to be added. What's more, T-SQL's ROUND function rounds away from zero and
SELECT round(123.45,1), round(123.35,1);
Produces
123.50 123.40
Oracle, PostgreSQL and MySQL's ROUND behave the same way, rounding away from zero. I suspect this is part of the standard but I haven't found a relevant link yet.
The FORMAT is far more powerful. It uses the same format strings as .NET and rounds away from zero.
SELECT FORMAT(123.45, 'n1'), FORMAT(123.35, 'n1');
Produces
123.5 123.4
Related
I have the following string and want to convert it to DECIMAL(38,0):
a321
The following code is OK:
SELECT CAST(CONVERT(binary(2), 'a321', 2) AS BIGINT); -- 41761
but this one fails:
SELECT CAST(CONVERT(binary(2), 'a321', 2) AS DECIMAL(38,0));
Msg 8114, Level 16, State 5, Line 7 Error converting data type
varbinary to numeric.
It is not a big deal to do two casts like this:
SELECT CAST(CAST(CONVERT(binary(2), 'a321', 2) AS BIGINT) AS DECIMAL(38,0));
but I want to know why is not working. Can anyone explain?
Convert to a bigint first and then convert to a decimal:
SELECT CONVERT(DECIMAL(38, 0), CONVERT(BIGINT, CONVERT(binary(2), 'a321', 2)))
The binary representation of decimals is quite different from integers, and not all binary representations can be converted to a decimal.
This may be because the number you are trying to convert is an integer, not a decimal.
SELECT CAST(41761 AS VARBINARY), CAST(41761.0 as VARBINARY)
The above give different results as I suspect that the later carries information about the decimal component. Likewise, when converting the decimal value to binary, it works as expected.
SELECT CAST(41761.0 as VARBINARY) -- gives 0x060100014A5F0600 on Azure SQL
SELECT CAST(0x060100014A5F0600 as DECIMAL) -- gives 41761 on Azure SQL
I have tagged the server in as MSDN states the results may very between versions.
Do not construct binary values, and then convert them to a data type of the numeric data type category. SQL Server does not guarantee that the result of a decimal or numeric data type conversion, to binary, will be the same between versions of SQL Server.
I have 2 columns which I need to divide sum(cola)/sum(ColB), but I am not getting the desired results since SQL server seems to truncate values after decimal
For eg. I have-
select 281370/1035
is giving 271 using simple division, whereas actual result of division is 271.8550724637681 and I want to display 271.8
I tried
SELECT cast(round(281370/1035,1) as numeric(36,1))
but that results 271.0
In SQL Server, you have to cast the integers to decimal and you could use Round to get desired precision.
SELECT cast(Round(CAST(281370 AS decimal) / CAST(1035 AS decimal),1,1) as decimal(10,1))
The problem is that you given the int number and want a decimal result
try this
select convert(decimal(30,10),281370.0/1035.0)
or
select Round(convert(decimal(30,10),281370.0/1035.0),1,1)
#Stormcloak gives the answer to specifically wanting a single position as a mantissa, however to return an exact answer you could "simply" implicitly change the datatype.
select 281370.0/1035
Returns:
271.855072
In Presto DB:
select (CAST(11 as decimal(8,6))/CAST(7 as decimal(8,6))) as result
result:1.571429
decimal(xp,xs)
xp--> total number of digits(before decimal point+ after decimal
point)
xs--> number of digits after the decimal point
reference: https://prestodb.io/docs/current/functions/decimal.html
I have a problem with round in SQL Server 2014: when I round a number to 2 decimal places sometimes the rounded number is different if I cast to float before or not.
For example, if I execute:
select round(cast(3.945 as float),2)
select round(3.945,2)
I have:
3.94
3.950
But if I execute:
select round(cast(3.935 as float),2)
select round(3.935,2)
I have:
3.94
3.940
It seems incorrect, rounding 3.935 and 3.945 casting to float before, I obtain the same value. Is this a bug?
The problem here is that float is a binary floating point type, where the representation is an approximation of the value. Floats do not losslessly convert to or from base 10, because there is no power of 10 that is also a power of 2. So when this is converted it is done in a way that leaves a roundoff error that pushes the value just before the rounding threshold.
Oddly I cannot reproduce the same behaviour on PostgreSQL and I am not entirely sure why (it may be that on PostgreSQL, round takes a numeric value and this forces a conversion back).
Never use floats where absolute accuracy is required. This occurs not only in databases, but in almost every programming language as well.
As #ChrisTravers says in his answer the issue with rounding a float is that you're not getting exact arithmetic. i.e. That explains why round(3.945,2) rounds up to 3.95 whilst round(3.945E0,2) effectively rounds down to 3.94.
If you're wondering why you see more than 2 decimal places in some cases, that's because of the type you're dealing with. i.e. 3.94 is a float, so doesn't have a specified number of decimal places; whilst 3.950 is the result of rounding a decimal(4,3); which even though we've rounded to 2 decimal places doesn't affect the precision of the type (i.e. it's still decimal(4,3); not converted to decimal(4,2) or decimal(3,2)).
If the purpose of this rounding is for display purposes, you're best of using the str function. i.e.
select str(3.945,4,2) --decimal
select str(3.945E0,4,2) --float
In the above the 4 is the length of the string (i.e. includes the decimal point as a character), and the 2 is the number of decimal places to show.
NB: In this scenario you're chaning the data type to varchar(4).
The below code allows you to see what type you get after performing an operation:
declare #result sql_variant = str(3.945E0,4,2)
select sql_variant_property(#result, 'BaseType') [BaseType]
,sql_variant_property(#result, 'MaxLength') [MaxLength]
,sql_variant_property(#result, 'Precision') [Precision]
,sql_variant_property(#result, 'Scale') [Scale]
When I execute this SQL
SELECT 1.4 UNION ALL
SELECT 2.0400 union all
SELECT 1.24
I get the following result:
1.4000
2.0400
1.2400
But when I execute the following SQL
SELECT sum(1.4) UNION ALL
SELECT sum(2.0400) union all
SELECT sum(1.24)
I get the following result:
1.4
2.0
1.2
Why is there a difference in what precision (scale) is applied for all records?
Shouldn't it always use the precision where no data is loss, just like the 1st sql?
Thx.
I know that this is quite an old question, but none of the existing answers seem to address the "why?" aspect to your question.
First, what is the data type for your literal expressions? I wasn't sure (and didn't look it up) so I ran the following:
select 1.4 union all
select 'frob'
which returns the error:
Msg 8114, Level 16, State 5, Line 1
Error converting data type varchar to numeric.
Okay, so 1.4 and the other literals are numeric - a.k.a decimal.
Next, what is the return type of the SUM function, if passed a decimal(p,s)1:
decimal(38, s)
Okay, so the data types of the 3 SUM expressions in your query are decimal(38,1), decimal(38,4) and decimal(38,2). Given those 3 data types available to pick from, decimal(38,1) is the final chosen type, based on the rules for differing precisions and scales.
The result precision and scale have an absolute maximum of 38. When a result precision is greater than 38, the corresponding scale is reduced to prevent the integral part of a result from being truncated.
So, finally, back to the documentation on decimal:
By default, SQL Server uses rounding when converting a number to a decimal or numeric value with a lower precision and scale. However, if the SET ARITHABORT option is ON, SQL Server raises an error when overflow occurs. Loss of only precision and scale is not sufficient to raise an error.
So that's your final result.
1 At first this type may seem surprising, until you realise that generally, sum will operate against multiple rows and it's easily possible for multiple values of a given precision and scale to overflow their own data type. decimal(38,s) gives the largest possible space to accommodate any overflows without losing any precision, for a particular SUM() occurrence, and means that the final data type can be decided upon before the query has executed.
Try this for same result,
SELECT Cast(Sum(1.4) As Numeric(18,4)) UNION ALL
SELECT Cast(Sum(2.0400) As Numeric(18,4)) union all
SELECT Cast(Sum(1.24) As Numeric(18,4))
Try this
SELECT sum(1.4)/1.0 UNION ALL
SELECT sum(2.0400)/1.0 union all
SELECT sum(1.24)/1.0
OR
SELECT sum(1.4)/1.0 UNION ALL
SELECT sum(2.0400) union all
SELECT sum(1.24)
Normal Query passing String Value otherwise using sum or decimal it is string convert to specified format this is fact
SELECT sum(convert(decimal,1.4,3)) UNION ALL
SELECT sum(2.0400) union all
SELECT sum(1.24)
Try this Query
SELECT convert(decimal(18,4),Sum(1.4)) UNION ALL
SELECT convert(decimal(18,4),Sum(2.0400)) UNION ALL
SELECT convert(decimal(18,4),Sum(1.24))
I have in my select clause:
AVG (cast(scale as decimal(5,2)))
I keep getting a number like: 0.6523412897, nothing I do seems to get me to my desired: 0.65.
Basically I want to have a scale of 2 (two decimal places).
Thanks
Cast the average, don't average the cast:
cast(AVG(scale) as decimal(5,2))
Update
Using ROUND() changes the value not the type. Eg. select round(0.12345,2) returns 0.12000 as it should becuase it rounds 0.12345 to 0.12 but keeps the original type with 5 decimals after the point, hence 0.12000. To change the type one must use cast, as in my post.
Try
ROUND(AVG(scale), 2)
It depends upon the database you are using, but ROUND is the answer for SQL Server:
ROUND