When I execute this SQL
SELECT 1.4 UNION ALL
SELECT 2.0400 union all
SELECT 1.24
I get the following result:
1.4000
2.0400
1.2400
But when I execute the following SQL
SELECT sum(1.4) UNION ALL
SELECT sum(2.0400) union all
SELECT sum(1.24)
I get the following result:
1.4
2.0
1.2
Why is there a difference in what precision (scale) is applied for all records?
Shouldn't it always use the precision where no data is loss, just like the 1st sql?
Thx.
I know that this is quite an old question, but none of the existing answers seem to address the "why?" aspect to your question.
First, what is the data type for your literal expressions? I wasn't sure (and didn't look it up) so I ran the following:
select 1.4 union all
select 'frob'
which returns the error:
Msg 8114, Level 16, State 5, Line 1
Error converting data type varchar to numeric.
Okay, so 1.4 and the other literals are numeric - a.k.a decimal.
Next, what is the return type of the SUM function, if passed a decimal(p,s)1:
decimal(38, s)
Okay, so the data types of the 3 SUM expressions in your query are decimal(38,1), decimal(38,4) and decimal(38,2). Given those 3 data types available to pick from, decimal(38,1) is the final chosen type, based on the rules for differing precisions and scales.
The result precision and scale have an absolute maximum of 38. When a result precision is greater than 38, the corresponding scale is reduced to prevent the integral part of a result from being truncated.
So, finally, back to the documentation on decimal:
By default, SQL Server uses rounding when converting a number to a decimal or numeric value with a lower precision and scale. However, if the SET ARITHABORT option is ON, SQL Server raises an error when overflow occurs. Loss of only precision and scale is not sufficient to raise an error.
So that's your final result.
1 At first this type may seem surprising, until you realise that generally, sum will operate against multiple rows and it's easily possible for multiple values of a given precision and scale to overflow their own data type. decimal(38,s) gives the largest possible space to accommodate any overflows without losing any precision, for a particular SUM() occurrence, and means that the final data type can be decided upon before the query has executed.
Try this for same result,
SELECT Cast(Sum(1.4) As Numeric(18,4)) UNION ALL
SELECT Cast(Sum(2.0400) As Numeric(18,4)) union all
SELECT Cast(Sum(1.24) As Numeric(18,4))
Try this
SELECT sum(1.4)/1.0 UNION ALL
SELECT sum(2.0400)/1.0 union all
SELECT sum(1.24)/1.0
OR
SELECT sum(1.4)/1.0 UNION ALL
SELECT sum(2.0400) union all
SELECT sum(1.24)
Normal Query passing String Value otherwise using sum or decimal it is string convert to specified format this is fact
SELECT sum(convert(decimal,1.4,3)) UNION ALL
SELECT sum(2.0400) union all
SELECT sum(1.24)
Try this Query
SELECT convert(decimal(18,4),Sum(1.4)) UNION ALL
SELECT convert(decimal(18,4),Sum(2.0400)) UNION ALL
SELECT convert(decimal(18,4),Sum(1.24))
Related
This question already has answers here:
Why is casting from float to varchar being rounded in SQL Server?
(3 answers)
Closed 3 years ago.
The following query gives the output as 123.5
SELECT STR(123.45, 6, 1);
GO
But.. The following query gives the output as 123.3
SELECT STR(123.35, 6, 1);
GO
Why it is not giving the result as 123.4 ?
SELECT STR(123.45, 6, 1);
GO
SELECT STR(123.35, 6, 1);
GO
The following query gives the output as 123.3
SELECT STR(123.35, 6, 1);
GO
Why it is not giving the result as 123.4 ?
From the documentation for SQL Server's STR function:
[the first parameter] Is an expression of approximate numeric (float) data type with a decimal point.
From my local testing, even if I pass in a DECIMAL value, the imprecision you are seeing continues, and the input parameter still gets treated as a float.
That is:
SELECT STR(CAST(123.45 AS DECIMAL(10,2)), 6, 1)
still returns 123.5.
If you want to truncate a numerical value exactly in SQL Server, then just try casting to a DECIMAL type, e.g.
SELECT CAST(123.45 AS DECIMAL(10,1))
returns 123.4 as you would expect.
Long story short, use FORMAT instead of STR, only to format strings. There is an inconsistency here indeed.
STR is rounding half to odd which is an .... odd behavior. The more common approach is round half to even, also known as Banker's rounding. Both strategies minimize the aggregated rounding error.
That's why
SELECT STR(123.45, 6, 1),STR(123.35, 6, 1);
Returns
123.5 123.3
STR is not a rounding function though, it's a string formatting function. The strings it produces aren't meant to be added. What's more, T-SQL's ROUND function rounds away from zero and
SELECT round(123.45,1), round(123.35,1);
Produces
123.50 123.40
Oracle, PostgreSQL and MySQL's ROUND behave the same way, rounding away from zero. I suspect this is part of the standard but I haven't found a relevant link yet.
The FORMAT is far more powerful. It uses the same format strings as .NET and rounds away from zero.
SELECT FORMAT(123.45, 'n1'), FORMAT(123.35, 'n1');
Produces
123.5 123.4
This query:
select nvl(0.75,0) from dual
gives me 0.75 (numeric) but this query:
select decode(1,0,null,0.75) from dual
gives me '.75' (string).
Why?
I tried to fix this by changing the second query to:
select decode(1,0,null,to_char(0.75,'0.99')) from dual
but in my actual code the 0.75 will be a field (NUMBER) that may have a different number of decimal places and I'm not suppose to add/remove anything from that value.
Any ideas on how to fix the missing zero issue but still support all possible decimal lengths?
It's because the 3rd parameter of your decode statement is NULL; as per the documentation1 (my emphasis).
Oracle automatically converts expr and each search value to the data type of the first search value before comparing.... If the first result has the data type CHAR or if the first result is null, then Oracle converts the return value to the data type VARCHAR2.
In your case the first result is NULL, which Oracle treats as a VARCHAR2. Your return value is being implicitly converted to a VARCHAR2. If you changed your DECODE() to the following you'd get a number:
select decode(1, 0, 0, 0.75)
and you could achieve your NULL by using the NULLIF() function:
select nullif(decode(1, 0, 0, 0.75), 0) ...
It's better to use a CASE statement, which enforces that all returned datatypes are the same:
select case 1 when 0 then null
else 0.75
end ...
1. which I've been caught out on as well.
You can use
select decode(1,0,to_number(null),0.75) from dual
In the first case, nvl() is returning a numeric value. How to display that is up to the program you are using to run your queries. TOAD displays it like you said, 0.75.
In the second example, decode() is returning a varchar2. When Oracle converts a number to a string without any formatting, this is what you get, i.e. ".75".
From the Oracle docs on decode():
If the first result has the datatype CHAR or if the first result is
null, then Oracle converts the return value to the datatype VARCHAR2.
You could use a number format and rtrim() to achieve your purpose, e.g.:
select rtrim(to_char(.75, '9990.99999999999999'),'0') from dual;
Result:
0.75
select to_number(decode(1,0,null,0.75)) from dual
I have 2 columns which I need to divide sum(cola)/sum(ColB), but I am not getting the desired results since SQL server seems to truncate values after decimal
For eg. I have-
select 281370/1035
is giving 271 using simple division, whereas actual result of division is 271.8550724637681 and I want to display 271.8
I tried
SELECT cast(round(281370/1035,1) as numeric(36,1))
but that results 271.0
In SQL Server, you have to cast the integers to decimal and you could use Round to get desired precision.
SELECT cast(Round(CAST(281370 AS decimal) / CAST(1035 AS decimal),1,1) as decimal(10,1))
The problem is that you given the int number and want a decimal result
try this
select convert(decimal(30,10),281370.0/1035.0)
or
select Round(convert(decimal(30,10),281370.0/1035.0),1,1)
#Stormcloak gives the answer to specifically wanting a single position as a mantissa, however to return an exact answer you could "simply" implicitly change the datatype.
select 281370.0/1035
Returns:
271.855072
In Presto DB:
select (CAST(11 as decimal(8,6))/CAST(7 as decimal(8,6))) as result
result:1.571429
decimal(xp,xs)
xp--> total number of digits(before decimal point+ after decimal
point)
xs--> number of digits after the decimal point
reference: https://prestodb.io/docs/current/functions/decimal.html
im sure i am not the first one to ask this but i can't find the answer to this:
I haver a select query on a datatable in a sqlite database.
select *, ((int_EndTime)-(int_StartTime))/60 as dou_usage_min FROM tbl_unautho_usage;
when i run this i get all the fields from the datatable including a new column calculated from to integer columns with unix time stamp values. However, i want my calculated column to be of the type double. With the query above i get a type integer.
select *, ((int_EndTime as float)-(int_StartTime as float))/60 as dou_usage_min FROM tbl_unautho_usage;
Afterwards I tried to change the column type of my integer-columns to float, but this gies me the following error:
near "as": syntax error:
i got the idea for that from the following post:
How to cast computed column with correct decimal/$ result
Try multiplying a value used within the arithmetic operation by 1.0.
select
*,
((int_EndTime*1.0)-(int_StartTime*1.0))/60 as dou_usage_min
FROM tbl_unautho_usage;
Probably only one value multiplied will be sufficient.
The correct syntax of a CAST expression is "CAST(something AS type)".
But in this case, for the division to be done with floating-point numbers, it is sufficient for at least one of the operands to be a floating-point number:
SELECT *, (int_EndTime - int_StartTime) / 60.0 ...
I am using sql server 2000 and facing round function issue like the following statement working fine.
SELECT ROUND(5 * 7.83, 1)
The result will be 39.2
But when I get these values from the table, it gives 39.1, meaning it truncates and does not round up.
SELECT ROUND(rate * qty, 1)
FROM tbl
The result will be 39.1
rate and qty columns data types are float. Insert 5 in qty and 7.83 in rate, then check it. How I can fix it?
Convert the table values to real,
SELECT ROUND(convert(real,rate)*convert(real,qty),1)
Your sample simply query is not reflective of the data types involved.
Try these two instead:
SELECT ROUND(5 * 7.83, 1)
SELECT ROUND(cast(5 as float) * cast(7.83 as float), 1)
The 2nd one matches your table data types. Float datatypes are not meant for precise decimal calculations, use a decimal type for those instead.
What Every Computer Scientist Should Know About Floating-Point Arithmetic
Without losing too much precision for normal numbers, you can just cast to decimal on the fly to force human-comprehensible decimal arithmetics, e.g.
SELECT ROUND(cast(rate as decimal(10,5)) * cast(qty as decimal(10,5), 1)
FROM tbl