DECLARE #TAX VARCHAR(30)=120.45
DECLARE #TaxRoundOf VARCHAR(30)
SET #TaxRoundOf=ROUND(#TAX,1)
SELECT #TaxRoundOf
This Gives Result (#TaxRoundOf=120.5)
DECLARE #TAX VARCHAR(30)=146.45
DECLARE #TaxRoundOf VARCHAR(30)
SET #TaxRoundOf=ROUND(#TAX,1)
SELECT #TaxRoundOf
This Gives Result (#TaxRoundOf=146.4)
But I need to return 146.50 . why this mismatch between two results?
any one can help plz?
Since you are using VARCHAR to store your numbers, SQL Server is having to do implicit conversion to float behind the scenes, which is having knock on effects on your calculations. You can reproduce this using the below query:
SELECT ROUND(CONVERT(FLOAT, 120.45),1), -- 120.5
ROUND(CONVERT(FLOAT, 146.45),1), -- 146.4
ROUND(CONVERT(DECIMAL(10, 2), 120.45),1), -- 120.50
ROUND(CONVERT(DECIMAL(10, 2), 146.45),1) -- 146.50
Since floating point numbers are not exact, 146.45 cannot be exactly represented as a float, and ends up being stored as a very slightly smaller number, so when this is passed to the round function, it is rounded down, instead of up.
The solution, as demonstrated by the 3rd and 4th columns in the above query, is to use a more precise data type.
You can use this:
SET #TaxRoundOf=ROUND(10 * CAST(#TAX AS FLOAT)) / 10
instead of:
SET #TaxRoundOf=ROUND(#TAX,1)
DEMO
PS as #GarethD already mentioned I wouldn't use #TAX as VARCHAR type.
You can also rely on numeric rounding instead of converting your string to a float, which can lose information.
Cast a string to numeric and then round:
select round(cast('146.45' as numeric(18,2)), 1)
-- 146.50
A decimal constant is already a decimal so there's no need to cast it:
select round(146.45, 1)
-- 146.50
Related
We are undergoing a migration in our tool. Some of the datatypes in our tables have been changed from real to float. To understand the impact I have run the following code:
declare #realVariable as real,
#floatVariable as float
set #realVariable=152304.11999512
set #floatVariable=152304.11999512
select cast(#realVariable AS decimal(15, 4)) as realcol,str(#floatVariable,15,4) as floatcol
The result is different for both real and float variables:
real=152304.1250
float= 152304.1200
How can I make sure that the float result is same as real?
float and real are both approximate-number data types for floating point numeric data.
real is the same as float(24). If you declare float without a suffix, then it means float(53).
So, you are converting from real (which is float(24)) to float(53). You are simply making the numbers more precise.
float(24) has a precision of 7 digits and float(53) has a precision of 15 digits.
See Books Online > float and real (Transact-SQL): https://msdn.microsoft.com/en-GB/library/ms173773.aspx
If we repeat your example and display the variables without conversion, we can see that the real variable is displayed with 7 digits precision, and float variable is displayed with 14 digits precision, which is exactly what you set it to.
DECLARE #realVariable as real,
#floatVariable as float;
-- Set to a number with 14 digits precision
SET #realVariable=152304.11999512;
SET #floatVariable=152304.11999512;
SELECT #realVariable AS '#realVariable', #floatVariable AS '#floatVariable';
The simple answer to your question is that the two numbers are not the same. The one with float(53) is more precise.
Calculating the ABS of the difference of the two numbers and then comparing this to an acceptable threshold is one way to do it. However this would not be satisfactory if the scale can vary widely.
To illustrate why this might not be satisfactory:
r = 0.0001234567
f = 0.00012345678901234
Comparing the difference to a threshold of 0.0005 would not be useful.
So it might be better to use a threshold that is a percentage of the number.
DECLARE #threshold AS float, #delta AS float;
SET #threshold = ABS(#floatVariable / 1000000);
SET #delta = ABS(#floatVariable - CAST(#realVariable AS float));
SELECT #realVariable AS '#realVariable',
#floatVariable AS '#floatVariable',
#threshold AS '#threshold',
#delta AS '#delta',
CASE
WHEN #delta < #threshold THEN N'OK'
ELSE N'Different'
END AS 'Comparison';
Note: This solution might need a bit of adjustment for numbers that are very close to zero.
#RichardCL's description describes the way things work (+1). Now, if you are converting a value stored as a real datatype value into a value stored as a float datatype value, then the entire and precise "original" value will be stored in the resulting value--you will lose no information from your existing set of real values.
Referring to your example, your value of 152304.11999512 cannot be stored precisely as a real value--it will be rounded and stored as 152304.125. Convert this to a float, and you will still have 152304.125.
You need a delta. Compute the absolute value of the difference between the real and float values then make sure that is less than your delta. The delta should be sufficiently small to meet your needs.
For example using your values:
abs(152304.1250 - 152304.1200) <= .0050 so the use .0050 as your delta value.
I'm trying the following using SQL Server 2008 R2 and SQL Server 2012 and I get the same results in both.
If I write the following statement:
select round(4.005, 2)
I get the expected result: 4.01
However if I write the following statements:
declare #result float
select #result = 4.005
select round(#result, 2)
I get an unexpected result: 4
But if I replace float with real in the previous statements:
declare #result real
select #result = 4.005
select round(#result, 2)
I get the expected result.
Can anyone tell me why is this happening?
because you haven't specify the number of bits that are used to store the mantissa of the float number in scientific notation , try this,
declare #result float(5)
select #result = 4.005
select round(#result, 2)
SQLFiddle Demo
float and real (Transact-SQL)
This has to do with float being an approximate data type.
see: http://technet.microsoft.com/en-us/library/ms187912(v=sql.105).aspx
The float and real data types are known as approximate data types. The behavior of float and real follows the IEEE 754 specification on approximate numeric data types.
Approximate numeric data types do not store the exact values specified for many numbers; they store an extremely close approximation of the value. For many applications, the tiny difference between the specified value and the stored approximation is not noticeable. At times, though, the difference becomes noticeable. Because of the approximate nature of the float and real data types, do not use these data types when exact numeric behavior is required, such as in financial applications, in operations involving rounding, or in equality checks. Instead, use the integer, decimal, money, or smallmoney data types.
Avoid using float or real columns in WHERE clause search conditions, especially the = and <> operators. It is best to limit float and real columns to > or < comparisons.
The IEEE 754 specification provides four rounding modes: round to nearest, round up, round down, and round to zero. Microsoft SQL Server uses round up. All are accurate to the guaranteed precision but can result in slightly different floating-point values. Because the binary representation of a floating-point number may use one of many legal rounding schemes, it is impossible to reliably quantify a floating-point value.
yes I had the same issue. I tested out some solutions...
SELECT round(cast(3.175 as float), 2) as roundingBAD,
CAST(round(TRY_CONVERT(DECIMAL(28,2), cast(3.175 as float)), 2) as
nvarchar(max)) as roundconvertBAD, cast(CAST(round(CAST(3.175 as
DECIMAL(18,10)), 4) as decimal(18,2)) as nvarchar(max)) as
roundconvert3, cast(FORMAT(round(CAST(3.175 as DECIMAL(18,10)), 2),
'0.######') as nvarchar(max)) as roundconvert4, cast(CAST(CAST(3.175
as DECIMAL(18,10)) as decimal(18,2)) as nvarchar(max)) as
roundconvert5P, cast(CAST(CAST(3.175 as DECIMAL(18,10)) as
decimal(18,1)) as nvarchar(max)) as roundconvert1DP
output: 3.17 3.17 3.18 3.18 3.18 3.2
I went with the last two options in my production Microsoft SQL Server code for rounding floats correctly.
First, I need to mention that my current sybase db version is Adaptive Server Enterprise 12.5.4. I aim to convert float data type into varchar via sybase convert function in order to concat several of these kinds of variables, format and store in string type.
Unfortunately, it is not the case. Simply using convert(varchar(20), float_var) or cast() function cannot correctly return the precise value.
For example, ...
declare #float_var float
begin
select #float_var =345.1237 --from table actually
select convert(varchar(20),#float_var) --return 345.1236999999
end
The incorrect string results returned occasionally have 99999 or 00001 suffix.
I tried many function including specify the precision, but there are still several cases not working on it. The sybase internal function does not exactly work on it.
I suppose this is a gerneral issue while using Sybase DB, however few answer found in serach. During my past experience, Sybase store procedure gammer always has sort of tolerance in runtime and internal fix when error encounter. This issue make me confused how Sybase works internally. Any advice would be appreciated, thanks in advance.
there are a couple of possible solutions for this.
firstly, let's try to convert the float to decimal first, then to varchar.
select cast(cast(#float_var as decimal(13,4)) as varchar)
alternatively, and this is where my ASE memory might fail me a little, would be to use the STR function like so:
Select ltrim(str(#float_var, 25, 5))
You have to TRIM the output as the STR function padding empty spaces on to the left of the result
this works for me:
declare #float_var float
begin
select #float_var = 96.332
select cast(cast(#float_var as decimal(13,4)) as varchar) -- Returns 96.3320
end
declare #float_var float
begin
select #float_var = 345.1237
select cast(cast(#float_var as decimal(13,4)) as varchar) -- Returns 345.1237
end
Using SQL Server 2000
I want to reduce the decimal length
Query
Select 23/12 as total
Output is showing as 1.99999999999
I don't want to round the value, I want to diplay like this 1.99
Tried Query
Select LEFT(23/12, LEN(23/12) - 3) as total
The above query is working only if there is decimal value like 12.444444, but if the total is single digit means like 12 or 4 or 11...., i am getting error at run time.
How to do this.
Need Query Help
There is a very simple solution. You can find it in BOL. Round takes an optional 3rd argument, which is round type. The values are round or truncate.
ROUND numeric_expression , length [ ,function ] )
...
function Is the type of operation to perform. function must be
tinyint, smallint, or int. When function is omitted or has a value of
0 (default), numeric_expression is rounded. When a value other than 0
is specified, numeric_expression is truncated.
So just do
Select ROUND(cast(23 as float)/12, 2, 1) as total
That gives 1.91. Note, if you were really seeing 1.999 - something is really wrong with your computer. 23/12 = 1.916666666(ad infinitum). You need to cast one of the numbers as float since sql is assuming they're integers and doing integer division otherwise. You can of course cast them both as float, but as long as one is float the other will be converted too.
Not terribly elegant, but works for all cases:
Select CONVERT(float,LEFT(CONVERT(nvarchar, 23.0/12.0),CHARINDEX('.',CONVERT(nvarchar, 23.0/12.0)) + 2)) as total
Scalar Function
-- Description: Truncate instead of rounding a float
-- SELECT dbo.TruncateNumber(23.0/12.0,2)
-- =============================================
CREATE FUNCTION TruncateNumber
(
-- Add the parameters for the function here
#inFloat float,
#numDecimals smallint
)
RETURNS float
AS
BEGIN
IF (#numDecimals < 0)
BEGIN
SET #numDecimals = 0
END
-- Declare the return variable here
RETURN CONVERT(float,LEFT(CONVERT(nvarchar, #inFloat),CHARINDEX('.',CONVERT(nvarchar, #inFloat)) + #numDecimals))
END
GO
I understand the host of issues in comparing floats, and lament their use in this case - but I'm not the table author and have only a small hurdle to climb...
Someone has decided to use floats as you'd expect GUIDs to be used. I need to retrieve all the records with a specific float value.
sp_help MyTable
-- Column_name Type Computed Length Prec
-- RandomGrouping float no 8 53
Here's my naive attempt:
--yields no results
SELECT RandomGrouping
FROM MyTable
WHERE RandomGrouping = 0.867153569942739
And here's an approximately working attempt:
--yields 2 records
SELECT RandomGrouping
FROM MyTable
WHERE RandomGrouping BETWEEN 0.867153569942739 - 0.00000001
AND 0.867153569942739 + 0.00000001
-- 0.867153569942739
-- 0.867153569942739
In my naive attempt, is that literal a floating point literal? Or is it really a decimal literal that gets converted later?
If my literal is not a floating point literal, what is the syntax for making a floating point literal?
EDIT: Another possibility has occurred to me... it may be that a more precise number than is displayed is stored in this column. It may be impossible to create a literal that represents this number. I will accept answers that demonstrate that this is the case.
EDIT: response to DVK.
TSQL is MSSQLServer's dialect of SQL.
This script works, and so equality can be performed deterministically between float types:
DECLARE #X float
SELECT top 1 #X = RandomGrouping
FROM MyTable
WHERE RandomGrouping BETWEEN 0.839110948199148 - 0.000000000001
AND 0.839110948199148 + 0.000000000001
--yields two records
SELECT *
FROM MyTable
WHERE RandomGrouping = #X
I said "approximately" because that method tests for a range. With that method I could get values that are not equal to my intended value.
The linked article doesn't apply because I'm not (intentionally) trying to straddle the world boundaries between decimal and float. I'm trying to work with only floats. This isn't about the non-convertibility of decimals to floats.
Response to Zinglon:
A literal value that can find my records, thanks.
DECLARE #Y binary(8)
SET #Y = 0x3FEAD9FF34076378
SELECT *
FROM MyTable
WHERE convert(binary(8), RandomGrouping) = #Y
is that literal a floating point literal? Or is it really a decimal
literal that gets converted later?
If my literal is not a floating point literal, what is the syntax for
making a floating point literal?
The 0.867153569942739 literal in SQL Server is a decimal type, not float.
The engine automatically picks appropriate scale and precision to represent the given literal.
To write a literal of the float type you should use the scientific notation, like this:
0.867153569942739E0
This is documented in Constants (Transact-SQL)
decimal constants
decimal constants are represented by a string of
numbers that are not enclosed in quotation marks and contain a decimal
point.
The following are examples of decimal constants:
1894.1204
2.0
float and real constants
float and real constants are represented by
using scientific notation.
The following are examples of float or real values:
101.5E5
0.5E-2
The sp_describe_first_result_set can tell us the types of columns
EXEC sp_describe_first_result_set N'SELECT 0.867153569942739, 0.867153569942739E0'
It returns numeric(15,15) for the first column and float for the second.
If your column RandomGrouping is indexed, it is much more efficient to use a float literal, because when you wrap RandomGrouping in convert(), an index can't be used.
The following query will use an index:
SELECT *
FROM MyTable
WHERE RandomGrouping = 0.867153569942739E0
The following query will not use an index:
SELECT *
FROM MyTable
WHERE convert(binary(8), RandomGrouping) = #Y
It is possible that the values are being truncated on display. I'm assuming the column doesn't have a unique constraint on it, otherwise the question would be moot. On my setup, SSMS truncates the more precise value in this script.
create table flt ( f float not null primary key )
insert into flt
select 0.111111111111111
union all
select 0.1111111111111111
select f, cast(f as binary(8)) from flt
Similarly, if these values are distinct you can cast them to binary(8) and identify them based on that value, like this:
select f from flt
where cast(f as binary(8)) = 0x3FBC71C71C71C71C
The problem is not whether it's a floating point literal or not.
The problem is that comparing two floats for equality in Sybase (or any DB server) is not deterministic, since 4.00000000000000000000... and 3.99999999999999999999... are the same exact number but aren't equal.
Your second solution is the only correct way to compare floats for "equality" (that is, are they the same up to a precision).
Why are you saying "approximately working" about your second approach?
Since you didn't provide the specific DB server you use, here's a fairly decent write-up of the problem (with basically the same conclusions as above) for MySQL
http://dev.mysql.com/doc/refman/5.0/en/problems-with-float.html