I can find a workaround, but it is really annoying and I may certainly be missing something. Redshift's ROUND function doesn't round to the number of decimals specified.
For example,
select round(cast(176 as float)/cast(492 as float),4) as result;
Above select statement will return 0.35769999999999996.
However, this statement:
select round(cast(229 as float)/cast(491 as float),4) as result;
... will return 0.4664.
Why? I can work around this, but seems like it should work and return only four decimal places.
If your issues is all those 9999s, then the issue is floating point representation. Convert to a decimal to get fixed-point precision:
select round(cast(176 as float)/cast(492 as float), 4)::decimal(10, 4) as result;
Elaborating more on Gordon's answer -
So you’ve written some absurdly simple code, say for example:
0.1 + 0.2
and got a really unexpected result:
0.30000000000000004
Because internally, computers use a format (binary floating-point) that cannot accurately represent a number like 0.1, 0.2 or 0.3 at all.
When the code is compiled or interpreted, your “0.1” is already rounded to the nearest number in that format, which results in a small rounding error even before the calculation happens.
What can I do to avoid this problem?
That depends on what kind of calculations you’re doing.
If you really need your results to add up exactly, especially when you work with money: use a decimal datatype.
If you just don’t want to see all those extra decimal places: simply format your result rounded to a fixed number of decimal places when displaying it.
Shamelessly stolen from : Floating Point
try multiplying by 10 to the power of your desired places after the decimal point, rounding, and then dividing it out again:
-- exclude decimal point inside ROUND(), include outside ROUND()
SELECT ROUND(10000 * 176 / 492) / 10000.0
which will return the expected 0.3577.
Related
I used String.format() to round to the third decimal place. However this didn't work and I solved it using DecimalFormat.
Is there anything I implemented wrong?
val value = 23.695f
Timber.e("format: ${"%.2f".format(value)}")
Expect: 23.70
Result: 23.69
In your string formatting example you're not rounding the number you're just taking the first two decimal places. So your code drops the .005 and keeps the .69
If .70 is the desired result then DecimalFormat sounds correct to me
Executing the following statement results in Access SQL:
CLNG((CCUR(1.225)/1)*100) = 123
The Conversion Goes, Decimal > Currency > Double > Double > Long
If I remove the CCUR conversion function:
CLNG(((1.225)/1)*100) = 122
The Conversion here goes , Decimal > Double > Double > Long
What is the difference between these two?
This extends to being different between Code And Access SQL
In Access SQL
clng((CCUR(1.015)/1)*100)/100 = 1.01 (Wrong Rounding)
In Access VBA
clng((CCUR(1.015)/1)*100)/100 = 1.02 (Appropriate Rounding Here)
Microsoft explain that the CLng function uses Banker's Rounding, here.
When the fractional part is exactly 0.5, CInt and CLng always round it to the nearest even number. For example, 0.5 rounds to 0, and 1.5 rounds to 2. CInt and CLng differ from the Fix and Int functions, which truncate, rather than round, the fractional part of a number. Also, Fix and Int always return a value of the same type as is passed in.
Looking at a similar question and the subsequent answer HERE, it explains that there are changes to the bit calculation behind the scenes, based on how it is calculated, but I'm not sure how the data type effects it.
What am I missing, and why is it calculating this way? How could I reproduce this behavior predictably in SQL Server?
EDIT
After some digging I believe that this is truly the result of a rounding point issue. In SQL server it will round floats to the nearest whole number if it is outside of the 15 digit max of precision. Access seems to hold more somehow, even though a Double is equivalent to a Float(53) in TSQL.
The difference in results is a combination of two different issues: Jet/ACE vs VBA expression evaluation and binary floating point representation of decimal numbers.
The first is that the Jet/ACE expression engine implicitly converts fractional numbers to Decimal while VBA converts them to Double. This can be easily demonstrated (note the Eval() function evaluates an expression using the Jet/ACE db engine):
?Typename(1.015), eval("typename(1.015)")
Double Decimal
The second issue is that of floating point arithmetic. This is somewhat more difficult to demonstrate because VBA always rounds its output, but the issue is more obvious using another language (Python, in this case):
>>> from decimal import Decimal
>>> Decimal(1.015)
Decimal('1.0149999999999999023003738329862244427204132080078125')
The Double type in VBA uses floating-point arithmetic, while the Decimal type uses integer arithmetic (it stores the position of the decimal point behind the scenes).
The upshot to this is that Banker's rounding or traditional rounding is a red herring. The determining factor is whether the binary floating point representation of the number is slightly greater or less than its decimal representation.
To see how this works in your original question see the following VBA:
?Eval("typename((CCUR(1.225)/1))"), Eval("typename(((1.225)/1))")
Double Decimal
?Eval("typename(CCUR(1.225))"), Eval("typename(1.225)")
Currency Decimal
And Python:
>>> Decimal(1.225)
Decimal('1.225000000000000088817841970012523233890533447265625')
I should also point out that your assumption of the conversion to Double in your second example is incorrect. The data type remains Decimal until the final conversion to Long. The difference between the first two functions is that multiplying a Decimal by a Currency type in Jet/ACE results in a Double. This seems like somewhat odd behavior to me, but the code bears it out:
?eval("TypeName(1.225)"), eval("TypeName(1.225)")
Decimal Decimal
?eval("TypeName(CCUR(1.225))"), eval("TypeName((1.225))")
Currency Decimal
?eval("TypeName(CCUR(1.225)/1)"), eval("TypeName((1.225)/1)")
Double Decimal
?eval("TypeName((CCUR(1.225)/1)*100)"), eval("TypeName(((1.225)/1)*100)")
Double Decimal
?eval("TypeName(CLNG((CCUR(1.225)/1)*100))"), eval("TypeName(CLNG(((1.225)/1)*100))")
Long Long
So the conversion in the two cases is actually:
Decimal > Currency > Double > Double > Long (as you correctly assumed); and
Decimal > Decimal > Decimal > Decimal > Long (correcting your initial assumption).
To answer your question in the comment below, Eval() uses the same expression engine as Jet/ACE, so it is functionally equivalent to entering the same formula in an Access query. For further proof, I present the following:
SELECT
TypeName(1.225) as A1,
TypeName(CCUR(1.225)) as A2,
TypeName(CCUR(1.225)/1) as A3,
TypeName((CCUR(1.225)/1)*100) as A4,
TypeName(CLNG((CCUR(1.225)/1)*100)) as A5
SELECT
TypeName(1.225) as B1,
TypeName((1.225)) as B2,
TypeName((1.225)/1) as B3,
TypeName(((1.225)/1)*100) as B4,
TypeName(CLNG(((1.225)/1)*100)) as B5
I know about the Math.Round and Math.Ceiling methods, but they return a Double and a Decimal. Does VB.NET have any built-in functions which always round a floating point number up, not down, with a return type of Integer? I know there's CInt, but this can round down if it's below 6.5.
From the comments I understood you wanted to round 6.1 to 7.
Just add 1 and truncate.
If it looks awkward, create a method for it.
Correction:
Unless the number already is truncated.
Addendum:
Note here that doing == with floats is not without problem; you should always have some sort of precision when trying == with floats.
Now when you have decided on this precision - then you can rewrite your code to add 0.999999 (according to precision) and the first add-0.9999-and-truncat works.
Note again that adding 0.9999 is does not really mean you add 0.9999 with our "normal" float.
So if you really want to add 0.9999 you have to work with BND and/or some monetary arithmetics.
Which you "always" should do when calculating money (or any exact decimal stuff)
Can any one please help me how to get float value as it is from text box
for Ex: I have entered 40.7
rateField=[[rateField text] floatValue];
I am getting rateField value as 40.7000008 but I want 40.7 only.
please help me.
thanks in advance
Thanks Every body,
I tried all the possibilities but I am not able to get what I want. I am not looking to print the value to convert into string.I want to use that value for computation. If i use Number Formatter again when i am converting from number to float it is giving same problem.So i want float value only but it should be whatever i have given in the text box it should not be padded with any values.This is my requirement.Please help me.
thanks®ards Balu
Thanks Every body,
I tried all the possibilities but I am not able to get what I want. I am not looking to print the value to convert into string.I want to use that value for computation. If i use Number Formatter again when i am converting from number to float it is giving same problem.So i want float value only but it should be whatever i have given in the text box it should not be padded with any values.This is my requirement.Please help me.
thanks®ards
Balu
This is ok. There is not guaranteed that you will get 40.7 if you will use even double.
If you want to output 40.7 you can use %.1f or NSNumberFormatter
Try using a double instead. Usually solves that issue. Has to do with the storage precision.
double dbl = [rateField.text doubleValue];
When using floating point numbers, these things can happen because of the way the numbers are stored in binary format in the computers memory.
It's similar to the way 1/3 = 0.33333333333333... in decimal numbers.
The best way to deal with this is to use number formatters in the textbox that displays the value.
You are already resolved float value.
Floating point numbers have limited precision. Although it depends on
the system, float relative error due to rounding will be around 1.1e-8
Non elementary arithmetic operations may give larger errors, and, of
course, error progragation must be considered when several operations
are compounded.
Additionally, rational numbers that are exactly representable as
floating point numbers in base 10, like 0.1 or 0.7, do not have an
exact representation as floating point numbers in base 2, which is
used internally, no matter the size of the mantissa. Hence, they
cannot be converted into their internal binary counterparts without a
small loss of precision. This can lead to confusing results: for
example, floor((0.1+0.7)*10) will usually return 7 instead of the
expected 8, since the internal representation will be something like
7.9999999999999991118....
So if you're using those numbers for output, you should use some rounding mechanism, even for double values.
I here is the problem:
I have a NSString that contain "1.7" (for example) and I have to get the float number = 1.7
I' ve tried with [mystring floatValue] but the result is 1.700000000004576
If I try with "1.74" the result is 1.74000000000000000067484
how can I fix it?
thank you!
You are correctly converting the string into a float. The problem is that floating point numbers cannot represent all real numbers exactly. A direct assignment:
float x = 1.7;
will still have a precision error. That's just how floating point numbers are.
The workaround depends on your needs. Some examples: If you need more precision for mathematical calculations, you can use doubles. If you're trying to generate output for the user, you can format the output so it limits the number of digits shown after the decimal point. If you're dealing with money, you could convert floating point dollar amounts into integer numbers of cents and perform all calculations using integers, only showing a decimal point on output to the user.
Floats need to be able to represent infinitely many real numbers, but a float contains a finite number of bits, so floats are approximations.
See this article for more.
You can trim the answer by using the formatting below.
The .1 will set the result to one decimal place.
NSLog(#"mystring = %.1f ",[mystring floatValue]);
Solved!
I used sqlite3_bind_text even if my database attribute is FLOAT value and I used mystring instead of myfloat and it work fine! Thank you!