I have a value of: "2.54334881002458E-37" and i keep getting "overflow" exception when i'm using a double.
what should i use to make this work?
Thank you
code snippet:
Dim curries, act, cat As Double
For Each dataRow As DataRow In dt.Rows
curries = dataRow("Activity")
getting the error when i try to assign Activity to curries.
but "activity" is a string in the database....
Double is already 64 bits worth of floating point number.
Can you post code where you are getting this overflow?
Decimal might be worth a shot, but you have to post code so that we can understand the issues you are encountering.
Based on your edit in your post, why are you storing numbers as strings in your database? That is a definite no no...unless you are not doing any sort of arithmaetic operation only then can you store them as varchar / string.
Give us a sample of what the data looks like...I think your issue stems from not converting the string to a decimal, if activity is a string convert it using DirectCast or CType (cast the value):
curries = CType(datarow("Activity"), Double)
Change your unit of measure, so that you're not working in 10^-37 of whatever it is you're dealing with. This problem just screams "I'm not solving this in the appropriate domain."
According to MSDN, Double should have no problem at all with your number:
Range:
-1.79769313486231570E+308 through -4.94065645841246544E-324 † for negative values;
4.94065645841246544E-324 through 1.79769313486231570E+308 † for positive values
and if the need to have large numbers and precise I suggest the use of decimal numbers, more information here.
http://msdn.microsoft.com/en-us/library/system.decimal.aspx
Bye
Related
The below line of code is returning as a 1 instead of a 2, for reasons I can't comprehend.
console.write(Format((87.20 \ 43.60)))
Surely this should return the result of 2 but I've checked in another environment and it returns a can anyone tell me why?
I have tried putting the code into a second environment but the result was the same I don't understand why it is returning 1 instead of 2, can anyone enlighten me?
Thanks for the help but found the answer.
Decimals are converted to Long before Integer division and due to this are subject to banker's rounding, multiplying both numbers by a 100 before the operation resolves the issue.
Source of information:
Learn Microsoft - VB.Net - \ Operator - Remarks
Thanks,
While I am glad you resolved your own question, I did want to provide an alternative.
When you use the integer division operator, you do not have control as to how the rounding should occur. Whereas if you do a floating-point division operator you can keep the precision and then use one of the built-in Math class methods (documentation) to specify how the number should round.
Take a look at this example:
Dim result = 87.20 / 43.60
Dim roundUp = Math.Ceiling(result)
Dim roundDown = Math.Floor(result)
Dim bankersRounding = Math.Round(result)
Fiddle: https://dotnetfiddle.net/LZEMXV
Because in your example you are using Console.Write (which treats the data as a String) you do not need to cast the value to an Integer data type. However, if you needed the value as an Integer, any one of those variables outside result can be safely converted to an Integer using the Parse method (documentation).
What is the difference between:
Dim a As Integer = CInt(2.2)
and
Dim a As Integer = Math.Round(2.2)
?
CInt returns an integer but will round the .5 to nearest even number so:
2 = CInt(2.5)
4 = CInt(3.5)
Are both true, which might not be what you want.
Math.Round can be told to round away from zero. But returns a double, so we still need to cast it
3 = CInt(Math.Round(2.5, MidpointRounding.AwayFromZero))
There is bigger differences in CInt(), Int() and Round()... and others.
Round has parameters of rounding, so it is flexible and user friendly. But it do not change variable type. No "type conversion".
Meanwhile CInt() is a bit cryptic as it rounds too. And it is doing "Type conversion" to integer.
2 = Int(2.555), 3 = CInt(2.555)
2 = Int(2.5), 2 = CInt(2.5)
Some documentation states:
When the fractional part of expression is exactly .5, CInt always rounds it to the nearest even number. For example, .5 rounds to 0, and 1.5 rounds to 2.
But I do not like that "exact 0.5", in real word it is "0.5000001"
So, doing integer math (like calculating bitmaps address Hi and Lo bytes) do not use CInt(). Use old school INT(). Until you get to negative numbers... see the fix() function.
If there is no need to convert type, use floor().
I think all this chaos of number conversion is for some sort of compatibility with some ancient software.
The difference between those two functions is that they do totally different things:
CInt converts to an Integer type
Math.Round rounds the value to the nearest Integer
Math.Round in this instance will get you 2.0, as specified by the MSDN documentation. You are also using the function incorrectly, see the MSDN link above.
Both will raise an Exception if conversion fails, you can use Try..Catch for this.
Side note: You're new to VB.NET, but you might want to try switching to C#. I find that it is a hybrid of VB.NET & C++ and it will be far easier for you to work with than VB.NET.
I've got a column called Amount, with a lot of numbers looking like this:
67000.00000000000000000000
Some of the columns have 2 numbers after the decimal that need to be retained.
Which should amount to $67,000.00
But my problem is, when I format it into currency or numbers, I get MUCH larger numbers than i would like, looking like this:
6.700.000.000.000.000.000.000.000,00
How can I get it into the right format?
Edit: For this scenario, the user was using ACC2013 and the Field Type was Short Text. The method of conversion that succeeded was : CCur(Val(FieldNameHere))
CCur(YourFieldName)
This will convert it to a currency format.
CLng(YourFieldName)
This will convert it to a long integer format. (It will cut off the decimals)
If you're looking for a reference, Microsoft has a few examples and goes into brief detail about some of these conversion functions.
CCur(Replace("67000.00000000000000000000", ".", Format(0, ".")))
You have to replace point symbol to actual decimal separator before conversion. Because you can't know actual seprator, choosen in regional settings, you have to find it out - and such Format() operation does dirty work.
I need to store an alphanumeric string in an integer column on one of my models.
I have tried:
#result.each do |i|
hex_id = []
i["id"].split(//).each{|c| hex_id.push(c.hex)}
hex_id = hex_id.join
...
Model.create(:origin_id => hex_id)
...
end
When I run this in the console using puts hex_id in place of the create line, it returns the correct values, however the above code results in the origin_id being set to "2147483647" for every instance. An example string input is "t6gnk3pp86gg4sboh5oin5vr40" so that doesn't make any sense to me.
Can anyone tell me what is going wrong here or suggest a better way to store a string like the aforementioned example as a unique integer?
Thanks.
Answering by request form OP
It seems that the hex_id.join operation does not concatenate strings in this case but instead sums or performs binary complement of the hex values. The issue could also be that hex_id is an array of hex-es rather than a string, or char array. Nevertheless, what seems to happen is reaching the maximum positive value for the integer type 2147483647. Still, I was unable to find any documented effects on array.join applied on a hex array, it appears it is not concatenation of the elements.
On the other hand, the desired result 060003008600401100500050040 is too large to be recorded as an integer either. A better approach would be to keep it as a string, or use different algorithm for producing a number form the original string. Perhaps aggregating the hex values by an arithmetic operation will do better than join ?
Can any one please help me how to get float value as it is from text box
for Ex: I have entered 40.7
rateField=[[rateField text] floatValue];
I am getting rateField value as 40.7000008 but I want 40.7 only.
please help me.
thanks in advance
Thanks Every body,
I tried all the possibilities but I am not able to get what I want. I am not looking to print the value to convert into string.I want to use that value for computation. If i use Number Formatter again when i am converting from number to float it is giving same problem.So i want float value only but it should be whatever i have given in the text box it should not be padded with any values.This is my requirement.Please help me.
thanks®ards Balu
Thanks Every body,
I tried all the possibilities but I am not able to get what I want. I am not looking to print the value to convert into string.I want to use that value for computation. If i use Number Formatter again when i am converting from number to float it is giving same problem.So i want float value only but it should be whatever i have given in the text box it should not be padded with any values.This is my requirement.Please help me.
thanks®ards
Balu
This is ok. There is not guaranteed that you will get 40.7 if you will use even double.
If you want to output 40.7 you can use %.1f or NSNumberFormatter
Try using a double instead. Usually solves that issue. Has to do with the storage precision.
double dbl = [rateField.text doubleValue];
When using floating point numbers, these things can happen because of the way the numbers are stored in binary format in the computers memory.
It's similar to the way 1/3 = 0.33333333333333... in decimal numbers.
The best way to deal with this is to use number formatters in the textbox that displays the value.
You are already resolved float value.
Floating point numbers have limited precision. Although it depends on
the system, float relative error due to rounding will be around 1.1e-8
Non elementary arithmetic operations may give larger errors, and, of
course, error progragation must be considered when several operations
are compounded.
Additionally, rational numbers that are exactly representable as
floating point numbers in base 10, like 0.1 or 0.7, do not have an
exact representation as floating point numbers in base 2, which is
used internally, no matter the size of the mantissa. Hence, they
cannot be converted into their internal binary counterparts without a
small loss of precision. This can lead to confusing results: for
example, floor((0.1+0.7)*10) will usually return 7 instead of the
expected 8, since the internal representation will be something like
7.9999999999999991118....
So if you're using those numbers for output, you should use some rounding mechanism, even for double values.