My code tries to multiply 12303 by 174596.
Any decent calculator is capable of providing an answer to this, so why do I get an OverflowException? It also happens when I execute it directly in the Immediate Window.
The code is meant to determine the position of a certain value in a binary file. The file itself is 7 Gb in size.
Is there any way to solve this?
Dim position As Long = hisFileHeader.StreamStartDataPosition +
(TSIdx * hisFileHeader.StreamDataBlockSize)
tsidx has a value of 12303 and StreamDataBlockSize has a value of 174596
I'm guessing that tsidx and StreamDataBlockSize are Integer types. The largest number an Integer type can hold is 2,147,483,647. The multiplication in brackets is then done expecting an integer result, but the answer is out of the range of Integer types. Change your code to ..
Dim position As Long = hisFileHeader.StreamStartDataPosition + (CLng(TSIdx) * hisFileHeader.StreamDataBlockSize)
and the multiplication will be done with the expectation of a Long type.
Related
How would I go about storing a very large number in number format and not scientific.
Please bear in mind that the number I will be storing is too big for the Long data type.
I've set it as a String.
I have a userform with a command button and a textbox.
Below is the sample code:
Private Sub Cmd_Click()
Dim bits As Integer
Dim out As String
bits = 64
out = 2 ^ (bits - 1)
Txt_Output.Value = out
End Sub
The above will return: 9.22337203685478E+18.
But I want 9223372036854775807.
Can anyone explain how to avoid this?
Thanks in advance.
P.S. I'm hoping to avoid having to use an array.
You can achieve that specific calculation using Decimal data types and a modification to the calculation routine:
Private Sub Cmd_Click()
Dim bits As Integer
Dim out As Variant
Dim i As Long
bits = 64
out = CDec(1)
For i = 1 to bits - 1
out = out * 2
Next
Txt_Output.Value = out
End Sub
By forcing out to be a Variant/Decimal, the calculation does not lose precision as it is being calculated. However some things, such as CDec(2) ^ CDec(63) would still lose precision as the calculation would be done using an intermediate Double precision, so you will need to be very careful as to what calculations you do.
This might give you clues as to how to generalise that method to achieve what you need.
If you have 64-bit Excel, you can use the LongLong data type.
It seems improbable that this is not a duplicate of a question that already has an answer but I can't find that question or answer so here I go...
In VBA, in the immediate window, if I type:
?8/7
I get the result:
1.14285714285714 which if I then multiply by 7 gives me a number that is slightly LESS than 8, i.e. 7.99999999999998. My pocket calculator provides more decimal places, so it's better than VBA? Right? ;-)
However, if I add 0.000000000000003 to the RESULT 1.142... before I multiply I get 8 (which, by the way, is also incorrect); but my question is:
How can I control the precision of the answer with respect to the number of decimal places shown for ?8/7?
I have seen this answer relating to the accuracy of floating points but it is a different question, it explains the reasons for the inaccuracies whereas I am interested in getting a few extra digits.
To this end I have tried writing a function that returns a double but it does not return enough decimal places... for example I'd like to see: 1.142857142857142857. I also found references to a decimal data type but I think the article is for VB6 (instead of VBA) think I have exhausted the available data types in VBA... what am I missing?
Function test() As Double
Dim a As Double
Dim b As Double
Dim c As Double
a = 8
b = 7
c = a / b
test = c
End Function
Graphing and scientific calculators are not built for speed. They can afford to use software implementation of decimal floats for their calculations rather than IEEE doubles. So it is quite possible that your pocket calculator has greater precision than VBA, but at a cost. What they do is likely to be more than an order of magnitude slower.
VBA lacks a built-in arbitrary precision floating point library, but its decimal type supports higher precision than doubles. Unfortunately, it isn't a full-fledged data type but is rather a subtype of Variant. To create them, you need to use CDec():
Sub test()
Dim x As Variant
x = CDec(8) / 7
Debug.Print x
End Sub
This displays 1.1428571428571428571428571429
When using Decimal, you need to be aware of the fact that if you are not careful, they can pop back to e.g. a Double depending on how you use them:
Sub test2()
Dim x As Variant
x = CDec(x)
Debug.Print TypeName(x)
x = 8 / 7
Debug.Print x
Debug.Print TypeName(x)
End Sub
Output:
Decimal
1.14285714285714
Double
Using the immediate window to do some debugging, I came across the following which I have simplified for the purpose of this question:
running this command:
?20000*2
produces an 'overflow' error. Let's assume this is because we haven't declared a data type, and VBE has assumed Integer - the result falls outside the boundaries of a signed integer and therefore an overflow occurs.
However if I run:
?39999+1
I get 40000 as expected.
Is this because I've initially started with a Long instead of an Integer (i.e. 20,000 vs 39,999)? And therefore memory is allocated based on the initial input data rather than the calculation result?
That's correct. VBA will take the largest of the input components and allocate memory for the results. Since both of the components in the first example are Int, that's all you get.
You can use a type declaration character to force the VBE to treat a number as a certain data type
?20000&*2
40000
?20000*2&
40000
In both those examples, the & (Long type declaration character) forces the memory allocation to a Long. It doesn't matter if it's the first component or a later one. I think there are some operations that get forced into particular data types. Exponentiation is one of them.
?2^2^2^2^2^2
4294967296
?typename(2^2^2^2^2^2)
Double
Even though all the components are Integers, the results is a Double - even when it doesn't have to be
?typename(2^2)
Double
This implicit typing isn't limited to the Immediate Window. The same overflows can occur in your code:
Sub foo()
Dim x As Long
x = 20000 * 2 'Overflow error
End Sub
Also, when a String is implicitly cast to a numeric type, it's cast as a Double:
?TypeName("123" + 6)
Double
What is the fastest way to convert a String into an array of Short integers representing the character codes?
I am now using this but it can probably be much faster:
Dim shortsarray(mystring.Length - 1) as Short
For i As Integer = 0 To mystring.Length - 1
shortsarray(i) = AscW(mystring.Chars(i))
Next
Thanks.
I do not know what kind of problem you are trying to solve. However, as an alternative, you could convert the string to an array of characters
Dim chars() As Char = mystring.ToCharArray()
Dim shortsarray(mystring.Length - 1) as Short //One calculation
For i As Integer = 0 To mystring.Length - 1 //One calculation, repeats n times
shortsarray(i) = AscW(mystring.Chars(i)) //Three calculations, repeats n times
Next
As a result, this should run in O(3n+1), or O(n). This is linear on the length of the input, and there isn't much improvement you can expect from that since you're doing a character-wise conversion. I think this is probably as good as you can expect, though there may be a library that will just convert the whole word in one go, making your code cleaner.
If you need performance increases, you want to first profile your whole program. This snippet may not be the problem.
Yesterday i had a look at how to set values of variables from nummbers stored in external txt files
the variables then needed to be added up so i used trial and error first
((XVAL) + (NEWVAL))
assuming that XVAL was set to 10 and NEWVAL was set to 20 i expected to get the answer of thirty but waqs presented with the new value of 10 20
VB.net pysicaly added the two values together but i wanted the mathematical product of the two which is ((10) + (20)) = 30
yep its a newb question could anyone explain how to achieve what im affter
XVAL and NEWVAL are strings, so they are simply being concatenated together. You need to convert them to integers, so that VB.NET will treat them as such. To do this, use the Int32.Parse() method.
Dim intXVAL As Integer = Int32.Parse(XVAL)
Dim intNEWVAL as Integer = Int32.Parse(NEWVAL)
Dim result = intXVAL + intNEWVAL
You want to cast them to a number first.
Try CDbl.
See http://msdn.microsoft.com/en-us/library/Aa263426 for more.
edit: Oops, thought you were talking about VBA.
Try using Double.Parse(YOURVALUE) if you're talking about VB.NET.
Have you tried the Val() function?
Val(XVAL) + Val(NEWVAL)
The + operator in VB.NET (for backwards-compatibility reasons) means both add and concatenate depending on the types of the variables it is being used with. With two numeric types (Integer, Single, Double, etc.), it adds the values together as you would expect. However, with String types, it concatenates the two strings.
Presumably, then, your XVAL and NEWVAL variables are String types because they're being read out of a text file, which is causing VB.NET to concatenate them into a new string instead of add them together. To get the behavior you're expecting, you need to convert them to numeric types.
Some of the other answers suggest casting simply casting the string values to numeric types (CInt, CSng, CDbl, etc.), but this may not work as expected if the value contained by your string cannot be converted to number. The Int32.Parse method will throw an exception if the value held by your string cannot be represented as a number. This is especially important to keep in mind if you're reading values from a text file that are not guaranteed to adhere to any particular constraints.
Instead, you probably want to use something like Int32.TryParse, which returns a Boolean value indicating whether or not the conversion succeeded and will not throw an exception.
As you are reading from a text file I assume that you are reading your values out as strings, so when you do this:
((XVAL) + (NEWVAL))
It is effectively concatenating the two strings together. In order to get the mathematical product of the two values these need to be int/integers which is the number type.
There are a number of ways you can do this, but in essence you have to 'cast' the strings to ints and then do your calculation.
So in vb.net it would be something like this (pseudo code):
Dim xval As String = "10"
Dim newval As String = "20"
Dim x As Integer = Int32.Parse(xval)
Dim n As Integer = Int32.Parse(newval)
Dim prod As Integer = x + n
Console.WriteLine(prod)
There are a number of other methods of doing this, for example using:
int.Parse(...)
or
Integer.TryParse(...)
More information on these sorts of type conversions can be found here:
http://dotnetperls.com/integer-parse-vbnet
One thing to bear in mind with these sorts of conversions is that you have to be certain that your input data is convertable. Otherwise your code will throw exceptions. This is where TryParse is useful as you can use this to check the inputs and handle invalid inputs without the need for exceptions.