Converting bytes to Extended Precision Floating Point not accurate enough - vb.net

Question

I have some legacy files that need mined for data. The files were created by Lotus123 Release 4 for DOS. I'm trying to read the files faster by parsing the bytes rather than using Lotus to open the files. I have value records of 10 bytes each. They are 80 bit Extended Precision Floating Point.
Debug.Print(ConvertLongDouble80(New Byte() {0, 0, 0, 0, 0, 0, 0, 128, 255, 191})) ' Value = -1
Debug.Print(ConvertLongDouble80(New Byte() {205, 204, 204, 204, 204, 204, 204, 204, 251, 191})) ' Value = -0.1
Debug.Print(ConvertLongDouble80(New Byte() {10, 215, 163, 112, 61, 10, 215, 163, 248, 191})) ' Value = -0.01
Debug.Print(ConvertLongDouble80(New Byte() {59, 223, 79, 141, 151, 110, 18, 131, 245, 191})) ' Value = -0.001
Debug.Print(ConvertLongDouble80(New Byte() {44, 101, 25, 226, 88, 23, 183, 209, 241, 191})) ' Value = -0.0001
Debug.Print(ConvertLongDouble80(New Byte() {35, 132, 71, 27, 71, 172, 197, 167, 238, 191})) ' Value = -0.00001
Debug.Print(ConvertLongDouble80(New Byte() {182, 105, 108, 175, 5, 189, 55, 134, 235, 191})) ' Value = -0.000001
Debug.Print(ConvertLongDouble80(New Byte() {0, 0, 0, 0, 0, 0, 0, 128, 255, 63})) ' Value = 1
Debug.Print(ConvertLongDouble80(New Byte() {205, 204, 204, 204, 204, 204, 204, 204, 251, 63})) ' Value = 0.1
Debug.Print(ConvertLongDouble80(New Byte() {10, 215, 163, 112, 61, 10, 215, 163, 248, 63})) ' Value = 0.01
Debug.Print(ConvertLongDouble80(New Byte() {59, 223, 79, 141, 151, 110, 18, 131, 245, 63})) ' Value = 0.001
Debug.Print(ConvertLongDouble80(New Byte() {44, 101, 25, 226, 88, 23, 183, 209, 241, 63})) ' Value = 0.0001
Debug.Print(ConvertLongDouble80(New Byte() {35, 132, 71, 27, 71, 172, 197, 167, 238, 63})) ' Value = 0.00001
Debug.Print(ConvertLongDouble80(New Byte() {182, 105, 108, 175, 5, 189, 55, 134, 235, 63})) ' Value = 0.000001
Debug.Print(ConvertLongDouble80(New Byte() {188, 66, 122, 229, 213, 148, 191, 214, 231, 63})) ' Value = 0.0000001
Function ConvertLongDouble80(ByVal TenBytes As Byte()) As Double
'https://en.wikipedia.org/wiki/Extended_precision
'get 15 bit exponent; remove the first bit which is the negative sign
Dim arrExp As Byte() = New Byte() {TenBytes(8), (TenBytes(9) << 1) >> 1}
Dim Expo As UInt16 = BitConverter.ToUInt16(arrExp, 0)
'flag bits
Dim Bit63 As UInt16 = Convert.ToUInt16(TenBytes(7) >> 7)
Dim Bits63_62 As UInt16 = Convert.ToUInt16(TenBytes(7) >> 6)
'fractional values
TenBytes(7) = (TenBytes(7) << 1) >> 1
Dim Bits62_0 As UInt64 = BitConverter.ToUInt64(TenBytes, 0)
TenBytes(7) = (TenBytes(7) << 2) >> 2
Dim Bits61_0 As UInt64 = BitConverter.ToUInt64(TenBytes, 0)
If Bit63 = 0 and Bits62_0 = 0 Then
Return 0
Else
Const ExponentBias As Integer = 16383
Dim isNegative As Boolean = (TenBytes(9) And (1 << 7)) <> 0
Dim NegVal As Int16 = IIf(isNegative, -1, 1)
Dim expVal As Double = Math.Pow(2, Expo - ExponentBias)
Dim LBits62 As Int16 = Len(CStr(Bits62_0))
Dim mantissa As Double = CDbl("1." & CStr(Bits62_0))
Dim result As Double = NegVal * expVal * mantissa
Return result
End If
End Function
Value -1 converts to -1
Value -0.1 converts to -0.09708764513821
Value -0.01 converts to -0.00983011263306
Value -0.001 converts to -0.00119273528211
Value -0.0001 converts to -0.00009697388128
Value -0.00001 converts to -0.00000981589215
Value -0.000001 converts to -0.00000138095333
Value 1 converts to 1
Value 0.1 converts to 0.09708764513821
Value 0.01 converts to 0.00983011263306
Value 0.001 converts to 0.00119273528211
Value 0.0001 converts to 0.00009697388128
Value 0.00001 converts to 0.00000981589215
Value 0.000001 converts to 0.00000138095333
Value 0.0000001 converts to 0.00000009686278
What am I doing wrong. My values are not close enough. How do I fix this?

Answer 1

The problem is that you treat the fractional part, which represents the fraction of a binary number as a whole number. I changed your function by interpreting the bits 0 to 63 as an integer number which is 263 too big. It's too big by 263 because bit 63 should be bit 0 (instead of 10110101..., we should have 1.0110101..., i.e., the leading 1 has to move 63 positions to the right). I then subtract 63 the the exponent to take this into account. I didn't check the logic related to the flags handling.
Function ConvertLongDouble80(ByVal TenBytes As Byte()) As Double
'https://en.wikipedia.org/wiki/Extended_precision
' 80-bit extended precision format
' --------------------------------
' bit 0 to 63 fraction, bytes 0 to 7
' bit 64 to 78 exponent (bias 16383, bytes 8 to 9)
' bit 79 sign /
'get 15 bit exponent; remove the first bit which is the negative sign
Dim arrExp As Byte() = New Byte() {TenBytes(8), (TenBytes(9) << 1) >> 1}
Dim Expo As UInt16 = BitConverter.ToUInt16(arrExp, 0)
'flag bits
Dim Bit63 As UInt16 = Convert.ToUInt16(TenBytes(7) >> 7)
Dim Bits63_62 As UInt16 = Convert.ToUInt16(TenBytes(7) >> 6)
Dim fraction = BitConverter.ToUInt64(TenBytes, 0) ' 2 ^ 63 too big
If Bit63 = 0 And fraction = 0 Then
Return 0
Else
Const ExponentBias As Integer = 16383
Dim isNegative As Boolean = (TenBytes(9) And (1 << 7)) <> 0
Dim sign = If(isNegative, -1, 1)
Dim expVal As Double = Math.Pow(2, Expo - ExponentBias - 63)
Dim result As Double = sign * expVal * fraction
Return result
End If
End Function