The sum of 2 floating values in postgres gives the result with higher precision.
Expected:
669.05 + 1.64 = 670.69
Actual:
SELECT CAST(669.05 AS FLOAT) + CAST(1.64 AS FLOAT)
------------------
670.6899999999999
The result is having higher precision than expected.
The same operation with different set of inputs behaves differently.
SELECT CAST(669.05 AS FLOAT) + CAST(1.63 AS FLOAT)
------------------
670.68
Here I have reduced the problem statement by finding the 2 numbers for which the issue exists.
The actual problem is when I do this on a whole table the result would be very big with higher precisions (depending on the values, and I do not have an explanation for exactly what/kind of values the precision shoots up) and we had to handle the scale in the application level.
Example query
SELECT numeric_column_1/ CAST(numeric_column_2 AS FLOAT) FROM input_table;
Note: The behaviour is same for FLOAT(53) as well.
As per postgresql documentation, float uses inexact precision. Better to use DECIMAL or NUMERIC, which supports exact user specified precision.
SELECT CAST(669.05 AS numeric) + CAST(1.64 AS numeric)
Floating-Point Types in PostgreSQL
The data types real and double precision are inexact,
variable-precision numeric types. On all currently supported
platforms, these types are implementations of IEEE Standard 754 for
Binary Floating-Point Arithmetic (single and double precision,
respectively), to the extent that the underlying processor, operating
system, and compiler support it.
Numeric Types
Name
Storage Size
Description
Range
smallint
2 bytes
small-range integer
-32768 to +32767
integer
4 bytes
typical choice for integer
-2147483648 to +2147483647
bigint
8 bytes
large-range integer
-9223372036854775808 to +9223372036854775807
decimal
variable
user-specified precision, exact
up to 131072 digits before the decimal point; up to 16383 digits after the decimal point
numeric
variable
user-specified precision, exact
up to 131072 digits before the decimal point; up to 16383 digits after the decimal point
real
4 bytes
variable-precision, inexact
6 decimal digits precision
double precision
8 bytes
variable-precision, inexact
15 decimal digits precision
smallserial
2 bytes
small autoincrementing integer
1 to 32767
serial
4 bytes
autoincrementing integer
1 to 2147483647
bigserial
8 bytes
large autoincrementing integer
1 to 9223372036854775807
DB Fiddle: Try it here
Related
Internal error occurred during runtime generation of Program (Dump ID: BCD_OVERFLOW)
No error during check but activation gives this error.
This issue occurs in any ABAP code if you try to assign a value to a numeric attribute or variable, which is "out of range" (so leads to an overflow). See here all the possible value range for numeric types:
Type Value Range
---------- ------------------------------------------------------------------------------
b 0 to 255
s -32,768 to +32,767
i -2,147,483,648 to +2,147,483,647
int8 -9,223,372,036,854,775,808 to +9,223,372,036,854,775,807
p The valid length for packed numbers is between 1 and 16 bytes. Two places are
packed into one byte, where the last byte contains only one place and the sign,
which is the number of places or places calculated from 2 * len1. After the
decimal separator, up to 14 decimal places are allowed ( the number of decimal
places should not exceed the number of places). Depending on the field length
len and the number of decimal places dec, the value range is: (-10^(2len-1)
+1) / (10^(+dec)) to (+10^(2len-1) -1) /(10^(+dec)) in increments of 10^(-dec).
Any intermediate values are rounded decimally. Invalid content produces undefined
behavior.
decfloat16 Decimal floating point numbers of this type are represented internally with 16
places in accordance with the IEEE-754-2008 standard. Valid values are numbers
between 1E385(1E-16 - 1) and -1E-383 for the negative range, 0 and +1E-383 to
1E385(1 - 1E-16) for the positive range. Values between the ranges form the
subnormal range and are rounded. Outside of the subnormal range, each 16-digit
decimal number can be represented exactly with a decimal floating point number
of this type.
decfloat34 Decimal floating point numbers of this type are represented internally with 34
places in accordance with the IEEE-754-2008 standard. Valid values are numbers
between 1E6145(1E-34 - 1) and -1E-6143 for the negative range, 0 to +1E-6143
and 1E6145(1 - 1E-34) for the positive range. Values between the ranges form
the subnormal range and are rounded. Outside of the subnormal range, each
34-digit decimal number can be represented exactly using a decimal floating
point number.
f Binary floating point numbers are represented internally according to the
IEEE-754 standard (double precision). In ABAP, 17 places are represented (one
integer digit and 16 decimal places). Valid values are numbers between
-1.7976931348623157E+308 and -2.2250738585072014E-308 for the negative range
and between +2.2250738585072014E-308 and +1.7976931348623157E+308 for the
positive range, plus 0. Both validity intervals are extended to the value zero
by subnormal numbers according to IEEE-754. Not every sixteen-digit number can
be represented exactly by a binary floating point number.
Minimal reproducible example:
REPORT ztest.
DATA num TYPE int1.
num = 1000. " <=== run time error
The solution is to use a larger data type like int2 (up to 32767) or I (integer on 4 bytes):
REPORT ztest.
DATA num TYPE int2. " <=== larger type
num = 1000. " <=== no more error
NB: decfloat34 is the larger possible numeric data type, it can handle virtually any value.
This issue can occur in methods, Function modules, or in Report also.
The main reason behind this issue is the runtime configuration of attributes present in code.
For example,
DATA: num type int1 value 256.
This statement is syntactically fine but the value that is being assigned to variable num is more than the range of type INT1. So it will dump while activation only.
Solution:
DATA: num type int1 value {<=255}
Similarly, this error can occur in any case where the compile-time and runtime configuration conflict.
I encounter strange DB2 behaviour. An example will illustrate it :
SELECT CAST(11458.5648 AS DECIMAL(30,10)) / CAST(120.1 AS DECIMAL(30,10)), 11458.5648 / 120.1 FROM MYTABLE FETCH FIRST 1 ROW ONLY
returns :
1 | 2
---------------------------
95.4 | 95.4085328893
Of course, the good result is in column 2, but why DB2 does this awful error ?
If I cast to DECIMAL(20,10), the result is good too, but starting with DECIMAL(22,10), I loose 1 digit of precision in the result at each step ...
Any idea about it ?
thanks
You need to understand how decimal arithmetic is handled (for your platform & version of Db2)
For Db2 LUW v11.5
Two decimal operands If both operands are decimal, the operation is
performed in decimal. The result of any decimal arithmetic operation
is a decimal number with a precision and scale that are dependent on
the operation and the precision and scale of the operands. If the
operation is addition or subtraction and the operands do not have the
same scale, the operation is performed with a temporary copy of one of
the operands. The copy of the shorter operand is extended with
trailing zeros so that its fractional part has the same number of
digits as the longer operand.
The result of a decimal operation cannot have a precision greater than
31. The result of decimal addition, subtraction, and multiplication is derived from a temporary result which can have a precision greater
than 31. If the precision of the temporary result is not greater than
31, the final result is the same as the temporary result.
Decimal arithmetic in SQL Use the formulas shown here to calculate the
precision and scale of the result of decimal operations in SQL. The
formulas use the following symbols:
p Precision of the first operand.
s Scale of the first operand.
p' Precision of the second operand.
s' Scale of the second operand.
Assuming the default mode, by casting the operands to decimal(30,10) your results has
p = 31
s = 31-30+10-10 ==> 1
moral of the story, don't artificially increase the precision and scale of your operands.
I have created a table where a column has the format NUMBER(2,3).
I try to insert the value 5.73 but it doesn't work.
The error is :
ORA-01438 - "value larger than specified precision allowed for this column"
Cause: When inserting or updating records, a numeric value was entered
that exceeded the precision defined for the column.*
I read the documentation but i don't understand the scale.
So, what is the format accepted value 0-99 with 3 values after the decimal point ?
Thanks.
You are misunderstanding precision and scale. You have a number with a precision of 2. That means that there are two significant digits. It has a scale of 3, which means that these are to the right of the decimal point.
So, your column can represent values between 0.000 and 0.099
What you want is NUMERIC(5, 3). "precision - scale" is the number of digits to the left of the decimal point.
This has come from here:
https://docs.oracle.com/cd/B28359_01/server.111/b28318/datatype.htm#CNCPT1832
Optionally, you can also specify a precision (total number of digits) and scale (number of >digits to the right of the decimal point):
column_name NUMBER (precision, scale)
So in your example you are allowed a total number of 2 digits ( and 3 digits to the right of the decimal point). Which doesn't work for 5.73, perhaps you need a type of number(3,2) which would allow 3 digits 2 of which can be right of the decimal point.
I've given a task of exporting data from an Oracle view to a fixed length text file, however I've been given specification of how data should be exported to a text file. I.e.
quantity NUM (10)
price NUM (8,2)
participant_id CHAR (3)
brokerage NUM (10,2)
cds_fees NUM (8,2)
My confusion arises in Numeric types where when it says (8,2). If I'm to use same as text, does it effectively means
10 characters (as to_char(<field name>, '9999999.99'))
or
8 characters (as to_char(<field name>, '99999.99'))
when exporting to fixed length text field in the text file?
I was looking at this question which gave an insight, but not entirely.
Appreciate if someone could enlighten me with some examples.
Thanks a lot.
According to the Oracle docs on types
Optionally, you can also specify a precision (total number of digits)
and scale (number of digits to the right of the decimal point):
If a precision is not specified, the column stores values as given. If
no scale is specified, the scale is zero.
So in your case, a NUMBER(8,2), has got:
8 digits in total
2 of which are after the decimal point
This gives you a range of -999999.99 to 999999.99
I assume that you mean NUMBER data type by NUM.
When it says NUMBER(8,2), it means that there will be 8 digits, and that the number should be rounded to the nearest hundredth. Which means that there will be 6 digits before, and 2 digits after the decimal point.
Refer to oracle doc:
You use the NUMBER datatype to store fixed-point or floating-point
numbers. Its magnitude range is 1E-130 .. 10E125. If the value of an
expression falls outside this range, you get a numeric overflow or
underflow error. You can specify precision, which is the total number
of digits, and scale, which is the number of digits to the right of
the decimal point. The syntax follows:
NUMBER[(precision,scale)]
To declare fixed-point numbers, for which you must specify scale, use
the following form:
NUMBER(precision,scale)
To declare floating-point numbers, for which you cannot specify
precision or scale because the decimal point can "float" to any
position, use the following form:
NUMBER
To declare integers, which have no decimal point, use this form:
NUMBER(precision) -- same as NUMBER(precision,0)
You cannot use constants or variables to specify precision and scale;
you must use integer literals. The maximum precision of a NUMBER value
is 38 decimal digits. If you do not specify precision, it defaults to
38 or the maximum supported by your system, whichever is less.
Scale, which can range from -84 to 127, determines where rounding
occurs. For instance, a scale of 2 rounds to the nearest hundredth
(3.456 becomes 3.46). A negative scale rounds to the left of the
decimal point. For example, a scale of -3 rounds to the nearest
thousand (3456 becomes 3000). A scale of 0 rounds to the nearest whole
number. If you do not specify scale, it defaults to 0.
NUMBER(p,s)
p(precision) = length of the number in digits
s(scale) = places after the decimal point
So Number(8,2) in your example is a '999999.99'
You can see more examples here.
I have the following column specified in a database: decimal(5,2)
How does one interpret this?
According to the properties on the column as viewed in SQL Server Management studio I can see that it means: decimal(Numeric precision, Numeric scale).
What do precision and scale mean in real terms?
It would be easy to interpret this as a decimal with 5 digits and two decimals places...ie 12345.12
P.S. I've been able to determine the correct answer from a colleague but had great difficulty finding an answer online. As such, I'd like to have the question and answer documented here on stackoverflow for future reference.
Numeric precision refers to the maximum number of digits that are present in the number.
ie 1234567.89 has a precision of 9
Numeric scale refers to the maximum number of decimal places
ie 123456.789 has a scale of 3
Thus the maximum allowed value for decimal(5,2) is 999.99
Precision of a number is the number of digits.
Scale of a number is the number of digits after the decimal point.
What is generally implied when setting precision and scale on field definition is that they represent maximum values.
Example, a decimal field defined with precision=5 and scale=2 would allow the following values:
123.45 (p=5,s=2)
12.34 (p=4,s=2)
12345 (p=5,s=0)
123.4 (p=4,s=1)
0 (p=0,s=0)
The following values are not allowed or would cause a data loss:
12.345 (p=5,s=3) => could be truncated into 12.35 (p=4,s=2)
1234.56 (p=6,s=2) => could be truncated into 1234.6 (p=5,s=1)
123.456 (p=6,s=3) => could be truncated into 123.46 (p=5,s=2)
123450 (p=6,s=0) => out of range
Note that the range is generally defined by the precision: |value| < 10^p ...
Precision, Scale, and Length in the SQL Server 2000 documentation reads:
Precision is the number of digits in a number. Scale is the number of digits to the right of the decimal point in a number. For example, the number 123.45 has a precision of 5 and a scale of 2.
Precision refers to the total number of digits while scale refers to the digits allowed after the decimal.
The example quoted by would have a precision of 7 and a scale of 2.
Moreover, DECIMAL(precision, scale) is an exact value data type unlike something like a FLOAT(precision, scale) which stores approximate numeric data.
For example, a column defined as FLOAT(7,4) is displayed as -999.9999. MySQL performs rounding when storing values, so if you insert 999.00009 into a FLOAT(7,4) column, the approximate result is 999.0001.
Let me know if this helps!