I am trying to calculate milliseconds into seconds for a field. I was using [field]/1000 and that works as long as the value is greater than 1. Once its under ``1 I get 0. So if the value is 460 I get 0 instead 0.46.
I tried the below:
RUNTIME/1000 as test,
CAST(RUNTIME/1000 as DECIMAL(5,2)) as test2
Refer to the Expressions article.
Two integer operands
If both operands of an arithmetic operator are integers, the operation
is performed in binary and the result is a large integer unless either
(or both) operand is a big integer, in which case the result is a big
integer. Any remainder of division is lost. The result of an integer
arithmetic operation (including negation by means of a unary minus
operator) must be within the range of the result type.
Related
Why does np.ceil give me different answers for what should equivalent expressions?
np.ceil(336)
Out[34]: 336.0
np.ceil(100*(2.85+0.43+.08))
Out[35]: 337.0
Ceil
The ceil of the scalar number x is the smallest number i, that is larger or equal to x (i.e., i>=x).
The first expression 336 is having data type int.
Whereas on evaluating the next expression it is found that the result is not 336 but 336.00000000000006 , due to preserving information nature of language , the expression is in which resulted in the ceil being calculated as 337 and not 336.
If you cast the expression to int like this int(100*(2.85+0.43+0.08)) then you will get np.ceil 336.
Internal error occurred during runtime generation of Program (Dump ID: BCD_OVERFLOW)
No error during check but activation gives this error.
This issue occurs in any ABAP code if you try to assign a value to a numeric attribute or variable, which is "out of range" (so leads to an overflow). See here all the possible value range for numeric types:
Type Value Range
---------- ------------------------------------------------------------------------------
b 0 to 255
s -32,768 to +32,767
i -2,147,483,648 to +2,147,483,647
int8 -9,223,372,036,854,775,808 to +9,223,372,036,854,775,807
p The valid length for packed numbers is between 1 and 16 bytes. Two places are
packed into one byte, where the last byte contains only one place and the sign,
which is the number of places or places calculated from 2 * len1. After the
decimal separator, up to 14 decimal places are allowed ( the number of decimal
places should not exceed the number of places). Depending on the field length
len and the number of decimal places dec, the value range is: (-10^(2len-1)
+1) / (10^(+dec)) to (+10^(2len-1) -1) /(10^(+dec)) in increments of 10^(-dec).
Any intermediate values are rounded decimally. Invalid content produces undefined
behavior.
decfloat16 Decimal floating point numbers of this type are represented internally with 16
places in accordance with the IEEE-754-2008 standard. Valid values are numbers
between 1E385(1E-16 - 1) and -1E-383 for the negative range, 0 and +1E-383 to
1E385(1 - 1E-16) for the positive range. Values between the ranges form the
subnormal range and are rounded. Outside of the subnormal range, each 16-digit
decimal number can be represented exactly with a decimal floating point number
of this type.
decfloat34 Decimal floating point numbers of this type are represented internally with 34
places in accordance with the IEEE-754-2008 standard. Valid values are numbers
between 1E6145(1E-34 - 1) and -1E-6143 for the negative range, 0 to +1E-6143
and 1E6145(1 - 1E-34) for the positive range. Values between the ranges form
the subnormal range and are rounded. Outside of the subnormal range, each
34-digit decimal number can be represented exactly using a decimal floating
point number.
f Binary floating point numbers are represented internally according to the
IEEE-754 standard (double precision). In ABAP, 17 places are represented (one
integer digit and 16 decimal places). Valid values are numbers between
-1.7976931348623157E+308 and -2.2250738585072014E-308 for the negative range
and between +2.2250738585072014E-308 and +1.7976931348623157E+308 for the
positive range, plus 0. Both validity intervals are extended to the value zero
by subnormal numbers according to IEEE-754. Not every sixteen-digit number can
be represented exactly by a binary floating point number.
Minimal reproducible example:
REPORT ztest.
DATA num TYPE int1.
num = 1000. " <=== run time error
The solution is to use a larger data type like int2 (up to 32767) or I (integer on 4 bytes):
REPORT ztest.
DATA num TYPE int2. " <=== larger type
num = 1000. " <=== no more error
NB: decfloat34 is the larger possible numeric data type, it can handle virtually any value.
This issue can occur in methods, Function modules, or in Report also.
The main reason behind this issue is the runtime configuration of attributes present in code.
For example,
DATA: num type int1 value 256.
This statement is syntactically fine but the value that is being assigned to variable num is more than the range of type INT1. So it will dump while activation only.
Solution:
DATA: num type int1 value {<=255}
Similarly, this error can occur in any case where the compile-time and runtime configuration conflict.
I encounter strange DB2 behaviour. An example will illustrate it :
SELECT CAST(11458.5648 AS DECIMAL(30,10)) / CAST(120.1 AS DECIMAL(30,10)), 11458.5648 / 120.1 FROM MYTABLE FETCH FIRST 1 ROW ONLY
returns :
1 | 2
---------------------------
95.4 | 95.4085328893
Of course, the good result is in column 2, but why DB2 does this awful error ?
If I cast to DECIMAL(20,10), the result is good too, but starting with DECIMAL(22,10), I loose 1 digit of precision in the result at each step ...
Any idea about it ?
thanks
You need to understand how decimal arithmetic is handled (for your platform & version of Db2)
For Db2 LUW v11.5
Two decimal operands If both operands are decimal, the operation is
performed in decimal. The result of any decimal arithmetic operation
is a decimal number with a precision and scale that are dependent on
the operation and the precision and scale of the operands. If the
operation is addition or subtraction and the operands do not have the
same scale, the operation is performed with a temporary copy of one of
the operands. The copy of the shorter operand is extended with
trailing zeros so that its fractional part has the same number of
digits as the longer operand.
The result of a decimal operation cannot have a precision greater than
31. The result of decimal addition, subtraction, and multiplication is derived from a temporary result which can have a precision greater
than 31. If the precision of the temporary result is not greater than
31, the final result is the same as the temporary result.
Decimal arithmetic in SQL Use the formulas shown here to calculate the
precision and scale of the result of decimal operations in SQL. The
formulas use the following symbols:
p Precision of the first operand.
s Scale of the first operand.
p' Precision of the second operand.
s' Scale of the second operand.
Assuming the default mode, by casting the operands to decimal(30,10) your results has
p = 31
s = 31-30+10-10 ==> 1
moral of the story, don't artificially increase the precision and scale of your operands.
When I calculate log(8) / log(2) I get 3 as one would expect:
?log(8)/log(2)
3
However, if I take the int of this calculation like this the result is 2 and thus wrong:
?int(log(8)/log(2))
2
How and why does this happen?
Likely because the actual number returned is of type double. Because floats and doubles cannot accurately represent most base 10 rational numbers the number returned is something like 2.99999999999. Then when you apply int() the .999999999 is truncated.
How floating-point number works: it dedicates a bit for the sign, a few bits to store an exponent, and the rest for the actual fraction. This leads to numbers being represented in a form similar to 1.45 * 10^4; except that instead of the base being 10, it's two.
I'm trying to divide one number by another and then immediately ceil() the result. These would normally be variables, but for simplicity let's stick with constants.
If I try any of the following, I get 3 when I want to get 4.
double num = ceil(25/8); // 3
float num = ceil(25/8); // 3
int num = ceil(25/8); // 3
I've read through a few threads on here (tried the nextafter() suggestion from this thread) as well as other sites and I don't understand what's going on. I've checked and my variables are the numbers I expect them to be and I've in fact tried the above, using constants, and am still getting unexpected results.
Thanks in advance for the help. I'm sure it's something simple that I'm missing but I'm at a loss at this point.
This is because you are doing integer arithmetic. The value is 3 before you are calling ceil, because 25 and 8 are both integers. 25/8 is calculated first using integer arithmetic, evaluating to 3.
Try:
double value = ceil(25.0/8);
This will ensure the compiler treats the constant 25.0 as a floating point number.
You can also use an explicit cast to achieve the same result:
double value = ceil(((double)25)/8);
This is because the expressions are evaluated before being passed as an argument to the ceil function. You need to cast one of them to a double first so the result will be a decimal that will be passed to ceil.
double num = ceil((double)25/8);