I'm writing a snowflake query that calculate 1/2940744 and get the result equals to 0
How to solve to get the actual calculation result?
From docs:
Division
When performing division:
The leading digits for the output is the sum of the leading digits of the numerator and the scale of the denominator.
Snowflake minimizes potential overflow in the output (due to chained division) and loss of scale by adding 6 digits to the scale of the numerator, up to a maximum threshold of 12 digits, unless the scale of the numerator is larger than 12, in which case the numerator scale is used as the output scale.
In other words, assuming a division operation with numerator L1.S1 and denominator L2.S2, the maximum number of digits in the output are calculated as follows:
Scale S = max(S1, min(S1 + 6, 12))
If the result of the division operation exceeds the output scale, Snowflake rounds the output (rather than truncating the output).
Returning to example:
SELECT 1/2940744;
-- 0
DESC RESULT LAST_QUERY_ID();
The value 0.00000034005 was rounded to 0. In order to change the behaviour one of the arguments could be explicitly casted:
SELECT 1::NUMBER(38,12)/2940744;
-- 0.00000034005
DESC RESULT LAST_QUERY_ID();
-- 1::NUMBER(38,12)/2940744 NUMBER(38,12)
Thanks for the answer above, I check this answer late and solve the question myself by converting the result to ::double -> 1/5000000::double
Related
I encounter strange DB2 behaviour. An example will illustrate it :
SELECT CAST(11458.5648 AS DECIMAL(30,10)) / CAST(120.1 AS DECIMAL(30,10)), 11458.5648 / 120.1 FROM MYTABLE FETCH FIRST 1 ROW ONLY
returns :
1 | 2
---------------------------
95.4 | 95.4085328893
Of course, the good result is in column 2, but why DB2 does this awful error ?
If I cast to DECIMAL(20,10), the result is good too, but starting with DECIMAL(22,10), I loose 1 digit of precision in the result at each step ...
Any idea about it ?
thanks
You need to understand how decimal arithmetic is handled (for your platform & version of Db2)
For Db2 LUW v11.5
Two decimal operands If both operands are decimal, the operation is
performed in decimal. The result of any decimal arithmetic operation
is a decimal number with a precision and scale that are dependent on
the operation and the precision and scale of the operands. If the
operation is addition or subtraction and the operands do not have the
same scale, the operation is performed with a temporary copy of one of
the operands. The copy of the shorter operand is extended with
trailing zeros so that its fractional part has the same number of
digits as the longer operand.
The result of a decimal operation cannot have a precision greater than
31. The result of decimal addition, subtraction, and multiplication is derived from a temporary result which can have a precision greater
than 31. If the precision of the temporary result is not greater than
31, the final result is the same as the temporary result.
Decimal arithmetic in SQL Use the formulas shown here to calculate the
precision and scale of the result of decimal operations in SQL. The
formulas use the following symbols:
p Precision of the first operand.
s Scale of the first operand.
p' Precision of the second operand.
s' Scale of the second operand.
Assuming the default mode, by casting the operands to decimal(30,10) your results has
p = 31
s = 31-30+10-10 ==> 1
moral of the story, don't artificially increase the precision and scale of your operands.
I am allocating a single unit across multiple rows using a calculation and storing the results into a table. I am then sum() the allocations and the sums are resulting in numbers that are not whole numbers. What is going on is that some of the allocations are ending up as numbers with repeating decimals, and then the sum of those not adding back up to the whole number (ala 1/3 + 1/3 + 1/3 != 1).
I have tried casting the numbers into different formats, however, Athena keep rounding the decimals at some arbitrary precision resulting in the problem.
I would like the sum of the allocations to equal the sum of the original units.
My Database is AWS Athena which I understand to use the Presto SQL language.
Example of my allocation:
case
when count_of_visits = 1 then 1
when count_of_visits = 2 then .5
when count_of_visits >= 3 then
case
when visit_seq_number = min_visit_seq_number then .4
when visit_seq_number = max_visit_seq_number then .4
else .2 / (count_of_visits - 2 )
end
else 0
end as u_shp_alloc_leads
In this allocation, the first and last visits get 40% of the allocation and all visits in between split 20%
A unit that is being allocated to 29 visits ends up dividing the 20% by 27 which equals 0.00740Repeating. The table is storing 0.007407407407407408 which when I go to sum the numbers the result is 1.0000000000000004 I would like the result to be 1
This is a limitation of databases or computers in general. When you work with fractions like that, some sort of rounding will always take place.
I would apply a reasonable degree of rounding to the x-th decimal on the sums you retrieve from your table, that will just cut off these residual decimals at the end.
If that's not sufficient for you, something you can do to at least theoretically have full precision is to store numerator and denominator separately in two columns. When computing sum( numerator_column/denominator_column ) you will see the same rounding effects, so summing up the numbers would be something a little more complicated like this:
SELECT sum(numerator_sum/denominator)
FROM (
SELECT
denominator,
sum(numerator) as numerator_sum
FROM your_allocation_table
GROUP BY denominator
)
I am new to Redshift, trying to do multiplication in my SQL as mentioned below
Case When COALESCE(height * width * length * weight, 0) = 0 then ...
getting below error message,
Invalid operation: Resulting scale overflows maximum precision
Details:
Can someone please guid me.
As Redshift document:
precision
The total number of significant digits in the whole value:
the number of digits on both sides of the decimal point. For example,
the number 48.2891 has a precision of 6 and a scale of 4. The default
precision, if not specified, is 18. The maximum precision is 38.
So I think it depends on what's current precision of your data type. One way to work around is casting:
Case When COALESCE(height::decimal(8,3) * width::decimal(8,3) * length::decimal(8,3) * weight::decimal(8,3), 0) = 0 then ...
I have a problem when i'm trying to calculate in a view a formula whose result is smaller than 1.
e.g. I have the next formula: Arenda*TotalArea/10000 as TotalArenda
If I have Arenda=10 and TotalArea=10 I get TotalArenda=0,00 when normally should be 0.01
Thanks
Make Arenda = 10.0 and TotalArea = 10.0 instead of 10 and 10. This will force SQL not to use integer math and you will get your needed accuracy.
In fact, the only way I can get 0.0 as the result is if the Arenda is 10 (integer) while at least one of TotalArea or 10000 contain a decimal and a trailing 0, and only if I override order of operations by grouping using parentheses such as
select 10.0* (10/10000) as blah
If all are integers you get 0. If all contain decimals you get 0.01. If I remove the parentheses, I get 0.01 if ANY of them are non-integer types.
If precision is highly important I would recommend you cast to decimals and not floats:
select CONVERT(decimal(10,2), Arenda) * CONVERT(decimal(10,2), TotalArea) / 10000.0
You are using colunns, so changing the type may not be feasible. SQL Server does integer division on integers (other databases behave differently). Try one of these:
cast(Arenda as float)*cast(TotalArea as float)/10000
or:
Arenda*TotalArea/10000.0
I am doing this small exercise.
declare #No decimal(38,5);
set #No=12345678910111213.14151;
select #No*1000/1000,#No/1000*1000,#No;
Results are:
12345678910111213.141510
12345678910111213.141000
12345678910111213.14151
Why are the results of first 2 selects different when mathematically it should be same?
it is not going to do algebra to convert 1000/1000 to 1. it is going to actually follow the order of operations and do each step.
#No*1000/1000
yields: #No*1000 = 12345678910111213141.51000
then /1000= 12345678910111213.141510
and
#No/1000*1000
yields: #No/1000 = 12345678910111.213141
then *1000= 12345678910111213.141000
by dividing first you lose decimal digits.
because of rounding, the second sql first divides by 1000 which is 12345678910111.21314151, but your decimal is only 38,5, so you lose the last three decimal points.
because when you divide first you get:
12345678910111.21314151
then only six decimal digits are left after point:
12345678910111.213141
then *1000
12345678910111213.141
because the intermediary type is the same as the argument's - in this case decimal(38,5). so dividing first gives you a loss of precision that's reflected in the truncated answer. multiplying by 1000 first doesn't give any loss of precision because that doesn't overload 38 digits.
It's probably because you lose part of data making division first. Notice that #No has 5-point decimal precision so when you divide this number by 1000 you suddenly need 8 digits for decimal part:
123.12345 / 1000 = 0.12312345
So the value has to be rounded (0.12312) and then this value is multiply by 1000 -> 123.12 (you lose 0.00345.
I think that's why the result is what it is...
The first does #No*1000 then divides it by 1000. The intermediates values are always able to represent all the decimal places. The second expression first divides by 1000, which throws away the last two decimal places, before multiplying back to the original value.
You can get around the problem by using CONVERT or CAST on the first value in your expression to increase the number of decimal places and avoid a loss of precision.
DECLARE #num decimal(38,5)
SET #num = 12345678910111213.14151
SELECT CAST(#num AS decimal(38,8)) / 1000 * 1000