I am doing this small exercise.
declare #No decimal(38,5);
set #No=12345678910111213.14151;
select #No*1000/1000,#No/1000*1000,#No;
Results are:
12345678910111213.141510
12345678910111213.141000
12345678910111213.14151
Why are the results of first 2 selects different when mathematically it should be same?
it is not going to do algebra to convert 1000/1000 to 1. it is going to actually follow the order of operations and do each step.
#No*1000/1000
yields: #No*1000 = 12345678910111213141.51000
then /1000= 12345678910111213.141510
and
#No/1000*1000
yields: #No/1000 = 12345678910111.213141
then *1000= 12345678910111213.141000
by dividing first you lose decimal digits.
because of rounding, the second sql first divides by 1000 which is 12345678910111.21314151, but your decimal is only 38,5, so you lose the last three decimal points.
because when you divide first you get:
12345678910111.21314151
then only six decimal digits are left after point:
12345678910111.213141
then *1000
12345678910111213.141
because the intermediary type is the same as the argument's - in this case decimal(38,5). so dividing first gives you a loss of precision that's reflected in the truncated answer. multiplying by 1000 first doesn't give any loss of precision because that doesn't overload 38 digits.
It's probably because you lose part of data making division first. Notice that #No has 5-point decimal precision so when you divide this number by 1000 you suddenly need 8 digits for decimal part:
123.12345 / 1000 = 0.12312345
So the value has to be rounded (0.12312) and then this value is multiply by 1000 -> 123.12 (you lose 0.00345.
I think that's why the result is what it is...
The first does #No*1000 then divides it by 1000. The intermediates values are always able to represent all the decimal places. The second expression first divides by 1000, which throws away the last two decimal places, before multiplying back to the original value.
You can get around the problem by using CONVERT or CAST on the first value in your expression to increase the number of decimal places and avoid a loss of precision.
DECLARE #num decimal(38,5)
SET #num = 12345678910111213.14151
SELECT CAST(#num AS decimal(38,8)) / 1000 * 1000
Related
I'm writing a snowflake query that calculate 1/2940744 and get the result equals to 0
How to solve to get the actual calculation result?
From docs:
Division
When performing division:
The leading digits for the output is the sum of the leading digits of the numerator and the scale of the denominator.
Snowflake minimizes potential overflow in the output (due to chained division) and loss of scale by adding 6 digits to the scale of the numerator, up to a maximum threshold of 12 digits, unless the scale of the numerator is larger than 12, in which case the numerator scale is used as the output scale.
In other words, assuming a division operation with numerator L1.S1 and denominator L2.S2, the maximum number of digits in the output are calculated as follows:
Scale S = max(S1, min(S1 + 6, 12))
If the result of the division operation exceeds the output scale, Snowflake rounds the output (rather than truncating the output).
Returning to example:
SELECT 1/2940744;
-- 0
DESC RESULT LAST_QUERY_ID();
The value 0.00000034005 was rounded to 0. In order to change the behaviour one of the arguments could be explicitly casted:
SELECT 1::NUMBER(38,12)/2940744;
-- 0.00000034005
DESC RESULT LAST_QUERY_ID();
-- 1::NUMBER(38,12)/2940744 NUMBER(38,12)
Thanks for the answer above, I check this answer late and solve the question myself by converting the result to ::double -> 1/5000000::double
I have created a table where a column has the format NUMBER(2,3).
I try to insert the value 5.73 but it doesn't work.
The error is :
ORA-01438 - "value larger than specified precision allowed for this column"
Cause: When inserting or updating records, a numeric value was entered
that exceeded the precision defined for the column.*
I read the documentation but i don't understand the scale.
So, what is the format accepted value 0-99 with 3 values after the decimal point ?
Thanks.
You are misunderstanding precision and scale. You have a number with a precision of 2. That means that there are two significant digits. It has a scale of 3, which means that these are to the right of the decimal point.
So, your column can represent values between 0.000 and 0.099
What you want is NUMERIC(5, 3). "precision - scale" is the number of digits to the left of the decimal point.
This has come from here:
https://docs.oracle.com/cd/B28359_01/server.111/b28318/datatype.htm#CNCPT1832
Optionally, you can also specify a precision (total number of digits) and scale (number of >digits to the right of the decimal point):
column_name NUMBER (precision, scale)
So in your example you are allowed a total number of 2 digits ( and 3 digits to the right of the decimal point). Which doesn't work for 5.73, perhaps you need a type of number(3,2) which would allow 3 digits 2 of which can be right of the decimal point.
As an example I have two numbers (40.25 & 1.88001). In reality the two numbers could be any number with up to five decimal places. However, the second number will always be <= the first number.
I need to round 1.88001 up until it is evenly divisible into 40.25. I need to find a factor of 40.25 that is greater than and closest to 1.88001.
I need to maintain the decimal precision if necessary, meaning I can't just round up to the nearest integer unless a whole number is indeed the closest factor.
I have found other solutions/questions that are similar but have found none that are applicable and/or SQL related.
So far I have a brute force solution that is too slow to be of any real use:
Declare #SheetLength as money = 40.25
Declare #PartLength as money = 1.88001
WHILE #SheetLength % #PartLength > 0
BEGIN
#PartLength = #PartLength + .00001
END
Select #PartLength
There must be a faster/more efficient way to do this ...
What do you mean "round up"? The simplest solution is to take the ratio of the two numbers, take the floor, and divide that into the larger number:
select #SheetLength / floor(#SheetLength / #PartLength)
For your example, this gives 1.91666667.
When I calculate log(8) / log(2) I get 3 as one would expect:
?log(8)/log(2)
3
However, if I take the int of this calculation like this the result is 2 and thus wrong:
?int(log(8)/log(2))
2
How and why does this happen?
Likely because the actual number returned is of type double. Because floats and doubles cannot accurately represent most base 10 rational numbers the number returned is something like 2.99999999999. Then when you apply int() the .999999999 is truncated.
How floating-point number works: it dedicates a bit for the sign, a few bits to store an exponent, and the rest for the actual fraction. This leads to numbers being represented in a form similar to 1.45 * 10^4; except that instead of the base being 10, it's two.
I have a problem when i'm trying to calculate in a view a formula whose result is smaller than 1.
e.g. I have the next formula: Arenda*TotalArea/10000 as TotalArenda
If I have Arenda=10 and TotalArea=10 I get TotalArenda=0,00 when normally should be 0.01
Thanks
Make Arenda = 10.0 and TotalArea = 10.0 instead of 10 and 10. This will force SQL not to use integer math and you will get your needed accuracy.
In fact, the only way I can get 0.0 as the result is if the Arenda is 10 (integer) while at least one of TotalArea or 10000 contain a decimal and a trailing 0, and only if I override order of operations by grouping using parentheses such as
select 10.0* (10/10000) as blah
If all are integers you get 0. If all contain decimals you get 0.01. If I remove the parentheses, I get 0.01 if ANY of them are non-integer types.
If precision is highly important I would recommend you cast to decimals and not floats:
select CONVERT(decimal(10,2), Arenda) * CONVERT(decimal(10,2), TotalArea) / 10000.0
You are using colunns, so changing the type may not be feasible. SQL Server does integer division on integers (other databases behave differently). Try one of these:
cast(Arenda as float)*cast(TotalArea as float)/10000
or:
Arenda*TotalArea/10000.0