The above query is giving 1.0 as ouput in MS Server. But it gives 1.4 for power(2.0, 1.0/2). I really appreciate it if someone explains the reason for it.
Thanks in advance!
1/2 uses integer division, which becomes zero. Two to the zeroth power is one.
Because 1 and 2 are the integer, then result converted to the integer. Like as CAST(0.5 AS INT) = 0
If least one of them had a decimal like "1.0/2" or "1/2.0" or "1.0/2.0", then result converted to decimal and result would be 0.5.
Related
FormatNumber(totalvat, 2)
above is my code to round off the double. but instead of 223.66 result. it gives me 223.67 result.
in calculator the result is 223.6607142857143
can someone help me with this ? thanks!
Try this approach:
dim s as decimal = 223.6607142857143
console.WriteLine(s.ToString("###.##"))
I have a problem when i'm trying to calculate in a view a formula whose result is smaller than 1.
e.g. I have the next formula: Arenda*TotalArea/10000 as TotalArenda
If I have Arenda=10 and TotalArea=10 I get TotalArenda=0,00 when normally should be 0.01
Thanks
Make Arenda = 10.0 and TotalArea = 10.0 instead of 10 and 10. This will force SQL not to use integer math and you will get your needed accuracy.
In fact, the only way I can get 0.0 as the result is if the Arenda is 10 (integer) while at least one of TotalArea or 10000 contain a decimal and a trailing 0, and only if I override order of operations by grouping using parentheses such as
select 10.0* (10/10000) as blah
If all are integers you get 0. If all contain decimals you get 0.01. If I remove the parentheses, I get 0.01 if ANY of them are non-integer types.
If precision is highly important I would recommend you cast to decimals and not floats:
select CONVERT(decimal(10,2), Arenda) * CONVERT(decimal(10,2), TotalArea) / 10000.0
You are using colunns, so changing the type may not be feasible. SQL Server does integer division on integers (other databases behave differently). Try one of these:
cast(Arenda as float)*cast(TotalArea as float)/10000
or:
Arenda*TotalArea/10000.0
I'm trying to divide one number by another and then immediately ceil() the result. These would normally be variables, but for simplicity let's stick with constants.
If I try any of the following, I get 3 when I want to get 4.
double num = ceil(25/8); // 3
float num = ceil(25/8); // 3
int num = ceil(25/8); // 3
I've read through a few threads on here (tried the nextafter() suggestion from this thread) as well as other sites and I don't understand what's going on. I've checked and my variables are the numbers I expect them to be and I've in fact tried the above, using constants, and am still getting unexpected results.
Thanks in advance for the help. I'm sure it's something simple that I'm missing but I'm at a loss at this point.
This is because you are doing integer arithmetic. The value is 3 before you are calling ceil, because 25 and 8 are both integers. 25/8 is calculated first using integer arithmetic, evaluating to 3.
Try:
double value = ceil(25.0/8);
This will ensure the compiler treats the constant 25.0 as a floating point number.
You can also use an explicit cast to achieve the same result:
double value = ceil(((double)25)/8);
This is because the expressions are evaluated before being passed as an argument to the ceil function. You need to cast one of them to a double first so the result will be a decimal that will be passed to ceil.
double num = ceil((double)25/8);
I have code like this:
Dim minutes As Integer = (55 / 60)
I want this to return 0 to me, integer 0 (no decimals), but VB.NET rounds this to 1.
How do I accomplish this?
OK for those with the same problem, try dividing with \ (yeah, it's not escape character in VB.NET).
For full decimal division you have to use another operator.
Try:
\
Dim minutes As Integer = Math.Round(55 / 60)
There are overloads that allow rounding to fractional values and specifying which way to round if the value is exactly between two integers.
So round down? Use:
?Math.Truncate(55/60)
0.0
I am doing this small exercise.
declare #No decimal(38,5);
set #No=12345678910111213.14151;
select #No*1000/1000,#No/1000*1000,#No;
Results are:
12345678910111213.141510
12345678910111213.141000
12345678910111213.14151
Why are the results of first 2 selects different when mathematically it should be same?
it is not going to do algebra to convert 1000/1000 to 1. it is going to actually follow the order of operations and do each step.
#No*1000/1000
yields: #No*1000 = 12345678910111213141.51000
then /1000= 12345678910111213.141510
and
#No/1000*1000
yields: #No/1000 = 12345678910111.213141
then *1000= 12345678910111213.141000
by dividing first you lose decimal digits.
because of rounding, the second sql first divides by 1000 which is 12345678910111.21314151, but your decimal is only 38,5, so you lose the last three decimal points.
because when you divide first you get:
12345678910111.21314151
then only six decimal digits are left after point:
12345678910111.213141
then *1000
12345678910111213.141
because the intermediary type is the same as the argument's - in this case decimal(38,5). so dividing first gives you a loss of precision that's reflected in the truncated answer. multiplying by 1000 first doesn't give any loss of precision because that doesn't overload 38 digits.
It's probably because you lose part of data making division first. Notice that #No has 5-point decimal precision so when you divide this number by 1000 you suddenly need 8 digits for decimal part:
123.12345 / 1000 = 0.12312345
So the value has to be rounded (0.12312) and then this value is multiply by 1000 -> 123.12 (you lose 0.00345.
I think that's why the result is what it is...
The first does #No*1000 then divides it by 1000. The intermediates values are always able to represent all the decimal places. The second expression first divides by 1000, which throws away the last two decimal places, before multiplying back to the original value.
You can get around the problem by using CONVERT or CAST on the first value in your expression to increase the number of decimal places and avoid a loss of precision.
DECLARE #num decimal(38,5)
SET #num = 12345678910111213.14151
SELECT CAST(#num AS decimal(38,8)) / 1000 * 1000