In Teradata when computing a percentage (see example below), I get 0 as an answer because Terada rounds results based on the number of decimal places the number have. to my knowledge I have found 2 possible solution to avoid the zero value return.
1- define the decimal number in table with a larger scale value. e.g decimal(18,4) instead of decimal(18,2)
2- cast the first arithmetic operation that occurs to a decimal number with a larger scale value. e.g. select (cast(2.0 as decimal(10,6)) / 10.0) * 100;
has anyone encounter this problem before and what is the solution you went with to solve the problem? thank you in advance.
these are the actual values and the correct answer should be 0.25
select ((28.97 + 28.97) / (11586.87 + 11586.87)) * 100
,(cast((28.97 + 28.97) as decimal(20,4)) / (11586.87 + 11586.87)) * 100
,((cast(28.97 as decimal(18,4)) + 28.97) / (11586.87 + 11586.87))
,(100 * (28.97 + 28.97) / (11586.87 + 11586.87))
,(28.97 + 28.97) , (11586.87 + 11586.87);
Related
I need to get the nearest airport in my database table from the current users position. I found this formula: https://de.scribd.com/presentation/2569355/Geo-Distance-Search-with-MySQL#page=7
So there are a few differences between the formula described in the link above and my current situation: The example was in MySQL, I'm using MS SQL (not a problem, I guess). lat and lon are considered to be database columns with numeric data type, but for some reason the database table was created with two corresponding columns of type varchar.
My problem is: When I want to use an ORDER BY clause, it throws Error converting data type nvarchar to numeric, without it, it works. I did some research on what rubbish was inserted as string and migrated it so that I just have some empty values.
I can't take all because I only need one. But if I do TOP 1 without ORDER BY I don't get any airport rather than the nearest airport. Does anyone know how to fix the query?
Thanks in advance!
SELECT TOP 1
temp.Distance
FROM (
SELECT
(
3956 * 2 * ASIN(
SQRT(
POWER(
SIN((53.6349994 - abs(CAST(latitude_deg AS numeric))) * pi() / 180 / 2), 2) + COS(53.6349994 * pi()/180) * COS(abs(CAST(latitude_deg AS numeric)) * pi()/180) * POWER(SIN((10.0117336 - CAST(longitude_deg AS numeric)) * pi()/180 / 2), 2) ))) AS Distance
FROM Airport_Airports
WHERE
isnumeric(longitude_deg) = 1 AND isnumeric(latitude_deg) = 1 AND
longitude_deg LIKE '%[^0-9.]%' AND latitude_deg LIKE '%[^0-9.]%'
) AS temp
WHERE
temp.Distance < 50000
Order BY
temp.Distance
First, this logic doesn't make sense:
WHERE isnumeric(longitude_deg) = 1 AND
isnumeric(latitude_deg) = 1 AND
longitude_deg LIKE '%[^0-9.]%' AND
latitude_deg LIKE '%[^0-9.]%'
The like is looking for non-numeric characters. I think you intend:
WHERE isnumeric(longitude_deg) = 1 AND
isnumeric(latitude_deg) = 1 AND
longitude_deg NOT LIKE '%[^0-9.]%' AND
latitude_deg NOT LIKE '%[^0-9.]%'
This ensures that the values are numeric.
The solution to your problem -- at least in SQL Server 2012+ -- is to use try_convert() or try_cast():
(3956 * 2 * ASIN(
SQRT(
POWER(
SIN((53.6349994 - abs(try_convert(numeric, latitude_deg))) * pi() / 180 / 2), 2) + COS(53.6349994 * pi()/180) * COS(abs(try_convert(numeric, latitude_deg)) * pi()/180) * POWER(SIN((10.0117336 - try_convert(numeric, longitude_deg)) * pi()/180 / 2), 2) ))) AS Distance
This will prevent any conversion errors.
You shouldn't use just numeric. Use either a floating point representation or something with decimal places, say numeric(20, 10).
The reason this occurs with the order by is because of the SQL optimizer. You clearly have some lat/long values that do not convert correctly to a numeric. SQL Server allows itself to re-arrange operations, so the conversion might take place before the filtering by the where clause. This is part of the overall query optimization.
The first line of code displays to 2 decimal points
The second line displays to 3 decimal points
SUM(CONVERT(DECIMAL(8,2),[can]/1.2)) + SUM(CONVERT(DECIMAL(8,2),[carton]/1.2)) AS [N],
(SUM(CONVERT(DECIMAL(8,2),[can]/1.2)) + SUM(CONVERT(DECIMAL(8,2),[carton]/1.2)))*0.2 AS [V]
Can someone please tell me how to get the second line to show 2 decimal points
If you want to control the display, the do the cast to decimal after the calculation:
CONVERT(DECIMAL(8, 2), SUM([can]/1.2)) + SUM([carton]/1.2)) AS [N],
You can do the conversion before, if you have a need to represent the some intermediate value as a decimal. Do note, though, that SQL Server has quite arcane rules for the precision and scale of intermediate and final results when doing decimal arithmetic.
Just cast the result to a 2 decimal place decimal:
select --just added to make the example runnable
SUM(CONVERT(DECIMAL(8,2),[can]/1.2)) + SUM(CONVERT(DECIMAL(8,2),[carton]/1.2)) AS [N],
CONVERT(DECIMAL(8,2), --add this line
(SUM(CONVERT(DECIMAL(8,2),[can]/1.2)) + SUM(CONVERT(DECIMAL(8,2),[carton]/1.2)))*0.2
) --and an additional close bracket
AS [V]
from (select 1.0 [can], 1.0 [carton] union all select 2.0, 3.0) x --just added to make the example runnable
The reason you're seeing 3 decimal places is because that's what the result of the calculation is; and when working with decimals SQL will output a decimal of the appropriate precision and scale to hold that result, unless you cast/convert it to something specific.
You convert to 2 decimals using CONVERT...
SELECT
SUM(CONVERT(decimal(8, 2), [can] / 1.2)) + SUM(CONVERT(decimal(8, 2), [carton] / 1.2)) AS [N],
CONVERT(decimal(8, 2), ((SUM(CONVERT(decimal(8, 2), [can] / 1.2)) + SUM(CONVERT(decimal(8, 2), [carton] / 1.2))) * 0.2)) AS [V]
I'm working in SQL Server 2008. I have an alias column that I'm creating via the following formula:
col1 / col2 * some_number
col1 and col2 are nvarchars in my table, but they're really integers. some_number is an integer. If I blindly do col1 / col2, I will get 0 for most. So, I need to cast them as decimals. I want there to be a maximum of 2 decimal places after the decimal point. Here is what I have currently:
CAST(col1 AS DECIMAL(10,2)) / CAST(col2 AS DECIMAL(10,2)) * 100
However, this returns far more decimal places than just 2. How do I fix my code to return just 2 decimal places?
According to the documents the scale of the result is given by:
s2 = max(6, s1 + p2 + 1)
where p2 represents the precision of the numerator and s2 represents scale of the denominator
So when dividing one DECIMAL(10, 2) by another you can substitute in values:
s2 = max(6, 2 + 10 + 1) = 13
Which is corroborated with a simple example:
SELECT CAST(1 AS DECIMAL(10,2)) / CAST(1 AS DECIMAL(10,2))
= 1.0000000000000 -- 13 decimal points
You need to use another cast on your result to reduce the scale:
SELECT CAST(CAST(col1 AS DECIMAL(10,2)) / CAST(col2 AS DECIMAL(10,2)) * 100 AS DECIMAL(10, 2))
I'd also suggest if the data type of you column is nvarchar, "but they're really integers" that you just bite the bullet and alter the column data type.
We have a fairly complicated SQL Server 2008 r2 sp2 query with this as one of the lines :-
SUM((t.Quantity * contract.ValueOfOnePoint) * ((
CASE contract.Style
WHEN 3
THEN 1 / (1.0 + ((100.0 - Val) / 100.0 * 90.0 / 365.0))
WHEN 2
THEN 1000 * (6.0 * (1.0 - (POWER((1.0 / (1.0 + ((100.0 - Val) / 200.0))), 20.0))) / ((100.0 - Val) / 200.0) + (100.0 * (POWER((1.0 / (1.0 + ((100.0 - Val) / 200.0))), 20.0))))
END
) - (
CASE contract.Style
WHEN 3
THEN 1.0 / (1.0 + ((100.0 - t.Price) / 100.0 * 90.0 / 365.0))
WHEN 2
THEN 1000 * (6.0 * (1.0 - (POWER((1.0 / (1.0 + ((100.0 - t.Price) / 200.0))), 20.0))) / ((100.00 - t.Price) / 200.00) + (100.0 * (POWER((1.0 / (1.0 + ((100.0 - t.Price) / 200.0))), 20.0))))
END
)
)) AS NativeAmount
I am testing this on a single row which has a style of 3 so only the first line in the case statement should have any affect yet leaving the "WHEN 2" clause in it reduces the accuracy of the formula.
Eg. if I remove both WHEN 2 conditions I get an answer such 123.45678 but with the WHEN 2 line left in I get 123.46. It seems to be rounding for some reason even though the second WHEN should never be in-play.
Any thoughts would be really appreciated - going mad!
Thanks.
James.
You need to combine two things. The return type of the case statement is the same for all the then and else clauses. This is a quote from the documentation:
[The case statement] returns the highest precedence type from the set of types in
result_expressions and the optional else_result_expression. For more
information, see Data Type Precedence (Transact-SQL).
So, the SQL Engine does care about all the clauses in the query (which is the answer to your question).
I don't fully understand what is happening in this case. When you call the power() function, the compiler has to decide on the precision of the numeric value, based on the constants and column types. Based on this SQL Fiddle, it chooses a precision of 38 and a scale of 1. However, simple arithmetic on the values produces a precision of 36 and a scale of 23. I'm not sure why, in the end, this results in rounding the value to two decimal places. Perhaps the logic for assigning types for the into clause doesn't quite match the logic for typing of expressions.
i am currently writing a scalar valued function and i'm having a few issue with the returned result.
i have narrowed the problem down to a calulation that convert the difference between two dates as a percentage/decimal. no matter what i try the return value is always a whole number
set #earnedpremium = (#premium * #pretripearnings) + ((#premium - (#premium * #pretripearnings)) * cast((datediff(day, #outdate, #experiencedate) / datediff(day, #outdate, #returndate))as decimal(5,2)))
the cast section needs to return the percentage, i know the rest is working fine through some elimination and testing.
can someone please help me figure out what im doing wrong??
It's because DATEDIFF returns an INTEGER and so you need to cast both parts of that operation to DECIMAL:
set #earnedpremium = (#premium * #pretripearnings) + ((#premium - (#premium * #pretripearnings))
* (CAST(datediff(day, #outdate, #experiencedate) AS DECIMAL(5,2)) /
CAST(datediff(day, #outdate, #returndate) AS DECIMAL(5,2)))