random floating point error with subquery order - sql

Question

I am running sql server 2008 database, i am using the following query in a web app, but for the point of debugging the error i am directly running the query in management studio.
I am getting the following error - An invalid floating point operation occurred. when running this query.
select p.Id as Id, p.CatId as CatId, p.MetaName as MetaName ,p.Active as Active,p.HasChildren as HasChildren ,p.Mlevel as Mlevel ,p.ParentId as ParentId ,p.Type as Type, p.VOrder as VOrder, p.UrlOrder as UrlOrder, Count('*') as VCount
from MetaDataValues as m
left join MetaData as p on m.MetaDataId = p.Id
left join Adverts as a on m.AdvertId = a.Id
where a.Status = 1
and a.ExpDate > current_timestamp and
m.AdvertId in
(select m2.AdvertId from MetaDataValues as m2 left join MetaData as p2 on m2.MetaDataId = p2.Id where p2.MetaName = 'meta1'
and m.AdvertId in (select m3.AdvertId from MetaDataValues as m3 left join MetaData as p3 on m3.MetaDataId = p3.Id where p3.MetaName = 'meta2'
and m.AdvertId in (select m4.AdvertId from MetaDataValues as m4 left join MetaData as p4 on m4.MetaDataId = p4.Id where p4.MetaName = 'meta3'
and m.AdvertId in (select ad9.Id from Adverts as ad9 where dbo.GetDist(ad9.X,ad9.Y,ad9.Z,52.9131514,-2.9313405) < 969))))
group by p.Id, p.CatId, p.MetaName,p.Active,p.HasChildren,p.Mlevel,p.ParentId,p.Type, p.VOrder, p.UrlOrder
To explain its the GetDist function that is causing the problem, if i move this up in the subqueries to the top level the query runs fine?? This isnt an ideal as the code that builds this query is coded in a certain way and i dont want to alter it. so here is the query that works, exactly the same but a different order!
select p.Id as Id, p.CatId as CatId, p.MetaName as MetaName ,p.Active as Active,p.HasChildren as HasChildren ,p.Mlevel as Mlevel ,p.ParentId as ParentId ,p.Type as Type, p.VOrder as VOrder, p.UrlOrder as UrlOrder, Count('*') as VCount
from MetaDataValues as m
left join MetaData as p on m.MetaDataId = p.Id
left join Adverts as a on m.AdvertId = a.Id
where a.Status = 1
and a.ExpDate > current_timestamp and
m.AdvertId in
(select m2.AdvertId from MetaDataValues as m2 left join MetaData as p2 on m2.MetaDataId = p2.Id where p2.MetaName = 'meta1'
and m.AdvertId in (select ad9.Id from Adverts as ad9 where dbo.GetDist(ad9.X,ad9.Y,ad9.Z,52.9131514,-2.9313405) < 969)
and m.AdvertId in (select m3.AdvertId from MetaDataValues as m3 left join MetaData as p3 on m3.MetaDataId = p3.Id where p3.MetaName = 'meta2'
and m.AdvertId in (select m4.AdvertId from MetaDataValues as m4 left join MetaData as p4 on m4.MetaDataId = p4.Id where p4.MetaName = 'meta3' )))
group by p.Id, p.CatId, p.MetaName,p.Active,p.HasChildren,p.Mlevel,p.ParentId,p.Type, p.VOrder, p.UrlOrder
GetDist code
USE [MVC]
GO
/****** Object: UserDefinedFunction [dbo].[GetDist] Script Date: 02/20/2013 17:05:00 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER FUNCTION [dbo].[GetDist]
(
#xaxis float,
#yaxis float,
#zaxis float,
#CenterLat float,
#CenterLon float
)
RETURNS float
AS
BEGIN
declare #CntXAxis float
declare #CntYAxis float
declare #CntZAxis float
declare #EarthRadius float
set #EarthRadius = 3961
set #CntXAxis = cos(radians(#CenterLat)) * cos(radians(#CenterLon))
set #CntYAxis = cos(radians(#CenterLat)) * sin(radians(#CenterLon))
set #CntZAxis = sin(radians(#CenterLat))
return (#EarthRadius * acos( #XAxis*#CntXAxis + #YAxis*#CntYAxis + #ZAxis*#CntZAxis))
END

Answer 1

It looks like GetDist may be calculating the distance between a pair of latitude/longitude values. I have a lot of experience with this. Most GetDist functions like this use the Arc Cosine function "ACos". The parameter for this function is limited to the range -1 to 1. If you try to pass a value outside this range, you will get a domain error in SQL Server. If your GetDist function uses a CLR function, the error would be within your .net code and would have a slightly different message.
When dealing with floats, you have to be aware of weird rounding issues. For example, if your calculations would return a value of 1.00000000000001, and you pass that in to the ACos function, you will get an error.
There's a lot of speculation here, and I could be completely off base, but please consider this and spend a couple minutes doing some research.
Based on your GetDist function posted above, I would suggest a relatively minor change:
ALTER FUNCTION [dbo].[GetDist]
(
#xaxis float,
#yaxis float,
#zaxis float,
#CenterLat float,
#CenterLon float
)
RETURNS float
AS
BEGIN
declare #CntXAxis float
declare #CntYAxis float
declare #CntZAxis float
declare #EarthRadius float
declare #Temp float
set #EarthRadius = 3961
set #CntXAxis = cos(radians(#CenterLat)) * cos(radians(#CenterLon))
set #CntYAxis = cos(radians(#CenterLat)) * sin(radians(#CenterLon))
set #CntZAxis = sin(radians(#CenterLat))
Set #Temp = #XAxis*#CntXAxis + #YAxis*#CntYAxis + #ZAxis*#CntZAxis
If #Temp > 1
Set #Temp = 1
Else If #Temp < -1
Set #Temp = -1
return (#EarthRadius * acos(#Temp))
END
Even if this doesn't solve your original problem, it will at least protect you from weird float/precision problems.

Answer 2

Is the problem an invalid operation or something more like "Error converting data type varchar to numeric"? This is a fairly common problem, when numeric data is being stored as a string. It works when the string looks right, but then fails at other times.
Are all the arguments to getDist() of the correct type? Is the return value a number?
I would guess that the filtering at the higher level filters out the bad values that are causing the problem.