Optimize a PowerBI data load containing a Nearest Neighbor calculation - sql

Question

I have a PowerBI report that shows metrics and visuals for a large amount of quote data extracted by an API, roughly 400k records a week. These quotes only contain latitude and longitude points for location, but shareholders need to slice views by our service areas. We have a fact table of areas with IDs and geography polygons that I am able to reference.
Currently, the report uses a gnarly custom SQL query that pulls this data from the transactional database, transforms it, and finds the nearest area through a cross apply method.
Here's an example of the code:
-- step 1 : get quotes from the first table
SELECT Col1, Col2...
INTO #AllQuotes
FROM Quotes1
LEFT JOIN (FactTables)
INNER JOIN([filtering self join])
WHERE expression
-- Step 2 : insert quotes from a separate table into our first temp table to get a table with all quote data
INSERT INTO #AllQuotes
SELECT Col1, Col2
FROM Quotes2
LEFT JOIN(Fact Tables)
INNER JOIN([filtering self join])
WHERE expression
-- Step 3 : Use CROSS APPLY to check the distance of every quote from every area, only selecting the shortest distance
SELECT *
FROM (SELECT *
FROM #AllQuotes as t
CROSS APPLY (SELECT TOP 1 a.AreaName,
a.AreaPoly.STDistance(geography::STGeomFromText('POINT('+ cast(t.PickLongitudeTemp as VARCHAR(20)) +' '+ cast(t.PickLatitudeTemp as VARCHAR(20)) +')', 4326).MakeValid()) AS 'DistanceToZone'
FROM Area as a
WHERE (a.AreaPoly.STIsValid() = 1)
AND (a.AreaPoly.STDistance(geography::STGeomFromText('POINT('+ cast(t.PickLongitudeTemp as VARCHAR(20)) +' '+ cast(t.PickLatitudeTemp as VARCHAR(20)) +')', 4326).MakeValid()) IS NOT NULL)
ORDER BY a.AreaPoly.STDistance(geography::STGeomFromText('POINT('+ cast(t.PickLongitudeTemp as VARCHAR(20)) +' '+ cast(t.PickLatitudeTemp as VARCHAR(20)) +')', 4326).MakeValid()) ASC) AS t2 ) AS llz;
This is obviously very computationally expensive and is making the PowerBI mashup engine work in overdrive. We are starting to have issues with CPU load on our database due to poor data load optimization. PowerBI rebuilds its data model every refresh and its query engine is not the strongest at using complex queries. Compounding this with the large amount of data, it quickly becomes a real issue with our stability.
Our database doesn't have a schema that is conducive to making efficient analytics queries, there is no transformation happening as it's loaded, and a process to hit a maps API to associate addresses with lat/longs. In order to produce reports with any value, I need to perform a lot of transformations within the query or within the loading process. This isn't the best thing to do, I know, but its what I got working and that could provide value.
I decided to try to move the query into something server side so that PowerBI only needed to load an already transformed and prepped dataset. With views I was able to get a dataset of all of my quotes and their lat/longs.
Now how would I go about running step 3? I have a few ideas:
Use a nested view
Refactor every temp table into a monolith of CTEs that then get transformed by a final view
Research a new method for solving a Lat/Long to Polygon matching problem.
I would like to have a final table that PowerBI can import with a simple SELECT * FROM #AllQuotes so that the mashup engine has to do less work constructing the data model. This would also allow me to implement incremental refresh and be able to only import a day's worth of data as time goes on rather than the full dataset.
Any solutions or ideas on how to match Lat/Long points to a list of geography Polygons in a PBI friendly way would be greatly appreciated.

Answer 1

Can't say I'm a spatial expert, but I don't think you are really using your index. STDistance has to run against every combo of quote/area then sort to find the smallest distance. So you need to reduce the number of areas each quote is compared against
If you review your data, I'd guess you find something like 30% of quotes are within 5,000 meters of your quote location. And 80% are within 10,000 meters.
With that in mind, I think we can add some queries to find those close matches first. This should be able use you spatial indexes efficiently as it can first filter down to only close matches, reducing the number of times you have to calculate the distance of a quote to each area.
Conceptual Code Approach: First Find Quick Matches within Predefined Distance(s)
/*First identify matches with in small distance like 5,000 meters*/
UPDATE #Quote
SET NearestAreaID = C.AreaID
FROM #Quote AS A
CROSS APPLY (SELECT QuoteGeogPoint = geography::Point(A.PickLatitudeTemp, A.PickLongitudeTemp, 4326)) AS B
CROSS APPLY ( SELECT TOP(1) B.AreaID
FROM Area AS DTA
/*STBuffer creates circle of 5000 meters around Quote location
STIntersects matches only on data that intersects with that point*/
WHERE B.QuoteGeogPoint.STBuffer(5000).STIntersects(DTA.AreaPoly)
ORDER BY DTA.AreaPoly.STDistance(Q.QuoteGeogPoint)
) AS C
/*Could run above query again for a medium distance say 10,000 meters*/
UPDATE #Quote
SET NearestAreaID = C.AreaID
FROM #Quote AS A
CROSS APPLY (SELECT QuoteGeogPoint = geography::Point(A.PickLatitudeTemp, A.PickLongitudeTemp, 4326)) AS B
CROSS APPLY ( SELECT TOP(1) B.AreaID
FROM Area AS DTA
WHERE B.QuoteGeogPoint.STBuffer(10000).STIntersects(DTA.AreaPoly)
ORDER BY DTA.AreaPoly.STDistance(Q.QuoteGeogPoint)
) AS C
WHERE A.NearestAreaID IS NULL /*No matches found*/
Match Quotes Regardless of Area Distance
Once you've found the easy matches, use this script (your current step 3) to clean up any stragglers
/*Find matches for any didn't have a match in defined distances*/
UPDATE #Quote
SET NearestAreaID = C.AreaID
FROM #Quote AS A
CROSS APPLY (SELECT QuoteGeogPoint = geography::Point(A.PickLatitudeTemp, A.PickLongitudeTemp, 4326)) AS B
CROSS APPLY ( SELECT TOP(1) B.AreaID
FROM Area AS DTA
ORDER BY DTA.AreaPoly.STDistance(Q.QuoteGeogPoint)
) AS C
WHERE NearestAreaID IS NULL /*No matches already found*/