Efficient (linear time) nested queries in SQL - sql

Question

From this table:
events
id
event_date
event_score
12
2020-04-10
13
2020-04-11
13
2020-04-14
8
13
2020-04-13
6
12
2020-04-15
14
2020-04-16
14
2020-04-17
14
2020-04-18
11
14
2020-04-19
14
2020-04-20
14
2020-04-22
12
2020-04-25
14
2020-04-30
I'm trying to get this result
results
id
first_score
last_score
12
13
6
8
14
11
11
One way to do that is through this query:
SELECT
DISTINCT id,
(
SELECT event_score
FROM events AS subquery
WHERE final_table.id=subquery.id
AND event_score IS NOT NULL
ORDER BY event_date
LIMIT 1
) AS `first score`,
(
SELECT event_score
FROM events AS subquery
WHERE final_table.id=subquery.id
AND event_score IS NOT NULL
ORDER BY event_date DESC
LIMIT 1
) AS `last score`
FROM sensors.events as final_table
but I suspect this takes quadratic time O(n*n) to compute. I know it can be done in linear time O(n) with Python but does anyone know how to do it in linear time with SQL?
The table is in MariaDB/MySQL

Answer 1

If you are running MariaDB 10.2.2 or higher, you could address this as a gaps-and-islands problem. The idea is to count how many non-null values appear on the preceding and following rows. We can then filter on the first non-null value in both directions, using conditional aggregation:
select id,
max(case when grp_asc = 1 then event_score end) as first_score,
max(case when grp_desc = 1 then event_score end) as last_score
from (
select e.*,
count(event_score) over(partition by id order by event_score ) as grp_asc,
count(event_score) over(partition by id order by event_score desc) as grp_desc
from events e
) e
group by id
order by id
I cannot assess the time complexity of this algorithm, but I would suspect that this should run faster than your original query, that requires executing two subqueries per distinct id.
Demo on DB Fiddle:
id | first_score | last_score
-: | ----------: | ---------:
12 | null | null
13 | 6 | 8
14 | 11 | 11

Answer 2

With a an index on (id, event_date, event_sore), then this should be quite fast:
SELECT id,
(SELECT event_score
FROM events AS subquery
WHERE final_table.id = subquery.id AND event_score IS NOT NULL
ORDER BY event_date
LIMIT 1
) AS `first score`,
(SELECT event_score
FROM events AS subquery
WHERE final_table.id=subquery.id AND event_score IS NOT NULL
ORDER BY event_date DESC
LIMIT 1
) AS `last score`
FROM (SELECT DISTINCT e.id
FROM sensors.events e
) as final_table;
Note that this moves the SELECT DISTINCT to a subquery. This is to ensure that MariaDB does not actually use a "distinct" algorithm for the SELECT DISTINCT -- the other columns would probably cause that to happen.
However, this is O(n log n) because the subqueries need to sort a small amount of data for each id -- as well as using an index to get to the right place.
I cannot think of a way to do this O(n) in SQL. I'm pretty sure the following constructs are all O(n log n):
Using an index for each row.
Sorting any portion of the data.
Using any window function with an order by -- although this might be true if there is just the right index.
But, SQL queries are still fast, particularly with indexes.