Annotate queryset with previous object in Django ORM - sql

Question

Example models:
class User(models.Model):
pass
class UserStatusChange(models.Model):
user = models.ForeignKey(User, related_name='status_changes')
status = models.CharField()
start_date = models.DateField()
I want to annotate UserStatusChanges queryset with end_date field, and end_date should be equal to start_date of next status change for the same user.
Eventually, I want to be able to do this:
qs = UserStatusChange.ojects.annotate(end_date=???)
qs = qs.filter(start_date__lte=some_date, end_date__gte=another_date)
Logically that annotation should be something like this:
qs.annotate(
end_date=qs.filter(
user=OuterRef('user'),
start_date__gt=OuterRef('start_date')
).order_by('start_date').first().start_date)
But it should be one DB query, if it is possible.
Solution:
subquery = UserStatusChange.objects.filter(user=OuterRef('user'),
start_date__gt=OuterRef('start_date')).order_by('start_date')
UserStatusChange.objects.annotate(end_date=Subquery(subquery.values('start_date')[:1]))
That works, thank to #hynekcer's answer. But with aggregate I got the error:
ValueError: This queryset contains a reference to an outer query and may only be used in a subquery.
UPD: in Django 2.0+ it can be solved with Lead Window function.
In SQL it will be something like this:
select
user_id, status_id, start_date,
LEAD(start_date, 1) over (partition by user_id order by start_date)
from user_status_change;

Answer 1

You can use Subquery() with OuterRef() in Django 1.11.
from django.db.models import Min, OuterRef, Subquery
from django.db.models.functions import Coalesce
default_end = now() # or the end of the recorded history
qs = (
UserStatusChanges.objects
.annotate(
end_date=Coalesce(
Subquery(
UserStatusChanges.objects
.filter(
user=OuterRef('user'),
start_date__gt=OuterRef('start_date')
)
.order_by()
.aggregate(Min('start_date'))
),
default_end
)
)
)
qs = qs.order_by('user', 'start_date')
# an optional filter
qs = qs.filter(start_date__lte=some_date, end_date__gte=another_date, user__in=[...])
It is compiled as one query when being executed, e.g. when combined with User filter by prefetch_related. If you want a meaningful end_date also for the last item then you can use Coalesce() with a default value equal to the current timestamp.