Complex GROUP BY with Django's ORM - sql

Question

I have a Django application that tracks electricity consumption and I'm having a hard time trying to come up with a way to use Django's ORM to fetch some information.
My specific use case is this: I have a set of electricity consumption readings, each with a datetime field, consumption and cost (and a few others but these are the relevant ones). I need to sum the consumption and cost values grouped by month, year, electricity meter and electricity price. In other words, I need to be able to get the total energy consumption value and corresponding cost for each month, of each year, for each price (easier to understand if you look at the table further down the post).
This is my ElectricityReading model and its parent Reading model (separated because we also have consumption readings for water and gas, which also derive from Reading):
from model_utils.models import TimeStampedModel
# Other imports here...
class Reading(TimeStampedModel):
meter = models.ForeignKey(Meter)
datetime = models.DateTimeField() # Terrible property name, I know :)
class Meta:
abstract = True
class ElectricityReading(Reading):
price = models.ForeignKey(ElectricityPrice)
consumption = models.DecimalField(max_digits=18, decimal_places=3,
null=True, blank=True, default=None)
cost = models.DecimalField(max_digits=18, decimal_places=3, null=True,
blank=True, default=None)
Right now I'm doing this with this raw SQL, which I build depending on a few parameters:
SELECT
(EXTRACT(YEAR FROM datetime)) AS reading_date_year,
(EXTRACT(MONTH FROM datetime)) AS reading_date_month,
SUM(consumption) as total_consumption,
SUM(cost) as total_cost,
COUNT(id) as num_readings,
price_id
FROM electricity_reading
WHERE meter_id IN (10)
AND datetime >= '2015-10-01 00:00'
AND datetime <= '2015-12-31 23:59'
GROUP BY reading_date_year, reading_date_month, price_id, meter_id
ORDER BY meter_id, reading_date_year, reading_date_month, price_id
This SQL query results in something like the following data (made up values and simplified column names for better formatting):
╔══════╦═══════╦═════════════╦══════╦══════════════╦═══════╗
║ year ║ month ║ consumption ║ cost ║ num_readings ║ price ║
╠══════╬═══════╬═════════════╬══════╬══════════════╬═══════╣
║ 2015 ║ 10 ║ 600 ║ 804 ║ 456 ║ 1 ║
║ 2015 ║ 10 ║ 728 ║ 471 ║ 1998 ║ 2 ║
║ 2015 ║ 10 ║ 848 ║ 792 ║ 1266 ║ 3 ║
║ 2015 ║ 10 ║ 256 ║ 705 ║ 744 ║ 5 ║
║ 2015 ║ 11 ║ 528 ║ 377 ║ 630 ║ 1 ║
║ 2015 ║ 11 ║ 016 ║ 687 ║ 1680 ║ 2 ║
║ 2015 ║ 11 ║ 240 ║ 826 ║ 1289 ║ 3 ║
║ 2015 ║ 11 ║ 736 ║ 522 ║ 720 ║ 5 ║
║ 2015 ║ 12 ║ 584 ║ 627 ║ 608 ║ 1 ║
║ 2015 ║ 12 ║ 776 ║ 078 ║ 1627 ║ 2 ║
║ 2015 ║ 12 ║ 600 ║ 401 ║ 1410 ║ 3 ║
║ 2015 ║ 12 ║ 864 ║ 842 ║ 744 ║ 5 ║
╚══════╩═══════╩═════════════╩══════╩══════════════╩═══════╝
Using Django's ORM, I think the code I need is something along the lines of the following:
objs = ElectricityReading.objects\
.filter(
meter=10,
datetime__gte='2015-05-01 00:00',
datetime__lte='2015-08-31 23:59'
).only('price_id')\
.annotate(reading_date_year=YearTransform('datetime'))\
.annotate(reading_date_month=MonthTransform('datetime'))\
.annotate(total_consumption=Sum('consumption'))\
.annotate(total_cost=Sum('cost'))\
.annotate(num_readings=Count('id'))\
.order_by('meter_id', 'reading_date_year', 'reading_date_month', 'price_id')
But the SQL it generates is not what I need:
SELECT
id,
price_id,
EXTRACT('year' FROM datetime AT TIME ZONE 'Europe/Lisbon') AS reading_date_year,
EXTRACT('month' FROM datetime AT TIME ZONE 'Europe/Lisbon') AS reading_date_month,
SUM(consumption) AS total_consumption, SUM(cost) AS total_cost,
COUNT(id) AS num_readings
FROM geratriz_electricityreading
WHERE (
datetime >= '2015-05-01 00:00:00+01:00'
AND datetime <= '2015-08-31 23:59:00+01:00'
AND meter_id = 10)
GROUP BY
id,
EXTRACT('year' FROM datetime AT TIME ZONE 'Europe/Lisbon'),
EXTRACT('month' FROM datetime AT TIME ZONE 'Europe/Lisbon')
ORDER BY meter_id ASC, reading_date_year ASC, reading_date_month ASC, price_id ASC
This results in a lot more rows being returned from the database due to not being grouped as I need them to be.
The part of the SQL query I can't seem to replicate with Django's ORM is the GROUP BY clause at the end. Django insists on grouping by ID and I can't seem to find a way to make it group by meter_id and price_id.
Given how much time I spent on this already, I'm inclined to say that what I am trying to accomplish simply isn't possible with Django's ORM but I would love that someone would tell me I am missing something.

Answer 1

Try using values()
objs = ElectricityReading.objects\
.filter(
meter=10,
datetime__gte='2015-05-01 00:00',
datetime__lte='2015-08-31 23:59'
.values('price_id')\
.annotate(reading_date_year=YearTransform('datetime'))\
.annotate(reading_date_month=MonthTransform('datetime'))\
.annotate(total_consumption=Sum('consumption'))\
.annotate(total_cost=Sum('cost'))\
.annotate(num_readings=Count('id'))\
.order_by('meter_id', 'reading_date_year', 'reading_date_month', 'price_id')
This should group the results on price_id. If you were displaying several meters at once instead of meter=10, then you could do values('price_id', 'meter') and it would group on both fields.