SQL GROUP BY When Many Tables and Fields are Queried - sql

Question

I've never understood the the GROUP BY clause because most of the examples that google provides are very simple. I have a real life example of 982 columns pulling from 6 tables and need to select MAX(iVersion) which requires GROUP BY. A duplicate record identifier can be entered into SQL database multiple time when it has a newer iVersion number. I need to get the most recent record version but SSMS keeps screaming at me with the usual
field is not part of an aggregate function.
This is my updated code
SELECT TOP (100) Certification.dCertifiedDate, ChildGeneral.dChildsDateOfBirth, ChildGeneral.cChildsFirstName, ChildGeneral.cChildsLastName,Father.cFathersFirstName, Father.cFathersLastName,
Mother.cMothersFirstName, Mother.cMothersLastName, ChildGeneral.cChildsID, MAX(ChildGeneral.iVersionID) AS iVersionID, RecordTypes.cRecordCode, ChildGeneralFlag.cStateFileNumber
FROM ChildGeneral
INNER JOIN Father ON ChildGeneral.cChildsID = Father.cChildsID
AND ChildGeneral.iVersionID = Father.iVersionID
INNER JOIN Mother ON ChildGeneral.cChildsID = Mother.cChildsID
AND ChildGeneral.iVersionID = Mother.iVersionID
INNER JOIN ChildGeneralFlag ON ChildGeneral.cChildsID = ChildGeneralFlag.cChildsID
AND ChildGeneral.iVersionID = ChildGeneralFlag.iVersionID
INNER JOIN RecordTypes ON ChildGeneral.cRecordType = RecordTypes.cListItemID
INNER JOIN Certification ON ChildGeneral.cChildsID = Certification.cChildsID
WHERE CAST(CONVERT(VARCHAR, ChildGeneral.dChildsDateOfBirth, 101) AS DATE) >= CAST('01/01/1971' AS DATE)
AND CAST(CONVERT(VARCHAR, ChildGeneral.dChildsDateOfBirth, 101) AS DATE) <= CAST('12/31/2010' AS DATE)
GROUP BY ChildGeneral.iVersionID, Certification.dCertifiedDate, ChildGeneral.dChildsDateOfBirth, cChildsTimeOfBirth,ChildGeneral.cChildsFirstName,
ChildGeneral.cChildsLastName, Father.cFathersFirstName, Father.cFathersLastName, Mother.cMothersFirstName,
Mother.cMothersLastName, ChildGeneral.cChildsID, RecordTypes.cRecordCode, ChildGeneralFlag.cStateFileNumber
ORDER BY ChildGeneralFlag.cStateFileNumber
There should only by one record for each ChildGeneralFlag.cStateFileNumber with the MAX(ChildGeneral.iVersionID) which could be anywhere from 1-99

Answer 1

So I chopped down to 12 columns from the 6 tables and I get this error until I have added every last column to the group by. Msg 8120, Level 16, State 1, Line 33 Column 'RecordTypes.cRecordCode' is invalid in the select list because it is not contained in either an aggregate function or the GROUP BY clause. Then I lose the MAX effect and it gives me all the records
Yep that's how it goes. It's logical when you think about it:
Suppose you have rows (it doesn't matter how many table they come from or how many columns there are. Here I present just two)
City, Age
New York, 23
New York, 24
New York, 25
Chicago, 22
You want the max age by city. You group by city
SQL looks at all the unique values of City, and in this case essentially sets out two buckets, one labelled "New York" and the other labelled "Chicago". Every row's age goes into one bucket or the other. Then it looks through each bucket finding the max age in each one. You get 25 for NY and 22 for Chicago
Suppose we add another level
City, District, Age
New York, Queens, 23
New York, Queens, 24
New York, Bronx, 25
Chicago, Central, 22
You can keep your group by as is, but you can't select the district too unless you group it or max it
If you group it, SQL sets out 3 buckets this time, "New York/Bronx", "New York/Queens" and "Chicago/Central". The Queens bucket has 2 ages thrown into it, the max age in Bronx is 25, Queens is 24. Ultimately you get 3 rows out of you query because of the 3 unique values of city+district
If you max the district and the age and keep the group by only on city you get 2 rows out, but the max of district is Queens - alphabetically "greater" that Bronx, and you get 25, which truly is the max age but there was never originally a New York/Queens/25 row. Bucket values are allowed to mix up in aggregations. By stating MAX(age) and MAX(district) sql just pulls those values out of their rows, throws them all in the bucket and then finds the highest. Values in buckets retain their individual identity (whether they're an age value or a district value) but they lose all association with other values on the same row they were originally from
There isn't a concept of "group by New York, give me the max age, and also give me the district that went with it" because that might see the db having to make a choice it isn't empowered to make. Suppose we had this:
City, District, Age
New York, Queens, 23
New York, Queens, 25
New York, Bronx, 25
Chicago, Central, 22
Max is 25, but there are two rows with this max - should the DB return both? Should it pick one to throw away because you've asked for GROUP BY city which means city should be unique in the output?
"Both Rows please" you might say. "Just pick one to discard" I might say..
The db won't choose, so instead you'll have to be explicit:
SELECT *
FROM Person p
INNER JOIN (select city, max(age) maxage from person group by city) m
ON p.city = m.city AND p.age = m.maxage
Here we explicitly say "do the group, join it back, thus give me both rows" - the group by query becomes an elaborate 2 column where clause, that filters to only rows having both "New York and 25", or "Chicago and 22". You get both your New York rows.
There isn't a specified way of saying "pick one to discard" - instead we leverage some way of being explicit to break the tie- you can say "I want the city, the max age the associated district and if there are ties I want the first one when districts are sorted alphabetically".
And if you end up with a situation where there are ties in district, you have to add another level of sorting (and if that ties you keep going until there are no ties or you don't care any more)
Often that query looks like this:
SELECT * FROM(
SELECT *, ROW_NUMBER() OVER(PARTITION BY city ORDER BY age DESC, district) rn
) WHERE rn = 1
Row number essentially does the same query we did above when we grouped by city and joined it back in - partition by is like "group by with auto join back to current row based on partitioned values". In essence this establishes a column with an incrementing counter that starts from 1 per city and counts up in the given order. Because we asked for age descending, tie breaker on district ascending, the 25,Bronx row gets the 1, the 25,Queens row gets 2. Because we then filter for only rows with 1, we get rid of the queens rows
If I want a "max this group by that, and also all the other data from that row" I either need to group/max just what I want and join it back to get the detail that was lost during the join (and suffer duplication if there are multiple matches), or I PARTITION and NUMBER my rows so the max row is in position 1 and the detail was never lost.
We cannot escape the fact that GROUP BY loses detail or mixes row data
ps; there are also ways of using numbering/partitioning queries to retain ties - this answer isn't meant to be a comprehensive cover of all windowing functions, just enough to cover the point that group by is a cake that you cannot always eat and have