I have a query that selects a maximum value for a certain group, e.g.
select *
from mytable
where value in (select max(value) from mytable group by mygroupfield)
This is fast enough (~ 0.1 sec), but when I add an extra condition, e.g.
select *
from mytable
where value in (select max(value) from mytable group by mygroupfield)
AND mygroupfield='something'
the query becomes very slow (~ 4-5 seconds)
Both value and mygroupfield are indexed (non unique, non clustered, if it matters). This table contains ~ 40,000 records.
I know I can do this:
where value in (select max(value) where mygroupfield='something') (which is also fast) but due to our architecture, that is not an option right now.
Is there a way to speed up this query?
Try a correlated subquery like the example below.
SELECT a.*
FROM mytable AS a
WHERE value IN (
SELECT MAX(value)
FROM mytable AS b
WHERE b.mygroupfield=a.mygroupfield
GROUP by mygroupfield
)
AND mygroupfield='something';
This index may help as well:
CREATE INDEX idx ON dbo.mytable(mygroupfield) INCLUDE(value);
You can try using row_number() though not sure whether it'll serve your purpose or not
with cte as
(
select *,row_number() over(partition by mygroupfield order by value desc) as rn
from mytable
where mygroupfield='something'
)
select * from cte where rn=1
I would suggest a correlated subquery:
select t.*
from mytable t
where t.value = (select max(t2.value)
from mytable t2
where t2.mygroupfield = t.mygroupfield
) and
t.mygroupfield = 'something';
In particular, this can take advantage of an index on mytable(mygroupfield, value)
Related
I have a SQL something like this:
SELECT A,B,C,FUN(A) AS A FROM someTable
The problem is FUN() is a function which quite slow, so if there are a lot of records in someTable, there will be a big performance issue.
If we using a pagination, we can resolve this issue, we do the pagination like this:
SELECT * FROM(
SELECT A,B,C,FUN(A), Row_number()OVER( ORDER BY B ASC) AS rownum FROM someTable
)T WHERE T.rownum >=1 AND T.rownum<20
In this script, the FUN() will only execute 20 times so the performance is OK.
But we need use alias to order by, so we can't write rownum inline, have to move to sub query or CTE, we chose CTE and it looks like this:
;WITH CTE AS (
SELECT A,B AS alias,C,FUN(A) FROM someTable
)
SELECT * FROM(
SELECT *,Row_number()OVER( ORDER BY alias ASC) AS rownum FROM CTE
)T WHERE T.rownum >=1 AND T.rownum<20
So far we are going fine, we get pagination to solve performance issue, we solve the alias order problem, but somehow we need to add DISTINCT to the query:
;WITH CTE AS (
SELECT DISTINCT A,B AS alias,C,FUN(A) FROM someTable
)
SELECT * FROM(
SELECT *,Row_number()OVER( ORDER BY alias ASC) AS rownum FROM CTE
)T WHERE T.rownum >=1 AND T.rownum<20
After this, the optimize of this SQL seems gone, the FUN() will execute many times as much as the records count in someTable, we get the performance issue again.
Basically we are blocked at here, is there any suggestions?
The problem is that in order to get distinct values, the database engine must run the fun(a) function on all the records being selected.
If you do the fun(a) only in the final select, the distinct should not effect it, so it should run only on the final 20 records.
I've changed the derived table you've used to another cte (but that's a personal preference - seems to me more tidy not to mix derived tables and ctes):
;WITH CTE1 AS (
SELECT DISTINCT A,B AS alias,C
FROM someTable
),
CTE2 AS
(
SELECT *, ROW_NUMBER() OVER(ORDER BY alias) As RowNum
FROM CTE1
)
SELECT *, FUN(A)
FROM CTE2
WHERE RowNum >= 1
AND RowNum < 20
Please note that since the fun function is not deterministic you might get results that are different from your original query - so before adapting this solution compare the results first.
When I use a view inside of a CTE, each subquery that references the CTE seems to re-query the view. There are large chunks of the execution plan that are repeated for each subquery. This isn't the case when selecting from a table. Is this expected? Is there any way to get around it?
WITH cte AS (
SELECT v.id
FROM test_view AS v
)
SELECT TOP 25 *,
(SELECT COUNT(*) FROM cte) AS subquery
FROM cte
I'm working with SQL Server 2005
EDIT:
I'm trying to get data from a view in pages with the query below. I need the total number of rows in the view, the number of rows that match a search, and a subset of the matching rows. This works well when selecting from tables, but using a view causes repeated execution of the CTE. I attempted to force intermediate materialization a variety of different ways from the link in Martin's answer, but didn't have any luck.
WITH tableRecords AS (
SELECT *
FROM example_view
),
filteredTableRecords AS (
SELECT *, ROW_NUMBER() OVER (ORDER BY id ASC) AS tableRecordNumber
FROM tableRecords
WHERE 1 = 1
)
SELECT *,
(SELECT COUNT(*) FROM tableRecords) AS totalRecords,
(SELECT COUNT(*) FROM filteredTableRecords) AS totalDisplayRecords
FROM filteredTableRecords
WHERE tableRecordNumber BETWEEN 1 AND 25
ORDER BY tableRecordNumber ASC
Yes it is largely expected.
See Provide a hint to force intermediate materialization of CTEs or derived tables
For the query in your question you can do this though
WITH CTE AS
(
SELECT v.id,
count(*) OVER () AS Cnt
FROM test_view AS v
)
SELECT TOP 25 *
FROM CTE
ORDER BY v.id
I suggest you re-write your query as below
There are a few improvements I did over your query
removal of where 1=1, it is unnecessary as it is always true.
the sub-query in select clause will be called everytime you execute the sql script, so instead of that you can actually use the cross apply to increase the performance.
;WITH tableRecords AS(
SELECT *
FROM example_view
),
filteredTableRecords AS (
SELECT *, ROW_NUMBER() OVER (ORDER BY id ASC) AS tableRecordNumber
FROM tableRecords
),TotalNumber
(
SELECT (SELECT COUNT(1) FROM tableRecords) AS totalRecords,
(SELECT COUNT(1) FROM filteredTableRecords) AS totalDisplayRecords
)
SELECT *
FROM filteredTableRecords F
CROSS APPLY TotalNumber AS T
WHERE tableRecordNumber BETWEEN 1 AND 25
ORDER BY tableRecordNumber ASC
I want to pull all rows except the last one in Oracle SQL
My database is like this
Prikey - Auto_increment
common - varchar
miles - int
So I want to sum all rows except the last row ordered by primary key grouped by common. That means for each distinct common, the miles will be summed (except for the last one)
Note: the question was changed after this answer was posted. The first two queries work for the original question. The last query (in the addendum) works for the updated question.
This should do the trick, though it will be a bit slow for larger tables:
SELECT prikey, authnum FROM myTable
WHERE prikey <> (SELECT MAX(prikey) FROM myTable)
ORDER BY prikey
This query is longer but for a large table it should faster. I'll leave it to you to decide:
SELECT * FROM (
SELECT
prikey,
authnum,
ROW_NUMBER() OVER (ORDER BY prikey DESC) AS RowRank
FROM myTable)
WHERE RowRank <> 1
ORDER BY prikey
Addendum There was an update to the question; here's the updated answer.
SELECT
common,
SUM(miles)
FROM (
SELECT
common,
miles,
ROW_NUMBER() OVER (PARTITION BY common ORDER BY prikey DESC) AS RowRank
FROM myTable
)
WHERE RowRank <> 1
GROUP BY common
Looks like I am a little too late but here is my contribution, similar to Ed Gibbs' first solution but instead of calculating the max id for each value in the table and then comparing I get it once using an inline view.
SELECT d1.prikey,
d1.authnum
FROM myTable d1,
(SELECT MAX(prikey) prikey myTable FROM myTable) d2
WHERE d1.prikey != d2.prikey
At least I think this is more efficient if you want to go without the use of Analytics.
query to retrieve all the records in the table except first row and last row
select * from table_name
where primary_id_column not in
(
select top 1 * from table_name order by primary_id_column asc
)
and
primary_id_column not in
(
select top 1 * from table_name order by primary_id_column desc
)
How can one filter a grouped resultset for only those groups that meet some criterion compared against the other groups? For example, only those groups that have the maximum number of constituent records?
I had thought that a subquery as follows should do the trick:
SELECT * FROM (
SELECT *, COUNT(*) AS Records
FROM T
GROUP BY X
) t HAVING Records = MAX(Records);
However the addition of the final HAVING clause results in an empty recordset... what's going on?
In MySQL (Which I assume you are using since you have posted SELECT *, COUNT(*) FROM T GROUP BY X Which would fail in all RDBMS that I know of). You can use:
SELECT T.*
FROM T
INNER JOIN
( SELECT X, COUNT(*) AS Records
FROM T
GROUP BY X
ORDER BY Records DESC
LIMIT 1
) T2
ON T2.X = T.X
This has been tested in MySQL and removes the implicit grouping/aggregation.
If you can use windowed functions and one of TOP/LIMIT with Ties or Common Table expressions it becomes even shorter:
Windowed function + CTE: (MS SQL-Server & PostgreSQL Tested)
WITH CTE AS
( SELECT *, COUNT(*) OVER(PARTITION BY X) AS Records
FROM T
)
SELECT *
FROM CTE
WHERE Records = (SELECT MAX(Records) FROM CTE)
Windowed Function with TOP (MS SQL-Server Tested)
SELECT TOP 1 WITH TIES *
FROM ( SELECT *, COUNT(*) OVER(PARTITION BY X) [Records]
FROM T
)
ORDER BY Records DESC
Lastly, I have never used oracle so apolgies for not adding a solution that works on oracle...
EDIT
My Solution for MySQL did not take into account ties, and my suggestion for a solution to this kind of steps on the toes of what you have said you want to avoid (duplicate subqueries) so I am not sure I can help after all, however just in case it is preferable here is a version that will work as required on your fiddle:
SELECT T.*
FROM T
INNER JOIN
( SELECT X
FROM T
GROUP BY X
HAVING COUNT(*) =
( SELECT COUNT(*) AS Records
FROM T
GROUP BY X
ORDER BY Records DESC
LIMIT 1
)
) T2
ON T2.X = T.X
For the exact question you give, one way to look at it is that you want the group of records where there is no other group that has more records. So if you say
SELECT taxid, COUNT(*) as howMany
GROUP by taxid
You get all counties and their counts
Then you can treat that expressions as a table by making it a subquery, and give it an alias. Below I assign two "copies" of the query the names X and Y and ask for taxids that don't have any more in one table. If there are two with the same number I'd get two or more. Different databases have proprietary syntax, notably TOP and LIMIT, that make this kind of query simpler, easier to understand.
SELECT taxid FROM
(select taxid, count(*) as HowMany from flats
GROUP by taxid) as X
WHERE NOT EXISTS
(
SELECT * from
(
SELECT taxid, count(*) as HowMany FROM
flats
GROUP by taxid
) AS Y
WHERE Y.howmany > X.howmany
)
Try this:
SELECT * FROM (
SELECT *, MAX(Records) as max_records FROM (
SELECT *, COUNT(*) AS Records
FROM T
GROUP BY X
) t
) WHERE Records = max_records
I'm sorry that I can't test the validity of this query right now.
I need, if possible, a t-sql query that, returning the values from an arbitrary table, also returns a incremental integer column with value = 1 for the first row, 2 for the second, and so on.
This column does not actually resides in any table, and must be strictly incremental, because the ORDER BY clause could sort the rows of the table and I want the incremental row in perfect shape always.
The solution must run on SQL Server 2000
For SQL 2005 and up
SELECT ROW_NUMBER() OVER( ORDER BY SomeColumn ) AS 'rownumber',*
FROM YourTable
for 2000 you need to do something like this
SELECT IDENTITY(INT, 1,1) AS Rank ,VALUE
INTO #Ranks FROM YourTable WHERE 1=0
INSERT INTO #Ranks
SELECT SomeColumn FROM YourTable
ORDER BY SomeColumn
SELECT * FROM #Ranks
Order By Ranks
see also here Row Number
You can start with a custom number and increment from there, for example you want to add a cheque number for each payment you can do:
select #StartChequeNumber = 3446;
SELECT
((ROW_NUMBER() OVER(ORDER BY AnyColumn)) + #StartChequeNumber ) AS 'ChequeNumber'
,* FROM YourTable
will give the correct cheque number for each row.
Try ROW_NUMBER()
http://msdn.microsoft.com/en-us/library/ms186734.aspx
Example:
SELECT
col1,
col2,
ROW_NUMBER() OVER (ORDER BY col1) AS rownum
FROM tbl
It is ugly and performs badly, but technically this works on any table with at least one unique field AND works in SQL 2000.
SELECT (SELECT COUNT(*) FROM myTable T1 WHERE T1.UniqueField<=T2.UniqueField) as RowNum, T2.OtherField
FROM myTable T2
ORDER By T2.UniqueField
Note: If you use this approach and add a WHERE clause to the outer SELECT, you have to added it to the inner SELECT also if you want the numbers to be continuous.