I have queries that can return large resultsets (> 100K rows). I need to display the number of results to the user and the user is able to page through the results in our application. However, nobody is going to page through 100K items when 25 are displayed on a page. So I want to limit the number of pageable results to 5K while still displaying the total number of results to the user.
Of course, I can fire two seperate queries to the database: one counting all results, one returning the TOP(5000). But the queries can be expensive.
Is there a smart way to combine these two queries into one? The queries below are over simplified:
SELECT COUNT(*) FROM TABLE WHERE field = 1;
SELECT TOP(5000) * FROM TABLE Where field = 1;
Can anyone help?
do cross join
select * from
(SELECT TOP(5000) * FROM TABLE Where field = 1) a
,(SELECT COUNT(*) as cnt FROM TABLE WHERE field = 1;) b
You can try below
SELECT TOP(5000) *,(SELECT COUNT(*) FROM TABLE WHERE field = 1)
FROM TABLE Where field = 1;
You can try below query:
SELECT TOP 5000 *,
COUNT(*)
OVER(ORDER BY (SELECT NULL))
FROM table
WHERE field = 1;
Related
I am looking to combine these 2 statement into one to run as a stored procedure if possible.
I have not used temp tables in queries before and may have to with this, not sure asking advice.
I did not write the original queries and manually run the first one which returns a table listing ID's with duplicate data nad how many records. Then each record ID is put into the 2nd query to remove all but the TOP 1 based on additional filtering criteria.
I have looked at using CTE from SQL select into delete DIRECTLY but am stil at a loss on how to pass each result row ID value into the delete query.
The queries, edited for public consumption are
SELECT id, count() FROM [DEV].[dbo].[7dtest] where FileVer = 1 and CALC_DATE > FORMAT(DATEADD(DD,-7,GETDATE()), 'yyyy-MM-dd') group by id having count() > 1 order by count(*) desc
returns a table with id and number of duplicate rows
then take the id of each row and put into this delete statement
delete from [DEV].[dbo].[7dtest] where AutoID not in (
SELECT TOP 1 AutoID FROM [DEV].[dbo].[7dtest] where FileVer = 1 and id = '123' and CALC_DATE > FORMAT(DATEADD(DD,-7,GETDATE()), 'yyyy-MM-dd')
order by COMPLETED_DATE_CHECK_3 desc, COMPLETED_DATE_CHECK_2 desc, COMPLETED_DATE_CHECK_1 desc)
and FileVer = 1 and id = '123' and CALC_DATE > FORMAT(DATEADD(DD,-7,GETDATE()), 'yyyy-MM-dd')
Can this be done with CTE or do I need to create a temp table and some looping to get the ID one row at a time? Is there a better way I should be doing this?
TIA
I need to pull back the first 300 rows from a 10MM row table, as well as getting a count of the total number of matching records.
I can do this in two queries, something like:
SELECT * FROM table WHERE field = value LIMIT 300;
SELECT count(*) FROM table WHERE field = value;
Or I could use an OVER():
SELECT *, COUNT(*) OVER() AS total FROM table WHERE field = value LIMIT 300;
Which would be the most efficient? I don't care about the need to run two queries, I'm after the most efficient solution. I'm no expert, and I've tried to run an "explain" but it doesn't make much sense to me. This is running on Amazon Redshift.
if your SortKey is timestamp field, the most efficient to run will be
select *
from(
select * , count(*) over() as total,
row_number () over(order by timestamp) as rank
from table
where filed =value)
where rank<301
I have over 300 000 rows, I would like to return total number of the records even if I pick only 20/50/100 records using TOP * statement.
Is it possible to filter select like
select top 50 * from table where rule1=rule1 and rule=rule2
Let's say that total number of records if database is 300 000 and select above would return 4 000 records. But physically it will return only 50 records, I need to know how manu is there in database using this where statement (it would return 4000).
Thanks
select top 50 *, count(*) over()
from table
where rule1=rule1 and rule=rule2
There are a lot of simple answers to this question, as other posters have pointed out. There is also a lot of subtlety depending on your scenario. There is a fairly in depth discussion of the issue # Efficient way of getting ##rowcount from a query using row_number
SELECT TOP 50
*
FROM TableName t1
INNER JOIN (SELECT
COUNT(*) AS CountOfRecords
FROM TableName) t2 on 1=1
You could do:
select top 50 *, (select count(*) from table)
from table
where rule1=rule1 and rule=rule2
This will give you the total number of rows as an extra column against each row returned by the main query. Not sure of the performance implications on this though...
Wasn't clear from your question if you need the count to be based on the filtered number of rows or not, but if so:
select top 50 *, (select count(*) from table where rule1=rule1 and rule=rule2)
from table
where rule1=rule1 and rule=rule2
This is a known question but the best solution I've found is something like:
SELECT TOP N *
FROM MyTable
ORDER BY Id DESC
I've a table with lots of rows. It is not a posibility to use that query because it takes lot of time. So how can I do to select last N rows without using ORDER BY?
EDIT
Sorry duplicated question of this one
You can get SQL server to select the last N rows with the following query:
select * from tbl_name order by id desc limit N;
I tested JonVD's code, but found it was very slow, 6s.
This code took 0s.
SELECT TOP(5) ORDERID, CUSTOMERID, OrderDate
FROM Orders where EmployeeID=5
Order By OrderDate DESC
You can do it by using the ROW NUMBER BY PARTITION Feature also. A great example can be found here:
I am using the Orders table of the Northwind database... Now let us retrieve the Last 5 orders placed by Employee 5:
SELECT ORDERID, CUSTOMERID, OrderDate
FROM
(
SELECT ROW_NUMBER() OVER (PARTITION BY EmployeeID ORDER BY OrderDate DESC) AS OrderedDate,*
FROM Orders
) as ordlist
WHERE ordlist.EmployeeID = 5
AND ordlist.OrderedDate <= 5
If you want to select last numbers of rows from a table.
Syntax will be like
select * from table_name except select top
(numbers of rows - how many rows you want)* from table_name
These statements work but differrent ways. thank you guys.
select * from Products except select top (77-10) * from Products
in this way you can get last 10 rows but order will show descnding way
select top 10 * from products
order by productId desc
select * from products
where productid in (select top 10 productID from products)
order by productID desc
select * from products where productID not in
(select top((select COUNT(*) from products ) -10 )productID from products)
First you most get record count from
Declare #TableRowsCount Int
select #TableRowsCount= COUNT(*) from <Your_Table>
And then :
In SQL Server 2012
SELECT *
FROM <Your_Table> As L
ORDER BY L.<your Field>
OFFSET <#TableRowsCount-#N> ROWS
FETCH NEXT #N ROWS ONLY;
In SQL Server 2008
SELECT *
FROM
(
SELECT ROW_NUMBER() OVER(ORDER BY ID) AS sequencenumber, *
FROM <Your_Table>
Order By <your Field>
) AS TempTable
WHERE sequencenumber > #TableRowsCount-#N
In a very general way and to support SQL server here is
SELECT TOP(N) *
FROM tbl_name
ORDER BY tbl_id DESC
and for the performance, it is not bad (less than one second for more than 10,000 records On Server machine)
Is "Id" indexed? If not, that's an important thing to do (I suspect it is already indexed).
Also, do you need to return ALL columns? You may be able to get a substantial improvement in speed if you only actually need a smaller subset of columns which can be FULLY catered for by the index on the ID column - e.g. if you have a NONCLUSTERED index on the Id column, with no other fields included in the index, then it would have to do a lookup on the clustered index to actually get the rest of the columns to return and that could be making up a lot of the cost of the query. If it's a CLUSTERED index, or a NONCLUSTERED index that includes all the other fields you want to return in the query, then you should be fine.
select * from (select top 6 * from vwTable order by Hours desc) T order by Hours
Here's something you can try without an order by but I think it requires that each row is unique. N is the number of rows you want, L is the number of rows in the table.
select * from tbl_name except select top L-N * from tbl_name
As noted before, which rows are returned is undefined.
EDIT: this is actually dog slow. Of no value really.
A technique I use to query the MOST RECENT rows in very large tables (100+ million or 1+ billion rows) is limiting the query to "reading" only the most recent "N" percentage of RECENT ROWS. This is real world applications, for example I do this for non-historic Recent Weather Data, or recent News feed searches or Recent GPS location data point data.
This is a huge performance improvement if you know for certain that your rows are in the most recent TOP 5% of the table for example. Such that even if there are indexes on the Tables, it further limits the possibilites to only 5% of rows in tables which have 100+ million or 1+ billion rows. This is especially the case when Older Data will require Physical Disk reads and not only Logical In Memory reads.
This is well more efficient than SELECT TOP | PERCENT | LIMIT as it does not select the rows, but merely limit the portion of the data to be searched.
DECLARE #RowIdTableA BIGINT
DECLARE #RowIdTableB BIGINT
DECLARE #TopPercent FLOAT
-- Given that there is an Sequential Identity Column
-- Limit query to only rows in the most recent TOP 5% of rows
SET #TopPercent = .05
SELECT #RowIdTableA = (MAX(TableAId) - (MAX(TableAId) * #TopPercent)) FROM TableA
SELECT #RowIdTableB = (MAX(TableBId) - (MAX(TableBId) * #TopPercent)) FROM TableB
SELECT *
FROM TableA a
INNER JOIN TableB b ON a.KeyId = b.KeyId
WHERE a.Id > #RowIdTableA AND b.Id > #RowIdTableB AND
a.SomeOtherCriteria = 'Whatever'
MS doesn't support LIMIT in t-sql. Most of the times i just get MAX(ID) and then subtract.
select * from ORDERS where ID >(select MAX(ID)-10 from ORDERS)
This will return less than 10 records when ID is not sequential.
This query returns last N rows in correct order, but it's performance is poor
select *
from (
select top N *
from TableName t
order by t.[Id] desc
) as temp
order by temp.[Id]
use desc with orderby at the end of the query to get the last values.
This may not be quite the right fit to the question, but…
OFFSET clause
The OFFSET number clause enables you to skip over a number of rows and then return rows after that.
That doc link is to Postgres; I don't know if this applies to Sybase/MS SQL Server.
DECLARE #MYVAR NVARCHAR(100)
DECLARE #step int
SET #step = 0;
DECLARE MYTESTCURSOR CURSOR
DYNAMIC
FOR
SELECT col FROM [dbo].[table]
OPEN MYTESTCURSOR
FETCH LAST FROM MYTESTCURSOR INTO #MYVAR
print #MYVAR;
WHILE #step < 10
BEGIN
FETCH PRIOR FROM MYTESTCURSOR INTO #MYVAR
print #MYVAR;
SET #step = #step + 1;
END
CLOSE MYTESTCURSOR
DEALLOCATE MYTESTCURSOR
In order to get the result in ascending order
SELECT n.*
FROM
(
SELECT *
FROM MyTable
ORDER BY id DESC
LIMIT N
) n
ORDER BY n.id ASC
I stumpled acros this issue while using SQL server
What i did to resolve it is order the results descending and giving row number to the results of that, After i filtered the results and turned them around again.
SELECT *
FROM (
SELECT *
,[rn] = ROW_NUMBER() OVER (ORDER BY [column] DESC)
FROM [table]
) A
WHERE A.[rn] < 3
ORDER BY [column] ASC
Easy copy paste answer
To display last 3 rows without using order by:
select * from Lms_Books_Details where Book_Code not in
(select top((select COUNT(*) from Lms_Books_Details ) -3 ) book_code from Lms_Books_Details)
Try using the EXCEPT syntax.
Something like this:
SELECT *
FROM clientDetails
EXCEPT
(SELECT TOP (numbers of rows - how many rows you want) *
FROM clientDetails)
I'm not sure if this is even a good question or not.
I have a complex query with lot's of unions that searches multiple tables for a certain keyword (user input). All tables in which there is searched are related to the table book.
There is paging on the resultset using LIMIT, so there's always a maximum of 10 results that get withdrawn.
I want an extra column in the resultset displaying the total amount of results found however. I do not want to do this using a separate query. Is it possible to add a count() column to the resultset that counts every result found?
the output would look like this:
ID Title Author Count(...)
1 book_1 auth_1 23
2 book_2 auth_2 23
4 book_4 auth_.. 23
...
Thanks!
This won't add the count to each row, but one way to get the total count without running a second query is to run your first query using the SQL_CALC_FOUND_ROWS option and then select FOUND_ROWS(). This is sometimes useful if you want to know how many total results there are so you can calculate the page count.
Example:
select SQL_CALC_FOUND_ROWS ID, Title, Author
from yourtable
limit 0, 10;
SELECT FOUND_ROWS();
From the manual:
http://dev.mysql.com/doc/refman/5.1/en/information-functions.html#function_found-rows
The usual way of counting in a query is to group on the fields that are returned:
select ID, Title, Author, count(*) as Cnt
from ...
group by ID, Title, Author
order by Title
limit 1, 10
The Cnt column will contain the number of records in each group, i.e. for each title.
Regarding second query:
select tbl.id, tbl.title, tbl.author, x.cnt
from tbl
cross join (select count(*) as cnt from tbl) as x
If you will not join to other table(s):
select tbl.id, tbl.title, tbl.author, x.cnt
from tbl, (select count(*) as cnt from tbl) as x
My Solution:
SELECT COUNT(1) over(partition BY text) totalRecordNumber
FROM (SELECT 'a' text, id_consult_req
FROM consult_req cr);
If your problem is simply the speed/cost of doing a second (complex) query I would suggest you simply select the resultset into a hash-table and then count the rows from there while returning, or even more efficiently use the rowcount of the previous resultset, then you do not even have to recount
This will add the total count on each row:
select count(*) over (order by (select 1)) as Cnt,*
from yourtable
Here is your answare:
SELECT *, #cnt count_rows FROM (
SELECT *, (#cnt := #cnt + 1) row_number FROM your_table
CROSS JOIN (SELECT #cnt := 0 AS variable) t
) t;
You simply cannot do this, you'll have to use a second query.