I'm using a stored procedure with a CTE and doing some paging. I also want to return an output parameter with the total count of the returned query before my paging.
My problem is that I get an error that "OrderedSet" is not a valid object name.
#ft INT,
#page INT,
#pagesize INT,
#count INT OUTPUT
AS
BEGIN
DECLARE #offset INT
SET #offset = #page * #pagesize
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
-- Insert statements for procedure here
WITH OrderedSet AS (
SELECT *, ROW_NUMBER() OVER (ORDER BY Id DESC) AS 'Index'
FROM tbl_BulkUploadFiles buf
WHERE
buf.FileType = #ft )
SELECT * FROM OrderedSet WHERE [Index] BETWEEN #offset AND (#offset + #pagesize)
SET #count = (SELECT COUNT(*) FROM OrderedSet)
END
So my issue is on the last line, error is that last OrderedSet is not a valid object name.
Thanks in advance for any help!
Here are 2 approaches that avoid copying and pasting all the CTEs multiple times.
Return total rows as column of result set
Benefit here is that you can calculate total rows without multiple queries and temp tables, but you have to add logic to your front end to get the total row count from the first row of the result set before iterating over it to display the paged set. Another consideration is that you must account for no rows being returned, so set your total row count to 0 if no rows returned.
;WITH OrderedSet AS (
SELECT
*,
ROW_NUMBER() OVER (ORDER BY Id DESC) AS Seq,
ROW_NUMBER() OVER (ORDER BY Id) AS SeqRev
FROM tbl_BulkUploadFiles buf
WHERE buf.FileType = #ft
)
SELECT *, Seq + SeqRev - 1 AS [TotalCount]
FROM OrderedSet
WHERE Seq BETWEEN #offset AND (#offset + #pagesize)
Utilize a temp table
While there is a cost of a temp table, if your database instance follows best practices for tempdb (multiple files for multi-cores, reasonable initial size, etc), 200k rows may not be a big deal since the context is lost after the stored proc completes, so the 200k rows don't exist for too long. However, it does present challenges if these stored procs are called quite often concurrently - doesn't scale too well. However, you are not keeping the entire table - just the paged rows, so hopefully your page sizes are much smaller than 200k rows.
The approach below tries to minimize the tempdb cost being able to calculate the row count by getting only the first row due to the method of ASC and DESC ROW_NUMBERs.
;WITH OrderedSet AS (
SELECT
*,
ROW_NUMBER() OVER (ORDER BY Id DESC) AS Seq,
ROW_NUMBER() OVER (ORDER BY Id) AS SeqRev
FROM #buf buf --tbl_BulkUploadFiles buf
WHERE buf.FileType = #ft
)
SELECT * INTO #T
FROM OrderedSet
WHERE Seq BETWEEN #offset AND (#offset + #pagesize)
SET #count = COALESCE((SELECT TOP 1 SeqRev + Seq - 1 FROM #T), 0)
SELECT * FROM #T
Note: The method used above for calculating row counts was adapted from How to reference one CTE twice? and http://www.sqlservercentral.com/articles/T-SQL/66030/.
You can't use the CTE in more than one select statement. From the MSDN docs (talking about the CTE itself).
This is derived from a simple query and defined within the execution
scope of a single SELECT, INSERT, UPDATE, or DELETE statement.
You either need to run the CTE twice (probably a bad idea) or select the results of the CTE into a temp table and then select the paged data from that along with the total count.
Only alternative I see is repeating the query as inline view
select #count = numrows FROM
(
SELECT count(*) as numrows,
ROW_NUMBER() OVER (ORDER BY Id DESC) AS 'Index'
FROM tbl_BulkUploadFiles buf
WHERE
buf.FileType = #ft
) XXX WHERE [Index] BETWEEN #offset AND (#offset + #pagesize)
Related
I have a view that returns two columns Ticket_Id and Price. Each ticket can have up to 2 different prices. Along with this, I have a stored procedure that returns the data from the view to the caller based on input parameters for pagination.
#page : indicates the page number
#pageSize : indicates the number of records per page.
When a user requests 100 (unique tickets)rows I will have to return at most 200 rows of data.
For which i am using pagination as follows
OFFSET ', #pageSize,' * (',#page,' - 1) ROWS FETCH NEXT ', #pageSize,' ROWS ONLY
But it returns only 100 rows of data including duplicates. Is there a way I can modify the pagination parameters to retrieve all 200 rows of data ?
Example :
view returns as follows :
ticket_id
price
ticket1
10
ticket1
12
ticket2
11
ticket2
13
ticket3
12
ticket3
14
when the user requests with the input parameters:
#page = 1 , #PageSize = 3
I need to return all 6 rows of data.
View(Using view because stored procedure dosent not have access to tickets table directly)
select tck.ticket_id, tck.cost as 'price'
--,RANK() OVER(ORDER BY tck.ticket_id) 'Rank'
from tickets tck with (NOLOCK)
Store procedure:
ALTER PROCEDURE [dbo].[p_trans_history_srch]
-- Add the parameters for the stored procedure here
#page int=1, --optional
#pageSize int=20 --optional
AS
BEGIN
declare #finalsqlstmt nvarchar(max)
declare #pageString nvarchar(max)
declare #pageCount nvarchar(max) = ''
declare #viewName nvarchar(max)
set #pageString =concat(' OFFSET ', #pageSize,' * (',#page,' - 1) ROWS FETCH NEXT ', #pageSize,' ROWS ONLY')
set #finalsqlstmt = concat('select * from ',dbo.f_get_dbname(),#viewName,'where ',#search ,' and created_date between ''',#startDate,''' and ''',#endDate,''' order by created_date desc ',#pageString)
set #pageCount =concat('select count(distinct ticket_id) from ',dbo.f_get_dbname(),#viewName,'where ',#search,' and created_date between ''',#startDate,''' and ''',#endDate,'''' )
exec (#finalsqlstmt)
exec (#pageCount)
END
Note: I tried using RANK() OVER(ORDER BY ticket_id) 'Rank' and returning data based on rank, but because of the huge table size the performance of the query reduced drastically.
You can get "100 rows" if you pivot your two prices into two columns (for example "A" and "B"). I don't know if this is an option in your situation but here is an example:
DECLARE #t6 TABLE (A VARCHAR(100), B INT)
INSERT INTO #t6 (A,B)
SELECT 'ticket1',10
UNION ALL SELECT 'ticket1',12
UNION ALL SELECT 'ticket2',11
UNION ALL SELECT 'ticket2',13
UNION ALL SELECT 'ticket3',12
UNION ALL SELECT 'ticket3',14
;WITH cte_topivot AS
(
SELECT
[A],
CASE WHEN ROW_NUMBER() OVER (PARTITION BY A ORDER BY B ASC) = 1
THEN 'ticketA' ELSE 'ticketB' END [pivotCol],
[B]
FROM #t6 t
)
SELECT p.*
FROM cte_toPivot tp
PIVOT(MIN(B) FOR pivotCol IN ([ticketA],[ticketB])) p
Otherwise, if you are always getting exactly half the rows you want, can you multiply the user supplied page size by 2?
Assume I have a table ordered by Name column.
At the first time I'd like to select the top 500 rows.
User can add new rows to the table.
Based on user requirements.
I'd like to retrieve the next 500 rows without retrieving the first 500 rows again.
Assume that table is order by name and he added new rows that might be at the top 500.
The question is How can I select the next 500 rows including the new rows that I couldn't get at the first time because it's new rows?
What you're describing is called Paging
Here's a nice article that describes it. Server Side Paging using SQL Server 2005
Which includes this sample
DECLARE #PageSize INT,
#PageNumber INT,
#FirstRow INT,
#LastRow INT
SELECT #PageSize = 20,
#PageNumber = 3
SELECT #FirstRow = ( #PageNumber - 1) * #PageSize + 1,
#LastRow = (#PageNumber - 1) * #PageSize + #PageSize ;
WITH Members AS
(
SELECT M_NAME, M_POSTS, M_LASTPOSTDATE, M_LASTHEREDATE, M_DATE, M_COUNTRY,
ROW_NUMBER() OVER (ORDER BY M_POSTS DESC) AS RowNumber,
ROW_NUMBER() OVER (ORDER BY M_NAME DESC) AS RowNumber2
FROM dbo.FORUM_MEMBERS
)
SELECT RowNumber, M_NAME, M_POSTS, M_LASTPOSTDATE, M_LASTHEREDATE, M_DATE, M_COUNTRY
FROM Members
WHERE RowNumber BETWEEN #FirstRow AND #LastRow
ORDER BY RowNumber ASC;
Note the pagesize and pagenumber variables. These could be parameters to a stored procedure instead.
Im assming that u have a column in users table called isnewuser which is set to true for everynew user added and is not shown in the list
while you are viewing next 500 records, while some other users have been added, i would suggest you to show them in seperate list below the original one saying " new users" etcc etc..
Its makes no sense to show newly added users, which could have been on first page , on page 2 of main list .
I have to work with a potentially large list of records and I've been Googling for ways to avoid selecting the whole list, instead I want to let users select a page (like from 1 to 10) and display the records accordingly.
Say, for 1000 records I will have 100 pages of 10 records each and the most recent 10 records will be displayed first then if the user click on page 5, it will show records from 41 to 50.
Is it a good idea to add a row number to each record then query based on row number? Is there a better way of achieving the paging result without too much overhead?
So far those methods as described here look the most promising:
http://developer.berlios.de/docman/display_doc.php?docid=739&group_id=2899
http://www.codeproject.com/KB/aspnet/PagingLarge.aspx
The following T-SQL stored procedure is a very efficient implementation of paging. THE SQL optimiser can find the first ID very fast. Combine this with the use of ROWCOUNT, and you have an approach that is both CPU-efficient and read-efficient. For a table with a large number of rows, it certainly beats any approach that I've seen using a temporary table or table variable.
NB: I'm using a sequential identity column in this example, but the code works on any column suitable for page sorting. Also, sequence breaks in the column being used don't affect the result as the code selects a number of rows rather than a column value.
EDIT: If you're sorting on a column with potentially non-unique values (eg LastName), then add a second column to the Order By clause to make the sort values unique again.
CREATE PROCEDURE dbo.PagingTest
(
#PageNumber int,
#PageSize int
)
AS
DECLARE #FirstId int, #FirstRow int
SET #FirstRow = ( (#PageNumber - 1) * #PageSize ) + 1
SET ROWCOUNT #FirstRow
-- Add check here to ensure that #FirstRow is not
-- greater than the number of rows in the table.
SELECT #FirstId = [Id]
FROM dbo.TestTable
ORDER BY [Id]
SET ROWCOUNT #PageSize
SELECT *
FROM dbo.TestTable
WHERE [Id] >= #FirstId
ORDER BY [Id]
SET ROWCOUNT 0
GO
If you use a CTE with two row_number() columns - one sorted asc, one desc, you get row numbers for paging as well as the total records by adding the two row_number columns.
create procedure get_pages(#page_number int, #page_length int)
as
set nocount on;
with cte as
(
select
Row_Number() over (order by sort_column desc) as row_num
,Row_Number() over (order by sort_column) as inverse_row_num
,id as cte_id
From my_table
)
Select
row_num+inverse_row_num as total_rows
,*
from CTE inner join my_table
on cte_id=df_messages.id
where row_num between
(#page_number)*#page_length
and (#page_number+1)*#page_length
order by rownumber
Using OFFSET
Others have explained how the ROW_NUMBER() OVER() ranking function can be used to perform pages. It's worth mentioning that SQL Server 2012 finally included support for the SQL standard OFFSET .. FETCH clause:
SELECT first_name, last_name, score
FROM players
ORDER BY score DESC
OFFSET 40 ROWS FETCH NEXT 10 ROWS ONLY
If you're using SQL Server 2012 and backwards-compatibility is not an issue, you should probably prefer this clause as it will be executed more optimally by SQL Server in corner cases.
Using the SEEK Method
There is an entirely different, much faster way to perform paging in SQL. This is often called the "seek method" as described in this blog post here.
SELECT TOP 10 first_name, last_name, score
FROM players
WHERE (score < #previousScore)
OR (score = #previousScore AND player_id < #previousPlayerId)
ORDER BY score DESC, player_id DESC
The #previousScore and #previousPlayerId values are the respective values of the last record from the previous page. This allows you to fetch the "next" page. If the ORDER BY direction is ASC, simply use > instead.
With the above method, you cannot immediately jump to page 4 without having first fetched the previous 40 records. But often, you do not want to jump that far anyway. Instead, you get a much faster query that might be able to fetch data in constant time, depending on your indexing. Plus, your pages remain "stable", no matter if the underlying data changes (e.g. on page 1, while you're on page 4).
This is the best way to implement paging when lazy loading more data in web applications, for instance.
Note, the "seek method" is also called keyset paging.
Try something like this:
declare #page int = 2
declare #size int = 10
declare #lower int = (#page - 1) * #size
declare #upper int = (#page ) * #size
select * from (
select
ROW_NUMBER() over (order by some_column) lfd,
* from your_table
) as t
where lfd between #lower and #upper
order by some_column
Here's an updated version of #RoadWarrior's code, using TOP. Performance is identical, and extremely fast. Make sure you have an index on TestTable.ID
CREATE PROC dbo.PagingTest
#SkipRows int,
#GetRows int
AS
DECLARE #FirstId int
SELECT TOP (#SkipRows)
#FirstId = [Id]
FROM dbo.TestTable
ORDER BY [Id]
SELECT TOP (#GetRows) *
FROM dbo.TestTable
WHERE [Id] >= #FirstId
ORDER BY [Id]
GO
Try this
Declare #RowStart int, #RowEnd int;
SET #RowStart = 4;
SET #RowEnd = 7;
With MessageEntities As
(
Select ROW_NUMBER() Over (Order By [MESSAGE_ID]) As Row, [MESSAGE_ID]
From [TBL_NAFETHAH_MESSAGES]
)
Select m0.MESSAGE_ID, m0.MESSAGE_SENDER_NAME,
m0.MESSAGE_SUBJECT, m0.MESSAGE_TEXT
From MessageEntities M
Inner Join [TBL_NAFETHAH_MESSAGES] m0 on M.MESSAGE_ID = m0.MESSAGE_ID
Where M.Row Between #RowStart AND #RowEnd
Order By M.Row Asc
GO
Why not to use recommended solution:
SELECT VALUE product FROM
AdventureWorksEntities.Products AS product
order by product.ListPrice SKIP #skip LIMIT #limit
I have this procedure used for getting items on the current page. I would also like the include an OUT var having the total number of items so I can calculate the total number of pages.
USE [some_name]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[GetFilteredProductDetails]
#start int,
#end int
AS
BEGIN
SET NOCOUNT ON;
SELECT *
FROM
(
SELECT *, (ROW_NUMBER() OVER (ORDER BY itemid)) AS row
/* the rest of a big complex query that, so far, works.*/
) AS q
WHERE
(
row BETWEEN #start AND #end
)
END
This is my current (stripped)query, how would I be able to get either get the last rownumber/total rowcount of the inner select, or include the last row alongside of the rows between #start and #end.
COUNT(*) with an empty OVER() clause will give you the total row count. You could then add that into the WHERE clause if you need the last row returned.
SELECT *
FROM
(
SELECT *,
(ROW_NUMBER() OVER (ORDER BY itemid)) AS row,
COUNT(*) OVER() AS row_count
/* the rest of a big complex query that, so far, works.*/
) AS q
WHERE
(
row BETWEEN #start AND #end or row=row_count
)
I have a query that returns a large number of heavy rows.
When I transform this rows in a list of CustomObject I have a big memory peak, and this transformation is made by a custom dotnet framework that I can't modify.
I need to retrieve a less number of rows to do "the transform" in two passes and then avoid the memory peak.
How can I split the result of a query by half? I need to do it in DB layer. I thing to do a "Top count(*)/2" but how to get the other half?
Thank you!
If you have identity field in the table, select first even ids, then odd ones.
select * from Table where Id % 2 = 0
select * from Table where Id % 2 = 1
You should have roughly 50% rows in each set.
Here is another way to do it from(http://www.tek-tips.com/viewthread.cfm?qid=1280248&page=5). I think it's more efficient:
Declare #Rows Int
Declare #TopRows Int
Declare #BottomRows Int
Select #Rows = Count(*) From TableName
If #Rows % 2 = 1
Begin
Set #TopRows = #Rows / 2
Set #BottomRows = #TopRows + 1
End
Else
Begin
Set #TopRows = #Rows / 2
Set #BottomRows = #TopRows
End
Set RowCount #TopRows
Select * From TableName Order By DisplayOrder
Set RowCount #BottomRows
Select * From TableNameOrder By DisplayOrderDESC
--- old answer below ---
Is this a stored procedure call or dynamic sql? Can you use temp tables?
if so, something like this would work
select row_number() OVER(order by yourorderfield) as rowNumber, *
INTO #tmp
FROM dbo.yourtable
declare #rowCount int
SELECT #rowCount = count(1) from #tmp
SELECT * from #tmp where rowNumber <= #rowCount / 2
SELECT * from #tmp where rowNumber > #rowCount / 2
DROP TABLE #tmp
SELECT TOP 50 PERCENT WITH TIES ... ORDER BY SomeThing
then
SELECT TOP 50 PERCENT ... ORDER BY SomeThing DESC
However, unless you snapshot the data first, a row in the middle may slip through or be processed twice
I don't think you should do that in SQL, unless you will always have a possibility to have the same record 2 times.
I would do it in an "software" programming language, not SQL. Java, .NET, C++, etc...