I have a list of records on the server sorted by a key and use pagination API to return list of segments one by one. Since items can be inserted in the middle of the list, I return the first key of the next page as a pagination token that has to be passed to get the next page.
However, I've found that DynamoDB uses the last key of the current page instead for querying API, which is null if the next page does not exist.
Question:
What are pros and cons between using the last item of the current page and the first item of the next page as a pagination token?
N.B:
As for me returning the first item is more intuitive since it's null only if the next page does not exist.
Using the "last item of the current page" (LICP) is better than using the "first item of the next page" (FINP) because it deals better with the possibility that, in the meantime, some item is inserted between these two items.
For example suppose the first page contains 3 alphabetically ordered names: Adam/Basil/Claude. And suppose the next page is Elon/Francis/Gilbert.
Then with LICP the token is Claude, while with FINP the token is Elon. If no new names are inserted, the result is the same when we get the next page.
However, suppose we insert the name Daniel after getting the first page but before getting the second page. In this case, when we get the second page with LICP we get Daniel/Elon/Francis, while with FINP we get Elon/Francis/Gilbert. That is to say, FINP will miss Daniel, while LICP will not.
Also, FINP may consume more computing resources than LICP, since you must retrieve one extra item (4 items, in the above example, instead of only 3).
Related
I have an API with 59 pages, on each page 1000 rows of data. I would like to retrieve all that data and store it in a Microsoft SQL Server.
When I use tloop with a condition run until i<59, it returns the first 1000 rows of data 59 times which is clearly not what I need.
I have tried to create a global variable next_page but I do not know how to connect it to the next_page present in the API, so that when "next_page"="" the program will know to break the loop.
I had a similar case (difference is that I didn't have a "nextPage" element but a "nextLink" which was giving me the complete URL to get to the nextPage).
I created a globalvariable "endJob" with value "false" at the beginning (tJava right before tLoop)
My tLoop is from int i=1, iteration is i++ , condition is !endJob (thus it will loop as long as the job is not marked as ended).
In a tJava right after tLoop, create the URL for your API request, using your page number, which is the tLoop_1_CURRENT_ITERATION
Then after my tRestClient, I put a tReplicate : first flow is for your needed transformations, the other one retrieve only the "nextPage" item. If nextPage is empty, then you update "endJob" variable to "true" : you want to stop the loop.
We are iteratively performing queries against the Google Plus API. They all look like the following query, where the pageToken value is changed on each iteration, updated with the nextPageToken from Google's response to the previous query.
https://www.googleapis.com/plus/v1/activities?maxResults=20&orderBy=recent&query={OUR QUERY}&pageToken={GIVEN PAGE TOKEN}&key={OUR API KEY}
The Google+ reference indicates that the nextPageToken should be absent from Google's response on the last page of results. However, this is what we experience: once we've apparently reached the last page of results, we keep receiving nextPageToken values along with an empty result set in the items field; the nextPageToken value changes on each iteration. This pattern repeats itself several times until one of the nextPageTokens finally results in a 500 error.
This is a known bug. You should star the issue to show your interest in having it fixed.
I'm using the YouTube Search API to grab 5 videos per time under a specific keyword. But I've been trying and trying, but couldent find the parameter for the start index. Does anyone know how to add it, so it gets the next 5 videos etc..?
Current URL I have:
https://www.googleapis.com/youtube/v3/search?part=snippet&q=wiz+khalifa&type=video&key=AIzaSyB9UW36sMDA9rja_J0ynSYVcNY4G25
In the results of your first query, you should get back a nextPageToken field.
When you make the request for the next page, you must send this value as the pageToken.
So you need to add pageToken=XXXXX to your query, where XXXXX is the value you received in nextPageToken.
Hope this helps
I was looking at Angel list api (https://angel.co/api) and I noticed a section on pagination. It says entries are limited to max of 50 (for eg. https://api.angel.co/1/users/135/roles has 2 pages work of data but returns only 1 page). The documentation mentions pagination but does not say how to get 2nd page.
Any ideas?
Chetan
Add ?page=2 to the end of the request to get the second page.
I'm using Google reader API to get all available items for any RSS feed. I use it as follows:
http://www.google.com/reader/atom/feed/[RSS FEED LINK]?n=[NUMBER OF ITEMS TO SHOW]&r=o&ot=[UNIX TIME STAMP FOR START DATE]
As I understand, this should return all items starting with the date specified by the time stamp (start date should not be older than one month ago). It works great for some feeds, but in most feeds, it doesn't show all available items (although they are available when using Google Reader).
For Example:
http://www.google.com/reader/atom/feed/http://www.360cities.net/rss/area/Greece.rss?n=1000&r=o&ot=1306959543
this link only shows items starting with 24-07-2011 to current date although it should show items starting with 26-06-2011. If the same link (http://www.360cities.net/rss/area/Greece.rss) is read by Google Reader, it'll show much more results.
Have any solutions?
Fortunately, I found the solution to my problem after a lot of research:
A url in this form returns the most recent N items of the RSS Feed
http://www.google.com/reader/atom/feed/[RSS]?n=[N]
[N] = Number of items to be displayed (max: 1000).
[RSS] = The url for the rss feed.
To get the next N older items, another parameter called Continuation String should be used. It can be found inside gr:continuation tag in each results' page. So, To get the N older items, a url in this form should be used:
http://www.google.com/reader/atom/feed/[RSS]?n=[N]&c=[C]
[N] = Number of items to be displayed (max: 1000).
[RSS] = The url for the rss feed.
[C] = Continuation string
Example:
Let's say we are interested to get results from http://www.360cities.net/rss/area/north-america.rss
To get newest 1000 item of this rss feed, The url to be used should look like:
http://www.google.com/reader/atom/feed/http://www.360cities.net/rss/area/north-america.rss?n=1000
To get the next older 1000 items, We should first search in the first result page and find the Continuation String. In this case the Continuation String is COnu-r7znpsC (it may be different when you view this post). Then, the url to be used should look like:
http://www.google.com/reader/atom/feed/http://www.360cities.net/rss/area/north-america.rss?n=1000&c=COnu-r7znpsC
To get the next older 1000 items, repeat the same process by finding the new Continuation String, etc...
If no Continuation String was found, this means that no more items are available.
I hope this would help someone.
Thanks