I am using google custom search API for searching a site and trying returning the maximum possible results by setting the num param to some high number 999 but this is sending error to me :
(400) Invalid Value
But when i set num value to 10 or lower it works perfectly, So it seems like google is putting some limit on returned results.
Here is my Google CSE link you can check by setting the num param
Google CSE API docs are here : API Docs
Any idea guys?
you can retrieve max 10 page with max 10 results.
in query you can use 'num' and 'start' params to lead your request.
num = 1-10 how many results to show
start = 1-100 the starting point
So, if you need max results, you must to do 10 request with num = 10 (default) and start = 1, 11, 21, ... 91
"queries": {
"nextPage": [
{
"title": "Google Custom Search - WTB rolex",
"totalResults": "3030",
"searchTerms": "WTB rolex",
"count": 10,
"startIndex": 11,
I think they want you to page through the result set. So, in this query, there are 3030 results and we're on page 1.
You can use the following parameters to specify which page you want:
"start": integer
It works, although I'm getting random 400s from it too (anything over 100 400s for me).
Related
I have been trying to use Bitly API.
This is the documentation I have been following.
I am trying to call a GET request over /v4/bitlinks/{bitlink}/clicks as shown in the above documentation.
params = {
'unit' : 'day',
'units' : -1
}
response = requests.get(f'https://api-ssl.bitly.com/v4/bitlinks/{domain}/2MIHzhO/clicks', headers=headers, params=params)
gives me 30 results i.e. for the last month (30 day wise entries).
However I was expecting more entries beyond that.
If i use units=40 instead, I get the following error with code 402
{'message': 'UPGRADE_REQUIRED',
'resource': 'bitlinks',
'description': 'Metrics request exceeds maximum queryable date range.'}
Is there a limit to the date range I can query because it is not explicitly stated in the API?
Yes, your guess is right. There is a limit to the date range you can query in bitly free plan - the link history limit is 30 days. That's why you get this error message when trying to query for metrics for 40 days.
You can find different bitly plans info here
When implementing a Rest API, with parameters for paging, should paging be zero indexed or start at 1. The parameters would be Page, and PageSize.
For me, it makes sense to start at 1, since we are talking about pages
There's no standard for it. Just have a look around: there are hundreds of thousands of APIs using different approaches.
Most of APIs I know use one of the following approaches for pagination:
offset and limit or
page and size
Both can be 0 or 1 indexed. Which is better? That's up to you.
Just pick the one that fits your needs and document it properly.
Additionally, you could provide some links in the response payload to make the navigation easier between the pages.
Consider, for example, you are reading data from page 2. So, provide a link for the previous page (page 1) and for the next page (page 3):
{
"data": [
...
],
"paging": {
"previous": "http://api.example.com/foo?page=1&size=10",
"next": "http://api.example.com/foo?page=3&size=10"
}
}
And remember, always make an API you would love to use.
True, there's no standard for this.
I find that Microsoft based products used (old ones like DAO for Visual Basic 6, Visual C++ 6 and similar products) to start their pagination from 1, but a lot of other tech stacks uses 0. Gradually I find that more and more libraries are using 0 instead of 1.
Why is this? It's because, mathematically speaking, it's easier to map pageIndex starting from 0 to rowNumber in DB or Array. Suppose you have a dataset fetched from a Table in DB with 100 records. Now you want to send the second page (pageSize = 10 for example). With pageIndex starting from 0, then you only need to write
startRowNumber = pageIndex * pageSize;
return dataSet[startRowNumber, startRowNumber + pageSize]
Because in most DBs and languages, arrays/lists are 0-indexed. And even if your Rest API language uses a 1-indexed array, you would still have a problem when mapping a 1-indexed pageIndex to recordIds. For example: Suppose you have a dataset indexed 1..100 (not 0..99), and you want to send the 11th to 20th records, as the second page (here pageSize=10 and pageIndex=2, because in your case you start with 1). This means you need use the formula
((pageIndex - 1) * pageSize) + 1 ; // to get the number 11.
You see that it's easier to have a 0-indexed paging for developers.
1-indexed pagination makes more sense to human users, because we start with 1 when counting everything.
https://developers.google.com/gmail/api/v1/reference/users/messages/list
The Gmail API allows to retrieve an estimate for the number of message for a given query (from:send1#gmail.com is:unread). Somehow the number that the API returns seems very different from the one shown on the webmail.
Any idea on how to return the actual number?
resultSizeEstimate is only an estimate and isn't guaranteed to be accurate for general queries. It should give more reasonable (still estimated) numbers for queries on specific labels ("label:MYLABEL" or "label:MYLABEL is:unread").
Unfortunately, there currently isn't a method of getting the actual numbers, other than retrieving all of them and looking at the size of the list returned.
Partial answer - I was hitting the 103 limit as well.
I've noticed that by playing with the maxResults parameter of the Google API I get different, yet still invalid, results.
With this value of maxResults:
request = service.users().messages().list(userId=user_id, labelIds=formatted_labels)
--> resultSizeEstimate = 103
With this value of maxResults:
request = service.users().messages().list(userId=user_id, labelIds=formatted_labels, maxResults=100)
--> resultSizeEstimate = 103
With this value of maxResults:
request = service.users().messages().list(userId=user_id, labelIds=formatted_labels, maxResults=1000)
--> resultSizeEstimate = 511
With this value of maxResults:
request = service.users().messages().list(userId=user_id, labelIds=formatted_labels, maxResults=50)
--> resultSizeEstimate = 52
This is not exactly the solution to all cases but it may help. You can get the exact number of emails under a given label using Users.labels:get
https://developers.google.com/gmail/api/v1/reference/users/labels/get
The messagesTotal value gives you that number.
Unfortunately, as others notice, listing messages with Users.messages:list or doing a search do not return accurate results.
I've built an app using twitter4j which pulls in a bunch of tweets when I enter a keyword, takes the geolocation out of the tweet (or falls back to profile location) then maps them using ammaps. The problem is I'm only getting a small portion of tweets, is there some kind of limit here? I've got a DB going collecting the tweet data so soon enough it will have a decent amount, but I'm curious as to why I'm only getting tweets within the last 12 hours or so?
For example if I search by my username I only get one tweet, that I sent today.
Thanks for any info!
EDIT: I understand twitter doesn't allow public access to the firehose.. more of why am I limited to only finding tweets of recent?
You need to keep redoing the query, resetting the maxId every time, until you get nothing back. You can also use setSince and setUntil.
An example:
Query query = new Query();
query.setCount(DEFAULT_QUERY_COUNT);
query.setLang("en");
// set the bounding dates
query.setSince(sdf.format(startDate));
query.setUntil(sdf.format(endDate));
QueryResult result = searchWithRetry(twitter, query); // searchWithRetry is my function that deals with rate limits
while (result.getTweets().size() != 0) {
List<Status> tweets = result.getTweets();
System.out.print("# Tweets:\t" + tweets.size());
Long minId = Long.MAX_VALUE;
for (Status tweet : tweets) {
// do stuff here
if (tweet.getId() < minId)
minId = tweet.getId();
}
query.setMaxId(minId-1);
result = searchWithRetry(twitter, query);
}
Really it depend on which API system you are using. I mean Streaming or Search API. In the search API there is a parameter (result_type) that is an optional parameter. The values of this parameter might be followings:
* mixed: Include both popular and real time results in the response.
* recent: return only the most recent results in the response
* popular: return only the most popular results in the response.
The default one is the mixed one.
As far as I understand, you are using the recent one, that is why; you are getting the recent set of tweets. Another issue is getting low volume of tweets that have the geological information. Because there are very few users added the geological information to their profile, you are getting very few tweets.
When performing user search with the new Twitter API (can be checked here https://dev.twitter.com/console), found the problem with limiting the number of returned results, as well as with paging.
So, let's say I want to get 5 results from searching and use count parameter:
https://api.twitter.com/1.1/users/search.json?q=Online&count=5
It works correctly, returns 5 records. But if I set count to zero, there is still one result returned:
https://api.twitter.com/1.1/users/search.json?q=Online&count=0
Is it expected?
Then i tried to use paging for the same purposes (wanted to get the first page, and limit results in it):
https://api.twitter.com/1.1/users/search.json?q=Online&per_page=5&page=1
Now it looks like the limit doesn't work at all, there are much more than 5 records returned.
Does anybody know if something is wrong with the queries, or it's an API bug?
**Try like this : **
https://api.twitter.com/1.1/users/search.json
Parameters:
1) page : Specifies the page
2) count : Number of user results to retrieve per page and max is 20.
Following example will return 1 result from page 2 for search keyword "wordpress"
<?php
require_once('api/TwitterAPIExchange.php');
require_once("token.php"); // For your access token - viral
$searchword = "wordpress";
$url = 'https://api.twitter.com/1.1/users/search.json';
$getfield = '?&page=2&count=1&q='.$searchword;
$requestMethod = 'GET';
$twitter = new TwitterAPIExchange($settings);
$followers = $twitter->setGetfield($getfield)
->buildOauth($url, $requestMethod)
->performRequest();
$json = json_decode($followers, true);
print_r($json);
?>
i think the 'per_page' is not a valid parameter in this 1.1 API.
And then, it returns the default number 20.
https://dev.twitter.com/docs/api/1.1/get/users/search