Below is one Json item returned from this query and this is the query:
https://www.googleapis.com/customsearch/v1?key={key}&cx={key}&q=Action+Motivation%2c+Inc.&alt=json
The "dc.type" in the Json is "Patent" and this is obviously patent data BUT I didn't specify that search engine. I've googled this to death but can't find anything re why patent data would be returned from a simple query like this. If Google "Action Motivation, Inc." on the regular google.com page, I get completely different (normal) results. Has anyone had this problem?
"items": [
{
"kind": "customsearch#result",
"title": "Patent US5622527 - Independent action stepper - Google Patents",
"htmlTitle": "Patent US5622527 - Independent \u003cb\u003eaction\u003c/b\u003e stepper - Google Patents",
"link": "https://www.google.com/patents/US5622527",
"displayLink": "www.google.com",
"snippet": "Apr 22, 1997 ... Original Assignee, Icon Health & Fitness, Inc., Proform Fitness ....",
"htmlSnippet": "Apr 22, 1997 \u003cb\u003e...\u003c/b\u003e Original Assignee, Icon Health & Fitness..."
"formattedUrl": "https://www.google.com/patents/US5622527",
"htmlFormattedUrl": "https://www.google.com/patents/US5622527",
"pagemap": {
"book": [
{
"description": "A motivational exercise stepping machine has a pair of independently operable pivoting treadles for operation..."
"url": "https://www.google.com/patents/US5622527?utm_source=gb-gplus-share",
"name": "Patent US5622527 - Independent action stepper",
"image": "https://www.google.com/patents?id=&printsec=frontcover&img=1&zoom=1"
}
],
"metatags": [
{
***"dc.type": "Patent"***,
"dc.title": "Independent action stepper",
"dc.contributor": "William T. Dalebout",
"dc.date": "1994-3-23",
"dc.description": "A motivational exercise stepping machine has a pair of independently operable pivoting treadles for operation by a user's feet. Each treadle..."
"dc.relation": "JP:S5110842"
}
]
}
},
{
When using their API, you can issue around 40 requests per hour. The results you see on the API is not what the real user sees. You are limited to what they give you, it's not really useful if you want to track ranking positions or what a real user would see. That's something you are not allowed to gather.
If you want a higher amount of API requests you need to pay.
60 requests per hour cost 2000 USD per year, more queries require a custom deal.
Related
Further update...
I got this working. Although Strava's documentation does not say any of the arguments in the call are mandatory it seems they all are. You need to put valid before and after arguments in epoch time and... (and this is the part that confused me a bit) you need to give a page number and items per page. The items per page default to 30 but the page number does not default. The way it works is if you say page 1 and 30 items per page you get items 1 - 30. If you say page 3 and 30 items per page you get items 31 - 60 and so on. You have to create a loop that keeps going until it gets a blank page. You then know you have retrieved all the activities. (At least that is how I think it works.)
Adrian
Question update...
After some digging and experimenting I have managed to solve some of my problem (as described below) on my own. When one creates an app on Strava listed under your settings under "My API Application" the token given has scope "read" and seems to be very, very limited.
After following the steps listed here Strava Authentication I was able to get a new token with the following scopes:
scope=read,activity:read,activity:read_all,profile:read_all,read_all
So... I thought I was "golden" as the saying goes.
Well now I am able to get individual activities using:
https://www.strava.com/api/v3/activities/2110745394?include_all_efforts="true"&access_token={{ADR_Strava_API_Key}}
But when I try to get a list of all activities I don't get any error messages but Strava simply returns
[] and this for an athlete that I know has over 1800 activities.
What I really want is to get the list of activities. Any help would be appreciated.
Thank you
Adrian
I can get athlete information back from Strava using postman using the following https request:
https://www.strava.com/api/v3/athletes/19133707?access_token={{ADR_Strava_API_Key}}
The following gets returned:
{
"id": 19133707,
"username": "adrian_geekie",
"resource_state": 2,
"firstname": "Adrian",
"lastname": "Geekie",
"city": "Gauteng, South Africa",
"state": "GP",
"country": "South Africa",
"sex": "M",
"premium": true,
"summit": true,
"created_at": "2017-01-03T16:07:37Z",
"updated_at": "2019-01-28T16:08:07Z",
"badge_type_id": 1,
"profile_medium": "https://dgalywyr863hv.cloudfront.net/pictures/athletes/19133707/5599004/2/medium.jpg",
"profile": "https://dgalywyr863hv.cloudfront.net/pictures/athletes/19133707/5599004/2/large.jpg",
"friend": null,
"follower": null
}
But when I try to get activities using this request:
https://www.strava.com/api/v3/19133707/activities?before=&after=1546293601&page=&per_page=&access_token={{ADR_Strava_API_Key}}
I get this returned:
{
"message": "Record Not Found",
"errors": [
{
"resource": "resource",
"field": "path",
"code": "invalid"
}
]
}
According to me I am asking for all records after the 1st of January 2019 i.e. epoch timestamp 1546293601. I know there are many activities for that athlete after that date. (More than 20).
I have also tried to get a single activity using:
https://www.strava.com/api/v3/activities/2110745394?include_all_efforts="true"&access_token={{ADR_Strava_API_Key}}
and I get the result:
{
"message": "Resource Not Found",
"errors": [
{
"resource": "Activity",
"field": "",
"code": "not found"
}
]
}
On the Strava developer's page the examples are given for HTTPie like this:
https://www.strava.com/api/v3/activities/{id}?include_all_efforts=" "Authorization: Bearer [[token]]
So I am replacing "Authorization: Bearer [[token]] with &access_token=
Perhaps that is my error but access_token works in the first example.
I am sorry if this is a total idiot question. I am a beginner and I would appreciate any help.
Thank you
I'm trying to structured data for producing the review like this on google search (please see the image) -
According to this link I've to write the following structured data markup -
<script type="application/ld+json">
{
"#context": "http://schema.org/",
"#type": "Review",
"itemReviewed": {
"#type": "Thing",
"name": "Super Book"
},
"author": {
"#type": "Person",
"name": "Joe"
},
"reviewRating": {
"#type": "Rating",
"ratingValue": "7",
"bestRating": "10"
},
"publisher": {
"#type": "Organization",
"name": "Washington Times"
}
}
</script>
But according to this link I've to get review from a trusted review authority. I'm wondering why we need the structured data markup (where we have static 'rating', 'bestRating' etc value definitely these shouldn't be static) or how we can combine this with trusted review authority for getting dynamic ratting that changes over time?
If I'm understanding your question correctly, I think you are confusing two issues. Google requires reviews to be created using Schema markup in order for the review to have a chance to rank directly in the SERPs.
It is the companies that provide reviews: Yelp, Angie's List, Washington Times, etc, that have to format their content management systems to upload user generated review data into the proper markup.
So if you're a web developer working for one of these companies, then it makes sense to code the CMS so that the listings are displayed using schema markup.
If you are the marketer, your job is to get reviews, not format the way they are getting displayed.
There are of course other ways to use Schema markup on your own site to boost organic traffic. Consider for example the first SERP screenshot displayed in this article.
Here the webmaster has used schema markup to list three upcoming events in their result, which gives them four links in a single listing. This causes the listing to stand out and gives increased incentive for users to click, almost guaranteeing a higher click-thru rate than if they'd have not used the markup.
I want to scrape sitelinks which are shown in the google search results(like About us Home Page etc..) . Is there any way I can retrieve them ?
enter image description here
I recently implement Google Search JSON API, and from my understanding, the only way to get the website links is through the JSON Callback where each result contains formattedUrl or htmlFormattedUrl. The query would be the site in question and hopefully the first results would give you relevant links of the site.
However, if I properly understood your question, you want to scrap the sub-links of a given website which is something that a web crawler would do. If you are the owner of the website, you can create a sitemap using many tools around the web, but if your intentions can be classified as "other", then I believe that you are barking at the wrong tree. See this question which will pinpoint you to create a simple WebCrawler.
// Example customsearch#result item in which the query was Deovandski.
"items": [
{
"kind": "customsearch#result",
"title": "Student Experience - College of Science and Mathematics (NDSU)",
"htmlTitle": "Student Experience - College of Science and Mathematics (NDSU)",
"link": "https://www.ndsu.edu/scimath/currentstudents/student_experience/",
"displayLink": "www.ndsu.edu",
"snippet": "Sep 16, 2015 ... Association for Computing Machinery Student Chapter Chair: Jordan Goetze \nAdvisor: Brian Slator. Upsilon Pi Epsilon President: Deovandski ...",
"htmlSnippet": "Sep 16, 2015 \u003cb\u003e...\u003c/b\u003e Association for Computing Machinery Student Chapter Chair: Jordan Goetze \u003cbr\u003e\nAdvisor: Brian Slator. Upsilon Pi Epsilon President: \u003cb\u003eDeovandski\u003c/b\u003e ...",
"cacheId": "pyzF9XJwrXsJ",
"formattedUrl": "https://www.ndsu.edu/scimath/currentstudents/student_experience/",
"htmlFormattedUrl": "https://www.ndsu.edu/scimath/currentstudents/student_experience/",
"pagemap": {
"cse_image": [
{
"src": "https://www.ndsu.edu/fileadmin/_processed_/csm_080117_anatomy_03med_9dbc3c8cce.jpg"
}
],
"cse_thumbnail": [
{
"width": "184",
"height": "275",
"src": "https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcTTL-GZRfSv30cyESsCnd_65BFoLMDdo8fqNS58mHfRbGiOTjSq-e-o28FE"
}
]
}
},
I've got a collection of images that I'm missing information for. I'd like to be able to do a Google reverse image search to find names, keywords, links to similar images, etc. I'm aware that scraping the search results page is against the TOS, and have gotten suggestions that using the Custom Search API is the right way to go about it, but I haven't been able to find anything in the documentation detailing reverse image search. Is anyone able to point me in the right direction if this is possible with the API, or verify if it is, in fact, supported?
Much appreciated!
As for current API I have not found any mention of reverse image search functionality. not even anything except string based queries. You can look for yourself in the detail API references for custom search.
https://developers.google.com/custom-search/docs/xml_results
https://developers.google.com/custom-search/json-api/v1/reference/cse/list
So Google-custom-search doesn't provide that facility.
After looking at other APIs provided, none of them also provide this functionality. Try looking at all the different kinds of APIs supported by the Google here:
https://developers.google.com/apis-explorer/#p/
Thus the conclusion, no reverse image search thru API form Google. (this might change, I cant say anything, look at the links provided above)
There are paid apis available from some vendors.
TinEye API https://services.tineye.com/TinEyeAPI
Incandescent API http://incandescent.xyz/pricing/
Otherwise you have to ask Google to pardon you for the little TOS violation for your pet project :)
SerpApi, a third party solution, supports scraping Google reverse image. It's a paid API with a free trial.
Let's for example use this image of Danny DeVito: https://i.imgur.com/HBrB8p0.png
Example python code (available in other libraries also):
from serpapi import GoogleSearch
params = {
"engine": "google_reverse_image",
"google_domain": "google.com",
"image_url": "https://i.imgur.com/HBrB8p0.png",
"api_key": "secret_api_key"
}
search = GoogleSearch(params)
results = search.get_dict()
Example JSON response:
...
"image_results": [
{
"position": 1,
"title": "Danny DeVito - Wikipedia",
"link": "https://en.wikipedia.org/wiki/Danny_DeVito",
"displayed_link": "https://en.wikipedia.org › wiki › Danny_DeVito",
"snippet": "Daniel Michael DeVito Jr. (born November 17, 1944) is an American actor, comedian, director, producer, and screenwriter. He gained prominence for his ...",
"cached_page_link": "https://webcache.googleusercontent.com/search?q=cache:EVb7AC9xwHYJ:https://en.wikipedia.org/wiki/Danny_DeVito+&cd=1&hl=en&ct=clnk&gl=us",
"related_pages_link": "https://www.google.com/search?q=related:https://en.wikipedia.org/wiki/Danny_DeVito&sa=X&ved=2ahUKEwi7uom3wJ_xAhWxHDQIHct6DmQQHzAAegQIBhAQ"
},
{
"position": 2,
"title": "Danny DeVito - IMDb",
"link": "https://www.imdb.com/name/nm0000362/",
"displayed_link": "https://www.imdb.com › name",
"snippet": "Danny DeVito, Actor: Matilda. Danny DeVito has amassed a formidable and versatile body of work as an actor, producer and director that spans the stage, ...",
"cached_page_link": "https://webcache.googleusercontent.com/search?q=cache:c6r3v14HA7cJ:https://www.imdb.com/name/nm0000362/+&cd=2&hl=en&ct=clnk&gl=us"
},
{
"position": 3,
"title": "Danny DeVito - Simple English Wikipedia, the free encyclopedia",
"link": "https://simple.wikipedia.org/wiki/Danny_DeVito",
"displayed_link": "https://simple.wikipedia.org › wiki › Danny_DeVito",
"thumbnail": "https://serpapi.com/searches/60cbb000ce87f8cca8f63685/images/9db7034fa3524b93ce0598116fd3b874800a67b8b9434cd54a009f2be5fd0809.jpeg",
"thumbnail_destination_url": "https://upload.wikimedia.org/wikipedia/commons/thumb/2/21/Danny_DeVito_by_Gage_Skidmore.jpg/1200px-Danny_DeVito_by_Gage_Skidmore.jpg",
"image_resolution": "1200 × 1427",
"snippet": "Daniel Michael \" · Danny\" · DeVito, Jr. (born November 17, 1944) is an American actor, director, producer and screenwriter. He has starred in and directed a number ...",
"cached_page_link": "https://webcache.googleusercontent.com/search?q=cache:2DR2mxjaZbsJ:https://simple.wikipedia.org/wiki/Danny_DeVito+&cd=32&hl=en&ct=clnk&gl=us",
"related_pages_link": "https://www.google.com/search?q=related:https://simple.wikipedia.org/wiki/Danny_DeVito&sa=X&ved=2ahUKEwi7uom3wJ_xAhWxHDQIHct6DmQQHzAfegQIFhAQ"
},
...
]
Check out the documentation for more details.
Disclaimer: I work at SerpApi.
I actually do not want to list each edit, but to get only the count of it.
this data is available for every article on the left panel in link:
https://en.wikipedia.org/w/index.php?title=Wikipedia&action=info
But this produces complete web page with tables, formatting etc and its exhaustive for wikipedia servers. So I ask if is there a way to only get those few numbers and ommit the whole website scraping.
Probably not the answer you want but there isn't a way to get this information yet.
As a workaround you can use the prop=revisions to get all the revisions contributed to the article. You will be able to count the rev tag from here:
http://en.wikipedia.org/w/api.php?format=xml&action=query&titles=Wikipedia&prop=revisions&rvprop=ids&rvlimit=max
Alternatively, you can ask YQL to count it for you with the following command:
SELECT * FROM xml
WHERE url="http://en.wikipedia.org/w/api.php?format=xml&action=query&titles=Wikipedia&prop=revisions&rvprop=ids&rvlimit=max"
AND itemPath="/api/query/pages/page/revisions/rev"
Example output (Link to full output):
{
"query": {
"count": 500, //This is the total amount of edits
"created": "2014-03-04T02:29:42Z",
"lang": "en-US",
"results": {
"rev": [{
"parentid": "597995345",
"revid": "598005528"
}, {
"parentid": "597994174",
"revid": "597995345"
}, {
"parentid": "597891867",
"revid": "597994174"
}]
}
}
}
Unfortunately, the upper limit for users to retrieve revision data is 500 and for bots it's 5000.
To get the exact count, you will have to set up a parser on your server to capture the exact count from the info page whenever a user queries the data from your side.