I am trying to send a request to Google.com via Custom Search API and get the response with a proper format (XML or HTML preferably). From the Custom Search API website, I have seen that this is actually possible via "Retrieving the Code for the Search Results" (it is here). The thing is that I cannot get it working. Every time the broken robot from Google shows up. I was wondering if anyone had any experience with it and they could help me. I am trying to use the search results for a small project.
Here are the things I have done:
I can use the Google search API in general (I have used it with the search text box)
I have set up my Custom Search API to search the entire web.
Here are a bunch of things that I am not going to do:
I'm not trying to have a Google search box in my site.
I'm not trying to grab what Google says by parsing the Google.com page.
Here is what I need to do:
I need the content of what Google returns as search results via whatever API Google has to offer.
I will be using PHP for writing this program. If anyone has a better way to get these search results in a proper format, it is very appreciated.
Related
When I Google for something and click on a heading from the Knowledge Graph, Google displays a list of the results just below the search bar:
Screenshot here
I typed in the same thing in my Knowledge Graph API:
https://kgsearch.googleapis.com/v1/entities:search?query=apple%20products&key=[MY_KEY]&indent=True&limit=100
and the results are not the same. Short of scraping, is there a way of obtaining the exact list shown within the results via a Google API? It doesn't seem to be Knowledge Graph, or at least the queries are not the same.
Short of scraping, is there a way of obtaining the exact list shown
within the results via a Google API? It doesn't seem to be Knowledge
Graph
Apparently not, you are using the correct API but unfortunately that is how the API works—it usually gives you a different result than if you do a simple Google search.
Why this is the case—I don't know. I decided scrape google instead.
The Knowledge Graph API doesn't have access to the same data sources as a simple Google search. You can read about here if interested (https://web.archive.org/web/20130329151128/http://zecblog.com/2012/09/16/the-short-life-of-the-open-knowledge-graph/).
But yes in short, you'll have to do your own scraping if you want the same information, although it's difficult because Google purposely obfuscates the HTML tags.
I am creating a kind of BOT. I would like to add web search functionality to my bot. i.e. I would like my BOT to search the web for user if your says so.
To implement that I am thinking of using GOOGLE search engine. But on overviewPage of its Custom Search API, it says
Create custom search engines that search across a specified collection
of sites or pages.
Is there any way that this specified collection defaults to whole web. or is there another way round this.
I want to use Google's Custom Search API to find images using GET request. Like described here. But I don't want to look on images on one specific site, but want to get results like this. How I can do this with the Custom Search API?
Sending this request returns nothing:
https://www.googleapis.com/customsearch/v1?key=[MY_KEY]&cx=017576662512468239146:omuauf_lfve&q=some+request&&safe=off&searchType=image&fileType=png. This cx is from google's manual
I must set cx param, but this value links my request to site, specified in cse control panel. However I don't want to limit my results to one site.
According to the documentation here:
Google Custom Search > Overview
Google Custom Search enables you to create a search engine for your website, your blog, or a collection of websites.
It doesn't sound like that is what you are trying to do (you don't seem to want to search a predefined subset of resources).
I'm aware the question is over 6 years old, but better late than never. It's possible to create a Google programmable search engine that can search the entire web for images. When you create a programmable search engine, there is an option to search the entire web instead of just web sites that you specify. See https://programmablesearchengine.google.com/about/.
As an alternative, you could try Bing Image Search API. There is a limit to the number of image searches you can submit per day for free, but I have rarely hit the limit.
I would like to understand if and how it is possible to achieve the following:
Create an app that sends search requests to Google (search requests
inputted by a user!)
Fetch the search results present in a different
format/graphics/layout the search results to the user.
Is that possible or Google would prevent me from doing such a thing? (via a
CAPTCHA for example)
If it is possible what kind of instrument would I need to use to capture (parse?) the search results???
My application would be developed for Android.
Screen-scraping the search results from the Google search site is a violation of their terms of service. I don't think they would use CAPTCHA to prevent you from doing so, but you might get a letter from their lawyers.
However, you could use the Google Custom Search API which allows you to search Google and customize layout of the results. However, you need to acquire a developer key and also follow their terms of service.
Also, Android has a search component built into it that you can probably use. See http://developer.android.com/guide/topics/search/index.html.
I'm working with a simple app, that allows user to enter some text, then delegates it to the google.com, searching for this text. Something like a Google search bar in Firefox.
In Google toolbar, there is a feature: when you start typing the text, some snippets appear (sorted, as far as I understand, by the number of search results), that allow you to enter your text quickly by choosing one of the snippets.
My question is how can I retrieve these snippets from Google? Does Google provide any API for doing it? I've googled this question, but Google seems to provide a search API, and allows to retrieve search results, not snippets.
p.s. I would appreciate any links, related to the question, or tutorials, or code snippets or something. I am writing a simple WinAPI application in C/C++. Thanks in advance!
Looks like I've got it!
Google provide a simple, but handy REST API. You form a special URL, then send an HTTP request and receive a formatted list of necessary search suggestions. Looks like Google supports many formatting options, for example JSON and XML.
see more info on:
http://blogoscoped.com/archive/2006-08-17-n22.html