I am creating a kind of BOT. I would like to add web search functionality to my bot. i.e. I would like my BOT to search the web for user if your says so.
To implement that I am thinking of using GOOGLE search engine. But on overviewPage of its Custom Search API, it says
Create custom search engines that search across a specified collection
of sites or pages.
Is there any way that this specified collection defaults to whole web. or is there another way round this.
Related
Requirement: From my web page user has the ability to add websites and fetch a collection of web pages having the input search term.
We have been using CSE V1 API to extract web pages based on a search term on the desired websites by making use of cref parameter. Since Google stopped supporting Link CSE feature (https://customsearch.googleblog.com/2017/02/refocusing-and-looking-forward-on.html), how do I implement the dynamic site search feature?
Thanks in Advance,
Raghu
On further analysis, custom site search is possible with the siteSearch parameter.
Ref: https://developers.google.com/custom-search/json-api/v1/reference/cse/list#siteSearch.
This parameter accepts only one website as the value, so for multiple website searches, invoke the Google CSE API multiple times for each website.
While creating the search engine, select the option "Search the entire web but emphasise included sites" so that you can hook websites while invoking the CSE API.
To enable siteSearch parameter of API
Hope it helps.
-Raghu
I want to use Google's Custom Search API to find images using GET request. Like described here. But I don't want to look on images on one specific site, but want to get results like this. How I can do this with the Custom Search API?
Sending this request returns nothing:
https://www.googleapis.com/customsearch/v1?key=[MY_KEY]&cx=017576662512468239146:omuauf_lfve&q=some+request&&safe=off&searchType=image&fileType=png. This cx is from google's manual
I must set cx param, but this value links my request to site, specified in cse control panel. However I don't want to limit my results to one site.
According to the documentation here:
Google Custom Search > Overview
Google Custom Search enables you to create a search engine for your website, your blog, or a collection of websites.
It doesn't sound like that is what you are trying to do (you don't seem to want to search a predefined subset of resources).
I'm aware the question is over 6 years old, but better late than never. It's possible to create a Google programmable search engine that can search the entire web for images. When you create a programmable search engine, there is an option to search the entire web instead of just web sites that you specify. See https://programmablesearchengine.google.com/about/.
As an alternative, you could try Bing Image Search API. There is a limit to the number of image searches you can submit per day for free, but I have rarely hit the limit.
I am trying to send a request to Google.com via Custom Search API and get the response with a proper format (XML or HTML preferably). From the Custom Search API website, I have seen that this is actually possible via "Retrieving the Code for the Search Results" (it is here). The thing is that I cannot get it working. Every time the broken robot from Google shows up. I was wondering if anyone had any experience with it and they could help me. I am trying to use the search results for a small project.
Here are the things I have done:
I can use the Google search API in general (I have used it with the search text box)
I have set up my Custom Search API to search the entire web.
Here are a bunch of things that I am not going to do:
I'm not trying to have a Google search box in my site.
I'm not trying to grab what Google says by parsing the Google.com page.
Here is what I need to do:
I need the content of what Google returns as search results via whatever API Google has to offer.
I will be using PHP for writing this program. If anyone has a better way to get these search results in a proper format, it is very appreciated.
I would like to understand if and how it is possible to achieve the following:
Create an app that sends search requests to Google (search requests
inputted by a user!)
Fetch the search results present in a different
format/graphics/layout the search results to the user.
Is that possible or Google would prevent me from doing such a thing? (via a
CAPTCHA for example)
If it is possible what kind of instrument would I need to use to capture (parse?) the search results???
My application would be developed for Android.
Screen-scraping the search results from the Google search site is a violation of their terms of service. I don't think they would use CAPTCHA to prevent you from doing so, but you might get a letter from their lawyers.
However, you could use the Google Custom Search API which allows you to search Google and customize layout of the results. However, you need to acquire a developer key and also follow their terms of service.
Also, Android has a search component built into it that you can probably use. See http://developer.android.com/guide/topics/search/index.html.
Technorarati's got their Cosmos api, which works fairly well but limits you to noncommercial use and no more than 500 queries a day.
Yahoo's got a Site Explorer InLink Data API, but it defines the task very literally, returning links from sidebar widgets in blogs rather than just links from inside blog content.
Is there any other alternative for tracking who's linking to a given URL (think of the discussion links that run below stories on Techmeme.com)? Or will I have to roll my own?
Well, it's not an API, but if you google (for example): "link:nytimes.com", the search results that come back show inbound links to that site.
I haven't tried to implement what you want yet, but the Google search API almost certainly has that functionality built in.
Is this for links to Urls under your control?
If so, you could whip up something quick that logs entries in the Referrer HTTP header.
If you wanted to do to this for an entire web site without altering application code, you could implement as an ISAPI filter or equivalent for your web server of choice.
Information available publicly from web crawlers is always going to be incomplete and unreliable (not that my solution isn't...).