I am having a little issue and need help.
I am trying to automate searching on website but get elements by name is not working on that specific website.
The same code is working with other websites such as bing, or google, but it is not working on my specific website that I want to use on.
I might be indexing the name location incorrect since there are many "names" in the website, however I tested one by one changing from 0 to 20, non of them worked.
I tried get elements by id but it is not working either.
Can anyone tell me what might be the reason and what can be the solution?
Do some websites restrict use of get element method, or web automation as a whole?
Related
I have a scrapy project which crawls all the internal links of a given website. This is working fine, however we have found a few situations where we want to limit the crawling to a particular section of a website.
For example, if you could imagine a bank has a special section for investor information, e.g. http://www.bank.com/investors/
So in the example above, everything in http://www.bank.com/investors/ only would be crawled. For example, http://www.bank.com/investors/something/, http://www.bank.com/investors/hello.html, http://www.bank.com/investors/something/something/index.php
I know I could write some hacky code on parse_url which scans the URL and does a pass if it doesn't meet my requirements (i.e. it's not /investors/), but that seems horrible.
Is there a nice way to do this?
Thank you.
I figured this out.
You need to add an allow() for the pattern you want to allow.
For example:
Rule(LinkExtractor(allow=(self.this_folder_only)), callback="parse_url", follow=True)
Everything else will be denied.
So, I've spent about 2 hours trying to get the I'm Feeling Lucky URL to work. It seems the URL doesn't like the periods in the search parameter, so does anyone have any potential tricks?
Search Value= 40.840.1/8Z
The first result in a regular Google search is the correct page.
Here's what I've tried:
http://www.google.com/search?btnI=I&q=40.840.1/8Z
http://www.google.com/search?btnI=I&q=40.840.1%2F8Z
http://www.google.com/search?btnI=I&q=40%2E840%2E1/8Z
http://www.google.com/search?btnI=I&q=40%2E840%2E1%2F8Z
http://www.google.com/search?btnI=I&q=40%2F840%2F1%2F8Z
(That one was actually pretty close)
http://www.google.com/search?btnI=I&q=40%20840%201%208Z
And all of the above surrounded in quotes (%22)
The problem is that the I'm Feeling Lucky aspect doesn't work. It finds the correct results, it just doesn't navigate to the first result. I'm open to alternatives besides the I'm Feeling Lucky URL parameters as well.
I'm trying to implement this into a .NET application that provides employees with resource information, which is best received from the manufacturer's website(s). The trick is that the resources are from many different suppliers and the links need to be somewhat automatic. Basically I don't whomever manages the software to update these links. To navigate, I'm simply using the Process.Start("http://www.example.com/") command which uses the default browser to navigate to the address.
This post helped a lot by the way.
I wasn't able to get any closer than your closest one.
But if it helps, here's an alternative way of writing the "I'm feeling lucky" URL.
http://google.com/search?q=haimer+usa+40%2F840%2F1%2F8Z&btnI
What I did to find the right url is to navigate to google.com. After this I turned my internet connection off. I entered the search details and pressed submit. You can now see the url in the address bar, but it doesn't redirect you to the first result. You can now copy the url and see how google treats your dots and other weird characters.
So to recap:
Go to google.com
Turn your internet connection off
Enter search term
Press 'I'm feeling lucky'
Copy the url from the address bar
You can create a google custom search engine of your own, and either exclude certain sites or include specific sites only, use http://cse.google.com to do this.
There is a SO tag for google custom search
I've searched all over the place and I can't figure out what I'm doing wrong. No matter what I still get a Page does not contain authorship markup on the structured data testing tool
I have two sites with almost identical pages. The rel=author tags are inserted the same way.
Here is an example of one page that works: http://bit.ly/18odGef
Here is an example of one page that doesn't: http://bit.ly/12vXdAm
I tried adding ?rel=author to the end of the Google+ profile URL, which doesn't seem to work on either site. I am not blocking anything via nofollow or robots.txt. The tool is not being blocked by a firewall or anything. Can anyone see what I'm doing wrong here and why it works for one site, but not the other?
FYI, the site that does not work used to work without a problem. I hadn't changed anything with how the author markup was organized until I realized it wasn't working anymore.
When I test both of those pages in Google's structured data test tool, it shows that authorship is working correctly for both pages.
Here are the results for the page you said was working: https://www.google.com/webmasters/tools/richsnippets?q=http%3A%2F%2Fnikonites.com%2Fd5100%2F2507-d5100-vs-d90.html%23axzz2rFFm1eVv
Here are the results for the page you said wasn't working: https://www.google.com/webmasters/tools/richsnippets?q=http%3A%2F%2Fcellphoneforums.net%2Fsamsung-galaxy%2Ft359099-enable-auto-correct-galaxy-note-ii.html%23axzz2rFFlwz3W
I'm a newbie at stackoverflow so please be patient with me :)
I'm trying to get access with the Google Custom Search API.
But I get return that I can't understand.
My query is like this:
https://www.googleapis.com/customsearch/v1?&key=********&q=red%2Bsox&cx=**********&start=0&num=10&cr=countryCA&lr=lang_fr&client=google-csbe&output=xml_no_dtd
And the result I get is this?
string '{"error": {"errors": [{"domain": "global","reason": "invalid","message": "Invalid Value"}],"code": 400,"message": "Invalid Value"}}' (length=172)
What am I doing wrong?
I want the result from Google to appear.
Thanks in advance :)
You don't have a cx.
Take a look at this answer
What happens is because this api is used mostly for adding a search option
for your site you have to specify you custom search engine (e.g. search only your site).
When you want this to search the web by code you need to do the above. Add a fake
site (where you would add your search textbox), configure it (search the web, or your site, or whatever else) and then delete the fake site
Update
Oh god, i just saw that. Sorry. Well the problem is that you start with 0. Valid is 1. Change start=0 with start=1 and i think you would be good to go. Take a look at this for valid values for the start parameter official page
I have a website and in my website I have, for example, a list of Audi models. I saw, using google webmaster tools, that my website appears in the google search by the word audi, but the target page was the 22nd page from my result set, not the first. I need my first page to appead, not my last (or middle), but I cannot tell google that this is a parameter, because my URLs are rewritten using mod rewrite. Any ideas?
BTW, I have read in a SEO forum, that it's a bad idea to use a cannonical tag. So is it really a bad idea in my case?
You can't force Google to do anything, however, they have made it easier to deal with pagination issues with a recent post on rel="next" and rel="prev".
But the primary problem you face is signalling to Google that your first (main) page is the starting point - this is achieved using internal link and back-link "juice" focussed on that page. You need to ensure that the first page of results is linked to properly from higher-value pages (like the home-page).
Google recently announced that you can use View All which will allow them to find and index entire articles that are normally broken up using pagination and display them all as one result.