As title, my program would go to a website, get reCAPTCHA audio file, recognize it, send the answer, then crawl something.
It works well when my webdriver is visible(not headless).
However when I change webdriver option to headless, reCAPTCHA detect that the webdriver is a bot when it trying to get audio file after few recognition times.
Is anything else way or option to hide my webdriver and avoid being detected as bot?
By the way, the webdriver I am using is EdgeDriver.
Related
I need to get past a Google login form using Selenium but am unable to do so. Following advice I've seen elsewhere, I've tried making the chrome driver launch with an existing chrome profile which is already logged in to Google:
options.addArguments("user-data-dir=/Users/myuser/Library/Application Support/Google/Chrome/");
options.addArguments("profile-directory=Profile 3");
This works insofar as the correct chrome profile is opened, but as soon as selenium navigates to the desired website the Google login form pops up. If I close Selenium and open the same website with the same chrome profile in a regular browser, the browser remembers me and I am able to go straight through without being prompted to login. Am I doing something wrong or has Google fixed this workaround? (I am using Mac).
I'm trying to get the e-mail of Youtube Channel. This is an example link. As you can see, to see the e-mail you have to LOGIN and then bypass RECAPTCHA. Honestly saying, I'm stuck at the first stage. Google doesn't think the selenium browser isn't safe to login and blocks it. The reason (I'm guessing) is because it thinks I'm an automized browser. So basically I have to let it think it's not a bot. I tried randomizing user-agent, and turning off the auto extension by code, but it still blocks my scraper. I have also checked if the selenium browser has "turned-off" the javascript since Google told me to check if it is.
So, I'm guessing there is a way to scrape using Scrapy or requests? Or I haven't tried enough with Selenium? I've seen an useful video that scrapes subtitles from Youtube without loggin by TechLead. So it seems it's not impossible. I'm still working on it, so if anyone has an answer or advice please let me know, thank you for reading.
Refence
Google login block Selenium Google Login Block
Randomizing User Agent Way to change Google Chrome user agent in Selenium?
Websites detecting Selenium Automation How to make Selenium script undetectable using GeckoDriver and Firefox through Python?
Try this code, for bypassing Gmail login.
Use Seleniumwire with undetected browser v2
Note: put chromedriver in your sys path.
from seleniumwire.undetected_chromedriver.v2 import Chrome, ChromeOptions
import time
options = {}
chrome_options = ChromeOptions()
chrome_options.add_argument('--user-data-dir=hash')
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--incognito")
chrome_options.add_argument("--disable-dev-shm-usage")
# chrome_options.add_argument("--headless")
browser = Chrome(seleniumwire_options=options, options=chrome_options)
browser.get('https://gmail.com')
browser.find_element_by_xpath('//*[#id="identifierId"]').send_keys('your-email')
browser.find_element_by_xpath('//*[#id="identifierNext"]/div/button').click()
time.sleep(5)
browser.find_element_by_xpath('//*[#id="password"]/div[1]/div/div[1]/input').send_keys('you-password')
browser.find_element_by_xpath('//*[#id="passwordNext"]/div/button').click()
In addition to this, selenium wire has many awesome features, check out Github repository
I am testing Salesforce Lightning pages on Chrome using Selenium Webdriver code that's in a C# NUnit project in Visual Studio 2019. I get the Login page every time I run a test.
I used to get the Verification page after the login page but after I added the IP address of the Chrome browser to the list of trusted addresses as recommended here,
How to Automate the salesforce login OTP feature using selenium WebDriver?
the verification page is not coming up anymore.
When I test Salesforce using UFT or Selenium IDE, the login page never comes up.
As can be seen in the attached image, I even have the "Remember Me" checkbox checked.
How can I stop the login page from coming up when I am testing Salesforce Lightning using Selenium Webdriver?
Also, how can Selenium recognize Javascript popups on the page like the Allow Notifications popup in the image below?
I use Nightwatch-Cucumber based on Nightwatch.js. Nightwatch.js is a NodeJS framework based on Selenium.
Currently I'm looking for a smart and simple solution to handle with Basic Authentication, especially with the Firefox. In Chrome I can do the Basic Auth via URL, like:
https://user:password#mydomain.com
But in Firefox, I get an anti-phishing dialogue box. I tried it manually on about:config in Firefox with the new entry:
network.http.phishy-userpass-length=255
But without success.
At How to Perform Basic Authentication for FirefoxDriver, ChromeDriver & IEdriver in Selenium WebDriver?, a lot of different solutions were named. But no solution is a smart one.
Maybe is it possible to send a Basic Auth header via Selenium? Or is there any other solution to realize such a Basic Auth, especially in Firefox. A generic solution for every browser/driver will be the smartest solution.
I am using selenium to perform screen scraping on some application(url).I am using IE/Firefox driver.Can we open the browser which selenium opens in an iframe instead of opening it in a new window.To be more precise driver.get should open the url in an iframe and not a window.
No. You can't open a browser inside an IFRAME. The contents of an IFRAME are always displayed by the same browser that displays the IFRAMEing page.
How to work around this strongly depends on what you are trying to accomplish and on whether you are using Selenium 1 (a browser plugin that rewrites pages by injecting JavaScript into them) or Selenium 2 (which uses WebDriver to communicate with browsers). But I'm pretty sure you'd need to rewrite some of the Selenium software.