I use Selenium::Remote::Driver and need to set up a "normal" user-agent. I've tried to use Selenium::UserAgent, but it has only some User-Agents for mobiles and tables, and none for usual desktop PC's. Maybe there's a way to expand the list of devices in Selenium::UserAgent, or how to set up a correct User-Agent (I'd like would be Firefox) in Selenium::Remote:Driver manually?
Update:
I'm trying to parse a website which is protected from bots (solve the puzzle if you are a human). When I try to parse it with the default Selenium+Firefox UserAgent - the protection appears.
I've tried to use Selenium::UserAgent and it worked - the protection has disappeared, but I wasn't able to scrape the needed data, because the target site promotes it's mobile apllication instead of showing the needed data this way.
So, after that I've checked the UserAgent of my home computer's browser and set it up using LWP::UserAgent:
my $ua = LWP::UserAgent->new( "Mozilla/5.0 (X11; Linux x86_64; rv:105.0) Gecko/20100101 Firefox/105.0" );
my $driver = Selenium::Remote::Driver->new( browser_name => 'firefox', ua => $ua );
But this way the protection arrived again.
After that, I've connected to my server through the VNC viewer, opened the same Firefox I've been using with Selenium, and there was no anti-bot protection this way. So, that's why I'm sure that I need to use a correct UserAgent and/or some other settings.
I am having a problem passing my test user's PKI certificates in the headless mode. I am using Java Selenium WebDriver 4.3.0. When I run my test suite in normal mode, my profile and certificates are picked up perfectly. Profile users are selected by the ChromeOptions class by identifying the --user-data-dir= . I have different profiles for each of my test users. Then the certificate is selected by the policy setting (i.e, AutoSelectCertificateForUrls). That also works perfectly. As I navigate to different URL locations my test certificates are presented and accepted correctly when I run in the normal mode.
When I change the mode to Headless=true (i.e., ChromeOptions.addArguents("--headless"), it all falls apart and no certificate is presented when I open a Chrome browser and hit any webpage.
I found that Firefox was extremely simple to manage profiles and PKI test certificates!!! When a test runs in normal mode and works perfectly, all I have to do is set the FirefoxOptions.addCommandLineOptions("--headless"); and it still works perfectly in the headless mode. Not so with Chrome!!!
Does anyone know the correct solution? I could use the information. I am really stuck here.... Is there a way to still make Chrome present PKI certificates in headless mode or does anyone know that this feature really does not work for Chrome/Chromium? Then I could stop wasting my time!
Thanks in advance for your help!
Well I actually found my own answer.
Unfortunately, it does not work!!!
It is all explained in the following issue.
Issue 1310715: Headless Chrome not using installed client (authentication) certificates from the store.
This issue shows the steps to reproduce.
UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.82 Safari/537.36
Steps to reproduce the problem:
Have a website behind a webserver that has browser client authentication with an SSL certificate.
Install the certificate in the certificate store of Chrome under $HOME/pki/nssdb folder. My certificate was in pk12 format, so I used pk12util to install it.
The installation must exit with: 'pk12util: PKCS12 IMPORT SUCCESSFUL' message.
Add a policy to your website under /etc/opt/chrome/policies/managed, so that Chrome provides the certificate automatically to the site's webserver.
Open the website.
What is the expected behavior?
When the browser is started in headless or headful mode, the browser should provide the certificate to the webserver and proceed further.
What went wrong?
Only in headful mode does the browser provide the certificate to the webserver. Headless mode does not. There's this error in the logs:
The issue response states the following:
Headless Chrome doesn't currently implement client certs. Switching this to a feature request that the headless folks can triage. Mechanically, client certs are come out of //content via CreateClientCertStore and SelectClientCertificate. Headless doesn't have a way to show UI, so it always continues without a client certificate.
https://bugs.chromium.org/p/chromium/issues/detail?id=1310715&q=component%3AInternals%3EHeadless&can=2
I encountered an issue with chrome when using selenium with python.
When the script runs the first time it works but after It won't.
The cached data have some issues, When I try to delete the cache in settings, it shows "Cached images and files
Calculating …" and keeps loading forever.
Also some websites don't work like WhatsApp web.
I discovered a workaround,
is to delete the cache directory under the Default directory and reopen chrome.
So I set my script to delete this directory before launching chrome, but I encounter a lot of errors (Permission denied, etc..), Or using the --disable-application-cache=0 option .
But this workarounds aren't best practices.
Is there any fix to this issue? (I'm sure the problem isn't in my selenium script because the bug stills even if I launch chrome by myself).
Here's my code:
def __init__(self, wait, session=None,headless=False,logging_in=False):
chrome_options = Options()
chrome_options.add_argument("--lang=en-US")
chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.41 Safari/537.36")
if session:
chrome_options.add_argument("--user-data-dir={}".format(os.path.abspath(session)))
chrome_options.add_argument("--profile-directory=Default")
if headless and os.path.exists(os.path.abspath(session)) and not os.path.exists(os.path.join(os.path.dirname(__file__),"not_logged.in")):
chrome_options.add_argument("headless")
chrome_options.add_experimental_option("excludeSwitches", ["enable-logging"])
self.browser = webdriver.Chrome(options=chrome_options) # we are using chrome as our webbrowser
else:
self.browser = webdriver.Chrome()
self.browser.get("https://web.whatsapp.com/")
I can add this to solve the issue:
chrome_options.add_argument("--disable-application-cache=0")
This disables caching which I don't want to do especially for WhatsApp.
The problem is that there is multiple instances of chromedriver.exe using the same user directory, wich causes the cache loading issue, so you need to ensure that you quit from every session you start (Even if the script fails) using:
driver.quit()
I have a problem with Google login. I want to login to my account but Google says that automation drivers are not allowed to log in.
I am looking for a solution. Is it possible to get a cookie of normal Firefox/Chrome and load it into the ChromeDriver/GeckoDriver? I thought that this can be a solution. But I am not sure is it possible or not..
Looking for solutions...
Also, I want to add a quick solution. I solved this issue by
using one of my old verified account. That can be a quick solution for
you.
I had the same problem and found the solution for it. I am using
1) Windows 10 Pro
2) Chrome Version 83.0.4103.97 (Official Build) (64-bit)
3) selenium ChromeDriver 83.0.4103.39
Some simple C# code which open google pages
var options = new ChromeOptions();
options.addArguments(#"user-data-dir=c:\Users\{username}\AppData\Local\Google\Chrome\User Data\");
IWebDriver driver = new OpenQA.Selenium.Chrome.ChromeDriver();
driver = new ChromeDriver(Directory.GetCurrentDirectory(), options);
driver.Url = "https://accounts.google.com/";
Console.ReadKey();
The core problem here you cant login when you use selenium driver, but you can use the profile which already logged to the google accounts.
You have to find where your Chrome store profile is and append it with "user-data-dir" option.
PS. Replace {username} with your real account name.
On linux the user profile is in "~/.config/google-chrome".
This error message...
...implies that the WebDriver instance was unable to authenticate the Browsing Context i.e. Browser session.
This browser or app may not be secure
This error can happen due to different factors as follows:
In the article "This browser or app may not be secure" error when trying to sign in with Google on desktop apps #Raphael Schaad mentioned that, if an user can log into the same app just fine with other Google accounts, then the problem must be with the particular account. In majority of the cases the possible reason is, this particular user account is configured with Two Factor Authentification.
In the article Less secure apps & your Google Account it is mentioned that, if an app or site doesn’t meet google-chrome's security standards, Google may block anyone who’s trying to sign in to your account from it. Less secure apps can make it easier for hackers to get in to your account, so blocking sign-ins from these apps helps keep your account safe.
Solution
In these cases the respective solution would be to:
Disable Two Factor Authentification for this Google account and execute your #Test.
Allow less secure apps
You can find a detailed discussion in Unable to sign into google with selenium automation because of "This browser or app may not be secure."
Deep Dive
However, to help protect your account, Web Browsers may not let you sign in from some browsers. Google might stop sign-ins from browsers that:
Doesn't support JavaScript or have Javascript turned off.
Have AutomationExtension or unsecure or unsupported extensions added.
Use automation testing frameworks.
Are embedded in a different application.
Solution
In these cases there are diverse solutions:
Use a browser that supports JavaScript:
Chrome
Safari
Firefox
Opera
Internet Explorer
Edge
Turn on JavaScript in Web Browsers: If you’re using a supported browser and still can’t sign in, you might need to turn on JavaScript.
If you still can’t sign in, it might be because you have AutomationExtension / unsecure / unsupported extensions turned on and you may need to turn off as follows:
public class browserAppDemo
{
public static void main(String[] args) throws Exception
{
System.setProperty("webdriver.chrome.driver", "C:\\Utility\\BrowserDrivers\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("start-maximized");
options.setExperimentalOption("useAutomationExtension", false);
options.setExperimentalOption("excludeSwitches", Collections.singletonList("enable-automation"));
WebDriver driver = new ChromeDriver(options);
driver.get("https://accounts.google.com/signin")
new WebDriverWait(driver, 10).until(ExpectedConditions.elementToBeClickable(By.xpath("//input[#id='identifierId']"))).sendKeys("gashu");
driver.findElement(By.id("identifierNext")).click();
new WebDriverWait(driver, 10).until(ExpectedConditions.elementToBeClickable(By.xpath("//input[#name='password']"))).sendKeys("gashu");
driver.findElement(By.id("passwordNext")).click();
System.out.println(driver.getTitle());
}
}
You can find a couple of relevant discussions in:
Gmail login using selenium webdriver in java
Selenium test scripts to login into google account through new ajax login form
Additional Considerations
Finally, some old browser versions might not be supported, so ensure that:
JDK is upgraded to current levels JDK 8u241.
Selenium is upgraded to current levels Version 3.141.59.
ChromeDriver is updated to current ChromeDriver v80.0 level.
Chrome is updated to current Chrome Version 80.0 level. (as per ChromeDriver v80.0 release notes)
Solution without redirecting, using firefox driver, or changing any google account settings:
If you have a specific Google account you want to access, create a chrome profile with it and then load the chrome profile when using selenium:
options = webdriver.ChromeOptions()
options.add_argument("--user-data-dir=C:/Users/{userName}/AppData/Local/Google/Chrome/User Data/Profile {#}/")
driver = webdriver.Chrome("C:/bin/chromedriver.exe", chrome_options=options)
Windows:
The profile {#} in the file path above will vary so I suggest checking inside of the User Data folder which profile you want to use. For example, if you currently only have 1 chrome account there will be no Profile directory (resorts to "Default" directory) in User Data but if you create a second chrome account there will be a "Profile 1" directory in User Data.
Note that you should create a new google chrome profile to use with selenium because attempting to use a chrome profile that is already in use (opened in another chrome window) will cause an error.
Mac:
This solution may or may not work on mac but to find the chrome account folder/filepath follow the instructions in the comment left by #bfhaha
One Solution that works for me: https://stackoverflow.com/a/60328992/12939291 or https://www.youtube.com/watch?v=HkgDRRWrZKg
Short: Stackoverflow Login with Google Account with Redirect
from selenium import webdriver
from time import sleep
class Google:
def __init__(self, username, password):
self.driver = webdriver.Chrome('./chromedriver')
self.driver.get('https://stackoverflow.com/users/signup?ssrc=head&returnurl=%2fusers%2fstory%2fcurrent%27')
sleep(3)
self.driver.find_element_by_xpath('//*[#id="openid-buttons"]/button[1]').click()
self.driver.find_element_by_xpath('//input[#type="email"]').send_keys(username)
self.driver.find_element_by_xpath('//*[#id="identifierNext"]').click()
sleep(3)
self.driver.find_element_by_xpath('//input[#type="password"]').send_keys(password)
self.driver.find_element_by_xpath('//*[#id="passwordNext"]').click()
sleep(2)
self.driver.get('https://youtube.com')
sleep(5)
username = ''
password = ''
Google(username, password)
I just tried something out that worked for me after several hours of trial and error.
Adding args: ['--disable-web-security', '--user-data-dir', '--allow-running-insecure-content' ] to my config resolved the issue.
I realized later that this was not what helped me out as I tried with a different email and it didn't work. After some observations, I figured something else out and this has been tried and tested.
Using automation:
Go to https://stackoverflow.com/users/login
Select Log in with Google Strategy
Enter Google username and password
Login to Stackoverflow
Go to https://gmail.com (or whatever Google app you want to access)
After doing this consistently for like a whole day (about 24 hours), try automating your login directly to gmail (or whatever Google app you want to access) directly... I've had at least two other people do this with success.
PS - You might want to continue with the stackoverflow login until you at least get a captcha request as we all went through that phase as well.
This might be still open / not answered
Here is an working (28.04.2021) example in this following thread:
https://stackoverflow.com/a/66308429/15784196
Use Firefox as driver. I tested his example and it did work!
#Mike-Fakesome on this https://gist.github.com/ikegami-yukino/51b247080976cb41fe93 thread suggest a solution that works
import undetected_chromedriver.v2 as uc
import random,time,os,sys
from selenium.webdriver.common.keys import Keys
GMAIL = '<GMAIL_HERE>'
PASSWORD = '<PASSWORD_HERE>'
chrome_options = uc.ChromeOptions()
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-popup-blocking")
chrome_options.add_argument("--profile-directory=Default")
chrome_options.add_argument("--ignore-certificate-errors")
chrome_options.add_argument("--disable-plugins-discovery")
chrome_options.add_argument("--incognito")
chrome_options.add_argument("user_agent=DN")
driver = uc.Chrome(options=chrome_options)
driver.delete_all_cookies()
driver.get("https://accounts.google.com/o/oauth2/v2/auth/oauthchooseaccount?redirect_uri=https%3A%2F%2Fdevelopers.google.com%2Foauthplayground&prompt=consent&response_type=code&client_id=407408718192.apps.googleusercontent.com&scope=email&access_type=offline&flowName=GeneralOAuthFlow")
driver.find_element_by_xpath("/html/body/div[1]/div[1]/div[2]/div/div[2]/div/div/div[2]/div/div[1]/div/form/span/section/div/div/div[1]/div/div[1]/div/div[1]/input").send_keys(GMAIL)
driver.find_element_by_xpath("/html/body/div[1]/div[1]/div[2]/div/div[2]/div/div/div[2]/div/div[1]/div/form/span/section/div/div/div[1]/div/div[1]/div/div[1]/input").send_keys(Keys.RETURN)
time.sleep(10)
also you can use import undetected_chromedriver as uc instead of import undetected_chromedriver.v2 as uc now as well
I solved this issue last week using following steps:
First two steps are out of your project code.
Create a new user directory for Chrome browser.
You can name this folder whatever you like and place it anywhere.
Run Chrome browser in debugger mode using just created directory
cd C:\Program Files\Google\Chrome\Application
chrome.exe --remote-debuggin-port=9222 --user-data-dir="C:\localhost"
You can use any free port but I followed this article:
https://chromedevtools.github.io/devtools-protocol/
Browser window opens.
Login manually to Google / Facebook / etc using opened window.
Close the browser.
In your project:
Copy chrome-user-directory you just created into 'resources' package.
Set debugging option for Chrome driver.
/**
* This method is added due to Google security policies changed.
* Now it's impossible to login in Google account via Selenium at first time.
* We use a user data directory for Chrome where we previously logged in.
*/
private WebDriver setWebDriver() {
ChromeOptions options = new ChromeOptions();
options.addArguments("--user-data-dir=" + System.getProperty("user.dir") + "/src/main/resources/localhost");
options.addArguments("--remote-debugging-port=9222");
return new ChromeDriver(options);
}
Enjoy.
PS: If you have another solution without copying chrome user-directory into the project, please share it)
I found a solution ,#theycallmepix and #Yinka Albi are correct but because(i think) google blacklisted accounts that did just programatically login the first time and so later they coudn't login normally. So Basically just use different a account and go to to Stackauth or StackoverFlow. Then manually login with Google(first link your account) And then manually login in google.com and then it should work prgramaticaly
P.S. pls comment if this doesn't work
Use the below given snippet method to Login to your Google Account.
Language: Python3
Redirect via: StackAuth (Reason explained at the end)
[Edit: You need to import the required packages. Make sure that the Automation that you do is running in Foreground, I mean, it's not minimised until you login completely. Once if the login is successful, then you can re-direct to the required website that you want.]
def login(username, password): # Logs in the user
driver.get('https://accounts.google.com/o/oauth2/auth/identifier?client_id=717762328687-iludtf96g1hinl76e4lc1b9a82g457nn.apps.googleusercontent'
'.com&scope=profile%20email&redirect_uri=https%3A%2F%2Fstackauth.com%2Fauth%2Foauth2%2Fgoogle&state=%7B%22sid%22%3A1%2C%22st%22%3A%2'
'259%3A3%3Abbc%2C16%3A561fd7d2e94237c0%2C10%3A1599663155%2C16%3Af18105f2b08c3ae6%2C2f06af367387a967072e3124597eeb4e36c2eff92d3eef697'
'1d95ddb5dea5225%22%2C%22cdl%22%3Anull%2C%22cid%22%3A%22717762328687-iludtf96g1hinl76e4lc1b9a82g457nn.apps.googleusercontent.com%22%'
'2C%22k%22%3A%22Google%22%2C%22ses%22%3A%2226bafb488fcc494f92c896ee923849b6%22%7D&response_type=code&flowName=GeneralOAuthFlow')
driver.find_element_by_name("identifier").send_keys(username)
WebDriverWait(driver, 10).until(expected_conditions.element_to_be_clickable((By.XPATH, "//*[#id='identifierNext']/div/button/div[2]"))).click()
driver.implicitly_wait(4)
try:
driver.find_element_by_name("password").send_keys(password)
WebDriverWait(driver, 2).until(expected_conditions.element_to_be_clickable((By.XPATH, "//*[#id='passwordNext']/div/button/div[2]"))).click()
except TimeoutException:
print('\nUsername/Password seems to be incorrect, please re-check\nand Re-Run the program.')
del username, password
exit()
except NoSuchElementException:
print('\nUsername/Password seems to be incorrect, please re-check\nand Re-Run the program.')
del username, password
exit()
try:
WebDriverWait(driver, 5).until(lambda webpage: "https://stackoverflow.com/" in webpage.current_url)
print('\nLogin Successful!\n')
except TimeoutException:
print('\nUsername/Password seems to be incorrect, please re-check\nand Re-Run the program.')
exit()
The above code, takes 2 parameters - gmailID and password. If the password or username is wrong, then you'll notified.
Why stackauth?
-> Stackauth uses OAuth 2.0 authorisation to access Google APIs(here, Google account login needs Google API to work) to securely login a user into his/her Google Account.
Click here to read more about OAuth.
Edit:
I just answered to my own question which I'd posted yesterday thinking that it might help you.
As of now, 2021, it can successfully bypass all the google restrictions that used to occur when logging in.
Feel free to revert back if it doesn't work.
Link to my answer is here
If your Chrome browser was spun up using Chromedriver, then there is detectable evidence that websites can use to determine if you're using Selenium, and then they can block you. However, if the Chrome browser is spun up before Chromedriver connects to it, then you have a browser that no longer looks like an automation-controlled one. Modern web automation libraries such as undetectable-chromedriver are aware of this, and so they make sure Chrome is spun up before connecting chromedriver to it.
The modern framework that I use for these situations is SeleniumBase in undetected chromedriver mode. Here's a script that you can use to get past automation detection on Google: (Run with python after installing seleniumbase with pip install -U seleniumbase)
from seleniumbase import SB
with SB(uc=True) as sb:
sb.open("https://www.google.com/gmail/about/")
sb.click('a[data-action="sign in"]')
sb.type('input[type="email"]', "NAME#gmail.com")
sb.click('button:contains("Next")')
sb.type('input[type="password"]', PASSWORD)
sb.click('button:contains("Next")')
sb.sleep(5)
A slow yet good solution would be delaying every key press. Why? because google uses a kind of captcha where it analyzes your typing speed and more things. So if you wanna type a mail or password like example#example.com, you'd have to do this:
for i in "example#example.com\n": #\n because the submit doesn't work in input fields in google sign in, so \n is equivalent of pressing enter
element.send_keys(i)
time.sleep(0.4) #don't forget to import time or something else with which you could delay your code!
time.sleep(1) #also this sleep because when the button will redirect url, it'd not reload the site and selenium will not wait so one more sleep
PS: if not working, try changing the values of sleep or any other delaying function
I am trying to login to my e-trade account via selenium and/or chrome_driver, but each time I attempt to login e-trade seems to be able to detect that I am using a webdriver and block the login. Is there any way for me to point my chrome driver to my normal chrome session? or prevent etrade from detecting that I am using one of these drivers? I have read a few other SO posts that have suggested some workarounds, but so far nothing has worked. I have tried both webdrivers on chrome and firefox, also tried clearing my cookies before login.
I am on ChromeDriver 76.0.3809.126 and Selenium server version: 3.141.59, any suggestions to get around this would be greatly appreciated.
I use python asyncio and pyppeteer with chrome to automate the login into etrade. A few months ago etrade changed something and I started getting blocked. With headless set to False I could see an error warning that stated to call etrade with the IP address. After a long back and forth they sent me a set of cookies (see below) that needed to be passed with a customer value specific to my account. Once they gave me the ETWLCUST1 value for the value key everything was golden from there on out. This isn't documented on the site so it was painful to figure out, but it worked for me. This may be what is blocking your access. I also passed user-agents prior to whatever etrade changed, and I still do, so that wasn't the issue for me.
Good luck if it works!
cookies = {'name':'SWH','value':'ETWLCUST1-xxxxxxx-xxxx','domain':'.etrade.com','secure':True,'httpOnly':True}
To achieve this you need to use the chrome User-Agent command-line option:
As you have not mentioned which language you are using, I am writing code in Java and Python.
To achieve this you need to use the chrome user-agent command line option.
JAVA:
ChromeOptions options = new ChromeOptions();
options.addArguments("--user-agent="+ PUT_USER_AGENT_HERE);
WebDriver driver = new ChromeDriver(options);
PYTHON:
opts = Options()
opts.add_argument("user-agent=PUT_USER_AGENT_HERE")
driver = webdriver.Chrome(chrome_options=opts)
How to get the User-Agent:
Open your actual browser.
Right-click and open inspect element.
Now go to console.
Now copy-paste below code.
Code:
navigator.userAgent
Example: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36