I am trying to login to my e-trade account via selenium and/or chrome_driver, but each time I attempt to login e-trade seems to be able to detect that I am using a webdriver and block the login. Is there any way for me to point my chrome driver to my normal chrome session? or prevent etrade from detecting that I am using one of these drivers? I have read a few other SO posts that have suggested some workarounds, but so far nothing has worked. I have tried both webdrivers on chrome and firefox, also tried clearing my cookies before login.
I am on ChromeDriver 76.0.3809.126 and Selenium server version: 3.141.59, any suggestions to get around this would be greatly appreciated.
I use python asyncio and pyppeteer with chrome to automate the login into etrade. A few months ago etrade changed something and I started getting blocked. With headless set to False I could see an error warning that stated to call etrade with the IP address. After a long back and forth they sent me a set of cookies (see below) that needed to be passed with a customer value specific to my account. Once they gave me the ETWLCUST1 value for the value key everything was golden from there on out. This isn't documented on the site so it was painful to figure out, but it worked for me. This may be what is blocking your access. I also passed user-agents prior to whatever etrade changed, and I still do, so that wasn't the issue for me.
Good luck if it works!
cookies = {'name':'SWH','value':'ETWLCUST1-xxxxxxx-xxxx','domain':'.etrade.com','secure':True,'httpOnly':True}
To achieve this you need to use the chrome User-Agent command-line option:
As you have not mentioned which language you are using, I am writing code in Java and Python.
To achieve this you need to use the chrome user-agent command line option.
JAVA:
ChromeOptions options = new ChromeOptions();
options.addArguments("--user-agent="+ PUT_USER_AGENT_HERE);
WebDriver driver = new ChromeDriver(options);
PYTHON:
opts = Options()
opts.add_argument("user-agent=PUT_USER_AGENT_HERE")
driver = webdriver.Chrome(chrome_options=opts)
How to get the User-Agent:
Open your actual browser.
Right-click and open inspect element.
Now go to console.
Now copy-paste below code.
Code:
navigator.userAgent
Example: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36
Related
I am having a problem passing my test user's PKI certificates in the headless mode. I am using Java Selenium WebDriver 4.3.0. When I run my test suite in normal mode, my profile and certificates are picked up perfectly. Profile users are selected by the ChromeOptions class by identifying the --user-data-dir= . I have different profiles for each of my test users. Then the certificate is selected by the policy setting (i.e, AutoSelectCertificateForUrls). That also works perfectly. As I navigate to different URL locations my test certificates are presented and accepted correctly when I run in the normal mode.
When I change the mode to Headless=true (i.e., ChromeOptions.addArguents("--headless"), it all falls apart and no certificate is presented when I open a Chrome browser and hit any webpage.
I found that Firefox was extremely simple to manage profiles and PKI test certificates!!! When a test runs in normal mode and works perfectly, all I have to do is set the FirefoxOptions.addCommandLineOptions("--headless"); and it still works perfectly in the headless mode. Not so with Chrome!!!
Does anyone know the correct solution? I could use the information. I am really stuck here.... Is there a way to still make Chrome present PKI certificates in headless mode or does anyone know that this feature really does not work for Chrome/Chromium? Then I could stop wasting my time!
Thanks in advance for your help!
Well I actually found my own answer.
Unfortunately, it does not work!!!
It is all explained in the following issue.
Issue 1310715: Headless Chrome not using installed client (authentication) certificates from the store.
This issue shows the steps to reproduce.
UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.82 Safari/537.36
Steps to reproduce the problem:
Have a website behind a webserver that has browser client authentication with an SSL certificate.
Install the certificate in the certificate store of Chrome under $HOME/pki/nssdb folder. My certificate was in pk12 format, so I used pk12util to install it.
The installation must exit with: 'pk12util: PKCS12 IMPORT SUCCESSFUL' message.
Add a policy to your website under /etc/opt/chrome/policies/managed, so that Chrome provides the certificate automatically to the site's webserver.
Open the website.
What is the expected behavior?
When the browser is started in headless or headful mode, the browser should provide the certificate to the webserver and proceed further.
What went wrong?
Only in headful mode does the browser provide the certificate to the webserver. Headless mode does not. There's this error in the logs:
The issue response states the following:
Headless Chrome doesn't currently implement client certs. Switching this to a feature request that the headless folks can triage. Mechanically, client certs are come out of //content via CreateClientCertStore and SelectClientCertificate. Headless doesn't have a way to show UI, so it always continues without a client certificate.
https://bugs.chromium.org/p/chromium/issues/detail?id=1310715&q=component%3AInternals%3EHeadless&can=2
I encountered an issue with chrome when using selenium with python.
When the script runs the first time it works but after It won't.
The cached data have some issues, When I try to delete the cache in settings, it shows "Cached images and files
Calculating …" and keeps loading forever.
Also some websites don't work like WhatsApp web.
I discovered a workaround,
is to delete the cache directory under the Default directory and reopen chrome.
So I set my script to delete this directory before launching chrome, but I encounter a lot of errors (Permission denied, etc..), Or using the --disable-application-cache=0 option .
But this workarounds aren't best practices.
Is there any fix to this issue? (I'm sure the problem isn't in my selenium script because the bug stills even if I launch chrome by myself).
Here's my code:
def __init__(self, wait, session=None,headless=False,logging_in=False):
chrome_options = Options()
chrome_options.add_argument("--lang=en-US")
chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.41 Safari/537.36")
if session:
chrome_options.add_argument("--user-data-dir={}".format(os.path.abspath(session)))
chrome_options.add_argument("--profile-directory=Default")
if headless and os.path.exists(os.path.abspath(session)) and not os.path.exists(os.path.join(os.path.dirname(__file__),"not_logged.in")):
chrome_options.add_argument("headless")
chrome_options.add_experimental_option("excludeSwitches", ["enable-logging"])
self.browser = webdriver.Chrome(options=chrome_options) # we are using chrome as our webbrowser
else:
self.browser = webdriver.Chrome()
self.browser.get("https://web.whatsapp.com/")
I can add this to solve the issue:
chrome_options.add_argument("--disable-application-cache=0")
This disables caching which I don't want to do especially for WhatsApp.
The problem is that there is multiple instances of chromedriver.exe using the same user directory, wich causes the cache loading issue, so you need to ensure that you quit from every session you start (Even if the script fails) using:
driver.quit()
So I have been trying to access my gmail or Google Colab Notebooks through selenium.
Instead of authenticating through Email and Password I am starting the chromedriver with an already saved Google Chrome Profile, but it throws this:
I have tried every single solution that used to work before but it ain't working now, some of the solutions that I tried are:
Disabling Two Factor authentication
Allowing Less secure app access
Disabling every single flag that I know that lets websites detect that website is being controlled by a bot.
Trying to login using some other site's Google OAuth.
Trying different Viewports
The user agent is also the same that is used when logging in without selenium manually.
options = Options()
options.add_argument('--user-data-dir=/home/mubbashir/.config/google-chrome')
options.add_argument('--profile-directory=Default')
options.add_argument("--disable-blink-features")
options.add_argument('--disable-web-security')
options.add_argument('--allow-running-insecure-content')
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
options.add_argument('--start-maximized')
options.add_argument('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36')
Nothing works at all! I am kinda stuck and any help would be very much appreciated.
Thank you so much!
For anyone facing this, the easiest way is to use cookies.
Download Colab's Cookies for that particular account.
In selenium, add them to your webdriver first and then go to the notebook link.
According to this Google does not support logging in with automation testing frameworks. I am trying to populate a calendar with game release data I scraped from here to make a public calendar I can share on reddit. I have scraped the data but now i can't even log into Google on the webdriver. What are the solutions to this?
driver = webdriver.Chrome()
driver.get("https://calendar.google.com/calendar/r")
pdb.set_trace()
#manual log in gives error
Error message: You are trying to sign in from a browser or app that doesn't allow us to keep your account secure.
Try using a different browser Learn More
There is some hack or workaround for this by setting the User-Agent in Chrome.
To achieve this you need to use the chrome user-agent command line option.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
opts = Options()
opts.add_argument("user-agent=PUT_USER_AGENT_HERE")
driver = webdriver.Chrome(chrome_options=opts)
driver.get("https://calendar.google.com/calendar/r")
pdb.set_trace()
#manual log in gives error
How to get the User-Agent:
Open your actual browser.
Right-click and open inspect element.
Now go to console.
Now copy-paste below code.
Code:
navigator.userAgent
Example: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36
I want to create a fake webcam stream for Firefox. At the moment I have the desired capability media.navigator.streams.fake but I'm not sure how to get a specific video file to play.
How can I do this?
For Firefox you can use the following code in Python:
from selenium import webdriver
options = webdriver.FirefoxOptions()
options.set_preference("media.navigator.streams.fake", True)
driver = webdriver.Firefox(firefox_options = options)
Or if you are using desired capabilities with other options then it will be like:
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
options = webdriver.FirefoxOptions()
options.set_preference("media.navigator.streams.fake", True)
desired = DesiredCapabilities.FIREFOX
desired.update(options.to_capabilities())
driver = webdriver.Firefox(desired_capabilities=desired)
PS: You can translate it into any needed programming language.
According to your question: Firefox does not support specific video file to play inside the fake webcam -- https://github.com/mozilla/geckodriver/issues/1429.
Hope it helps you!
An alternative would be that you use User Agent in chrome and then injecting the video file:
ChromeOptions options = new ChromeOptions();
options.addArguments("--use-fake-ui-for-media-stream");
options.addArguments("--use-fake-device-for-media-stream");
options.addArguments("--use-file-for-fake-video-capture=path/to/video.y4m");
options.addArguments("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0");
webDriver = new ChromeDriver(options)
PS: I create the issue that Ratmir Asanov commented
github.com/mozilla/geckodriver/issues/1429
If you want to play a specific file with firefox, I think you will have to simulate a webcam on your instance (with v4l2loopback for example), and then play your file on the virtual webcam with smthing like ffmpeg.
With selenium, you will also have to update your firefox profile for allowing access to your fake webcam.