selenium automation signing to google cloud throws networks error - selenium

I'm using selenium & selenium wire in my project. I'm writing flows to log in google cloud portals..
I enter my google cloud mail then, press on continue in Google sign in and then it log in to gcp.
I got some errors:
Request has invalid authentication credentials. Expected OAuth 2 access token, login cookie or other...
net:: ERR_PROXY_CONNECTION_FAILED
when I do the same flow manually without automation, with the same credentials, it works fine and no any network error.
my web driver
from seleniumwire import webdriver
from seleniumwire.webdriver import ChromeOptions
def test_gcp_flow():
options = ChromeOptions()
options.add_experimental_option("detach", True)
options.add_argument('--no-sandbox')
options.add_argument('--single-process')
options.add_argument('--disable-dev-shm-usage')
options.add_argument("--start-maximized")
options.add_argument('--auto-open-devtools-for-tabs')
options.add_argument('--log-level=2')
options.add_argument('--disable-features=IsolateOrigins,site-per-process')
options.add_argument("--ignore_ssl")
options.add_argument('--ignore-ssl-errors')
options.add_argument('--ignore-certificate-errors')
options.add_argument("--disable-extensions")
options.add_argument("--disable-setuid-sandbox")
options.add_argument("--dns-prefetch-disable")
options.add_argument('ignore-certificate-errors')
options.add_argument('disable-web-security')
options.add_argument('--allow-insecure-localhost')
driver = webdriver.Chrome(options=options)
driver.get('....any-hidden-url')
# more flow actions - then it open gcp portal
I added openssl.cnf (without this openssl, it shows me TLS ssl issue) to run it locally in my test using Pycharm:
openssl_conf = openssl_init
[openssl_init]
ssl_conf = ssl_sect
[ssl_sect]
system_default = system_default_sect
[system_default_sect]
Options = UnsafeLegacyRenegotiation
I tried to add some change the chrome options that added to selenium driver but nothing has changed, still same error.
I tried to use Firefox webdriver and it works good without any network issue.
Maybe it's any chrome cache issue? because some days ago it worked me with chrome..
what I'm expecting is that to sign in to gmail without network/token issues.

I can think of two possible options:
Google detects Selenium and prevents login with it for security reasons. Well selenium web driver injects recognizable javascript to the browser, and you can detect it using a simple script.
You have an authentication cookie in your browser that is not used with Selenium.
Each Selenium session initiates itself completely clean. You can save your cookies with pickle for instance and then inject it to Selenium web driver with Python. Although I would avoid it if you cant store it encrypted in a secure place and fetch it during run time (or else - at some point it will be stolen no matter how much you think you are secured).
By the way if its something that you really need - YOU are a paying customer, maybe google has an available solutions for this kind of need.
Good luck!

Try using selenium wire with an undetected chrome-driver.
pip install selenium-wire
pip install undetected-chromedriver
from seleniumwire.undetected_chromedriver.v2 import Chrome, ChromeOptions
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.ui import WebDriverWait
if __name__ == "__main__":
options = {}
chrome_options = ChromeOptions()
chrome_options.add_argument('--user-data-dir=hash')
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--incognito")
chrome_options.add_argument("--disable-dev-shm-usage")
browser = Chrome(seleniumwire_options=options, options=chrome_options)
browser.get('your url')
... your rest of the code
I hope it resolves the issue.

Related

Open a chrome profile with full functionality in Selenium

I have a code in cmd that opens an arbitrary Chrome profile and it works very well.
start "" G:\Multi_Chrome_Part_1\ChromeFirstProVip\GoogleChromePortable.exe" --profile-directory="Profile 17". Tt shows up as usual:
But I'm having a lot of trouble opening with Selenium. I use the code below:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("user-data-dir=G:\\Multi_Chrome_Part_1\\ChromeFirstProVip\\Data\\profile\\Profile 17")
driver = webdriver.Chrome(service=Service("D:\\Python\\FirstProject\\chromedriver_99_0_4844_51.exe"), options=options)
driver.get("https://www.google.co.in")
With selenium, it cannot open a complete profile, missing gmail. It looks like this:
As you can see, in addition to not accepting pre-imported gmail, it also lacks the extensions that I have installed before, as well as other components. Some websites are also not accessible with Selenium
I want to ask is there a way to turn the code above into the bottom? Is there a way to make Selenium like a normal Chrome profile? I don't understand why opening with cmd is perfect, but selenium has problems. Thank you very much if you can help

How to run Selenium chromedriver headless while using a user profile

I have coded a script to perform some actions while logged into my youtube account, using my Chrome user data folder:
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
options = Options()
options.add_argument("user-data-dir=C:\\Users\\Username\\AppData\\Local\\Google\\Chrome\\User Data")
options.add_argument('--headless')
driver = webdriver.Chrome(chrome_options=options)
driver.get(https://youtube.com)
driver.get_screenshot_as_file("screenshot.png")
thing = driver.find_element_by_xpath(
'someXpath')
thing.click()
It works perfectly without the line:
options.add_argument('--headless')
However, when "headless" is added as an option, the screenshot taken by driver.get_screenshot_as_file shows that chrome is prompting a login to a google account, indicating that the specified chrome profile is not being used.
If possible, I'd like to use both headless mode and use my chrome profile at the same time. Thanks for any help.

How to get this Selenium script to login without being detected?

Before this gets taken down as a duplicate, I have already tried all of the options available here: Is there a version of Selenium WebDriver that is not detectable?.
I am trying to create a Selenium bot in Python which can successfully login without getting blocked due to bot detection. My current script is below and I have created a test account for anyone to access and help me get this working. The correct answer here would ideally be a working version of this script that I can use.
Username: TestAccount
Password: StackOverflowPass1
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
import time
##### open depop 'likes' page #####
url = "https://www.depop.com/login/"
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(url)
driver.implicitly_wait(2)
driver.get(url)
time.sleep(2)
accept_cookies = driver.find_element_by_xpath('/html/body/div[1]/div/div[3]/div[2]/button[2]')
accept_cookies.click()
time.sleep(2)
username = driver.find_element_by_xpath('/html/body/div[1]/div/div/div[1]/form/div[1]/input')
username.send_keys('TestAccount1')
password = driver.find_element_by_xpath('/html/body/div[1]/div/div/div[1]/form/div[3]/div/input')
password.send_keys('StackOverflowPass1')
login_button = driver.find_element_by_xpath('/html/body/div[1]/div/div/div[1]/form/button')
login_button.click()
I have had the same issue with logging in to gmail. I could not solve it for a week. I am 99% sure that you can't bypass the bot detection. How I solved it:
I have created Chrome account to which my gmail will be logged in all the time (disabled 2FA). Then I just start chrome as that user. Bot detection was no more an issue when I was already logged into gmail. Maybe you should try the same, add this command to your driver class:
options.add_argument("user-data-dir=/Users/alexiseggermont/Library/Application Support/Google/Chrome/Default/")
This path is usable for Mac or Linux OS. Also, rename "Default" to your profile id (for me, and in most cases, it was "Profile 1")

Sign in to gmail account fails (selenium automation)

I have a Selenium service that has to login to my gmail account as the first step. This functionality was working couple of weeks ago, but suddenly the login starts to fails and i am seeing this Error in browser, tried both in Chrome and Firefox drivers in selenium -
This Error comes after the selenium service inserts the email,password and clicks on the sign in button. A similar error was also reported in Google support Forum here- https://support.google.com/accounts/thread/10916318?hl=en, They said that "Google seems to have introduced automation tools detection on their login flow!" but there is no solution in this thread.
Some Other Details which might be useful-
I am not able to login manually to Google accounts in the browsers
opened by Selenium.
But I am able to login manually to these accounts in the Google
Chrome application.
Let me know if you need to take a look at the code, i will post it here.
Thanks in Advance!
Edit
Adding Sample code to refer.
public void loginGoogleAccount(String emailId, String password) throws Exception {
ChromeOptions options = new ChromeOptions();
options.addArguments("--profile-directory=Default");
options.addArguments("--whitelisted-ips");
options.addArguments("--start-maximized");
options.addArguments("--disable-extensions");
options.addArguments("--disable-plugins-discovery");
WebDriver webDriver = new ChromeDriver(options);
webDriver.navigate().to("https://accounts.google.com");
Thread.sleep(3000);
try {
WebElement email = webDriver.findElement(By.xpath("//input[#type='email']"));
email.sendKeys(emailId);
Thread.sleep(1000);
WebElement emailNext = webDriver.findElement(By.id("identifierNext"));
emailNext.click();
Thread.sleep(1000);
WebDriverWait wait = new WebDriverWait(webDriver, 60);
wait.until(ExpectedConditions.invisibilityOfElementLocated(By.id("identifierNext")));
Thread.sleep(3000);
WebElement passwordElement = webDriver.findElement(By.xpath("//input[#type='password']"));
passwordElement.sendKeys(password);
Thread.sleep(1000);
WebElement passwordNext = webDriver.findElement(By.id("passwordNext"));
passwordNext.click();
} catch (Exception e) {
LOGGER.info(String.format("No email/password field available or it is already logged in: [%s]: ",
e.getMessage()));
}
}
Toggle "Allow Less Secure App Access"
There is a setting on your account that you can toggle that may help with this. It is the "Allow Less Secure App Access". You should be able to visit the link below to toggle that setting if you are already logged into the gmail account you want to modify.
Link to change setting on google account:
https://myaccount.google.com/lesssecureapps
Further information(source):
https://support.google.com/accounts/answer/6010255
I just tried something out that worked for me after several hours of trial and error.
Adding args: ['--disable-web-security', '--user-data-dir', '--allow-running-insecure-content' ] to my config resolved the issue.
I realized later that this was not what helped me out as I tried with a different email and it didn't work. After some observations, I figured something else out and this has been tried and tested.
Using automation:
Go to https://stackoverflow.com/users/login
Select Log in with Google Strategy
Enter Google username and password
Login to Stackoverflow
Go to https://gmail.com (or whatever Google app you want to access)
After doing this consistently for like a whole day (about 24 hours), try automating your login directly to gmail (or whatever Google app you want to access) directly... I've had at least two other people do this with success.
PS - You might want to continue with the stackoverflow login until you at least get a captcha request as we all went through that phase as well.
This issue was because of the selenium chrome profile. Create a new chrome profile and logged into it with the email id with which you were facing the issue. Then Turn on sync.
With this chrome profile in place I can skip the login steps and directly do the main process. Use: Chrome Options to add newly created chrome profile as an argument.
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.setBinary("Binary path of the Chrome");
Hope this one helps you.
Check if your Chrome version is >= 79 and if so, downgrade to 78. Here's what I did (Windows):
I uninstalled "retail Chrome" which constantly kept upgrading itself to the latest version using sneaky tricks such as Google Chrome services that would check for updates in the background.
Even disabling those system services didn't help because retail Chrome also installs timer events that would re-enable said services in the middle of the night, so you'd wake up to a new version, and not even notice until things break.
I installed v78 from the "offline installer" found here, which doesn't seem to install any "helpful" auto upgrade features: https://www.neowin.net/news/google-chrome-780390470-offline-installer/
The above problem went away like magic. It appears that v79 has some anti-feature built in that calls home with information that allows Google to conclude that a bot is at work.
Hope this works for you... if not, you could invest a lot of time and create your own "Chrome simulator" by patching and compiling Chromium accordingly...
The only one solution which worked for me, just create fresh random google account, and that's it. Worked without any problems.
I fixed it by using selenium-stealth module and editing chromedriver.exe.
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
import time
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium_stealth import stealth
chrome_options = Options()
chrome_options.add_experimental_option("useAutomationExtension", False)
chrome_options.add_experimental_option("excludeSwitches",["enable-automation"])
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument('--disable-logging')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument('--disable-blink-features=AutomationControlled')
caps = DesiredCapabilities.CHROME
caps['goog:loggingPrefs'] = {'performance': 'ALL'}
# for editing chromedriver exe so that its not detected(run only once)
with open("chromedriver.exe","rb") as d:
file_data=d.read()
file_data=file_data.replace(b"cdc_",b"tch_")
with open("chromedriver.exe","wb") as d:
d.write(file_data)
driver = webdriver.Chrome('chromedriver.exe',chrome_options=chrome_options,desired_capabilities=caps)
# for injecting js to that tab so that browser not detected
stealth(driver,languages=["en-US", "en"],vendor="Google Inc.",platform="Win32",webgl_vendor="Intel Inc.",renderer="Intel Iris OpenGL Engine",fix_hairline=True,)
driver.get("https://accounts.google.com")
time.sleep(3)
driver.switch_to.active_element.send_keys("myemail#gmail.com\n")
time.sleep(3)
driver.switch_to.active_element.send_keys("mypassword\n")
This works for me:
from selenium import webdriver
from time import sleep
username=raw_input("username: ")
password=raw_input("password: ")
driver=webdriver.Chrome('...')
driver.get('https://stackoverflow.com/users/signup')
sleep(3)
driver.find_element_by_xpath('//*[#id="openid-buttons"]/button[1]').click()
driver.find_element_by_id('identifierId').send_keys(username)
driver.find_element_by_id('identifierNext').click()
sleep(3)
driver.find_element_by_name('password').send_keys(password)
driver.find_element_by_id('passwordNext').click()
sleep(2)
driver.get('https://mail.google.com/mail/u/0/#inbox')
I found a similar solution here.

How to Handle Browser Authentication Popup(Corporate web policy) using Selenium Webdriver in Google Chrome?

Does anyone know how to handle the browser authentication popup(corporate web policy) using selenium webdriver in google chrome?
Note: I am using Linux-Cent OS 7.
I tried using the "http://username:password#example.com/yourpage", and automating using AutoIT, but since it is linux OS, AutoIt doesnt work.
No error logs are generated.
Details are as follows:
OS: Linux - Cent OS-7.
Google Chrome version: 62
Selenium-Java version: 3.9.0
Thanks.
I still don't know how to handle the browser authentication popup with Chrome (In my case, I am using Chrome 63).
I could solve it with Firefox:
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get('https://<host>:<port>/<something>')
alert = driver.switch_to.alert
alert.send_keys('<username>' + Keys.TAB + '<password>')
alert.accept()
time.sleep(5)
print(driver.title)
driver.quit()