Selenium Google Login Blocked in Automation - selenium

As of today, a user cannot login to Google account using selenium in a new profile. I found that, Google is blocking the process(rejecting?) even on trying with stackauth. (Experienced this after updating to v90).
This is the answer that I'd posted previously for Google login using OAuth and that was working till very recently!
In short, you'll be logging in in-directly via stackauth.
The only way that I could do to bypass the restrictions is by disabling the Secure-App-Access or adding the below given argument.(Which I don't prefer as I cannot convince my users(100+) who use my app to disable that!)
options.add_argument('user-data-dir=C:/Users/{username}/path to data of browser/')
The other sole way to login is by using stealth to fake the user agent to DN which is mentioned here and it works pretty good.
The major disadvantage that I found was you cannot open another tab when the automation is running, else, the process is interrupted. But this works perfectly with that disadvantage.
But the disadvantage that I found was, once if you login, you cannot get your job done, as the website that you're visiting restricts you and forces you to update the browser in order to access the website(Google Meet in my case).
On the other hand, theoritically, one can open up the automation with the user data, but in the new window. And I feel its pretty optimal when compared to others except OAuth as it was the best way to do it.
Any other optimal working suggestions to bypass these restrictions by Google?

Finally, I was able to bypass Google security restrictions in Selenium successfully and hope it helps you as well. Sharing the entire code here.
In short:
You need to use old/outdated user-agent and revert back.
In detail:
Use selenium-stealth for faking the user agent.
Set user-agent to DN initially, before login.
Then, after logging in, revert back to normal.(not really, but chrome v>80)
That's it.
No need to keep the user data, enable less secure app access, nothing!
Here's the snippet of my code that currently works adn it's quite long tho!.(comments included for better understanding).
# Import required packages, modules etc.. Selenium is a must!
def login(username, password): # Logs in the user
driver.get("https://stackoverflow.com/users/login")
WebDriverWait(driver, 60).until(expected_conditions.presence_of_element_located(
(By.XPATH, '//*[#id="openid-buttons"]/button[1]'))).click()
try:
WebDriverWait(driver, 60).until(expected_conditions.presence_of_element_located(
(By.ID, "Email"))).send_keys(username) # Enters username
except TimeoutException:
del username
driver.quit()
WebDriverWait(driver, 60).until(expected_conditions.element_to_be_clickable(
(By.XPATH, "/html/body/div/div[2]/div[2]/div[1]/form/div/div/input"))).click() # Clicks NEXT
time.sleep(0.5)
try:
try:
WebDriverWait(driver, 60).until(expected_conditions.presence_of_element_located(
(By.ID, "password"))).send_keys(password) # Enters decoded Password
except TimeoutException:
driver.quit()
WebDriverWait(driver, 5).until(expected_conditions.element_to_be_clickable(
(By.ID, "submit"))).click() # Clicks on Sign-in
except TimeoutException or NoSuchElementException:
print('\nUsername/Password seems to be incorrect, please re-check\nand Re-Run the program.')
del username, password
driver.quit()
try:
WebDriverWait(driver, 60).until(lambda webpage: "https://stackoverflow.com/" in webpage.current_url)
print('\nLogin Successful!\n')
except TimeoutException:
print('\nUsername/Password seems to be incorrect, please re-check\nand Re-Run the program.')
del username, password
driver.quit()
USERNAME = input("User Name : ")
PASSWORD = white_password(prompt="Password : ") # A custom function for secured password input, explained at end.
# Expected and required arguments added here.
options = Options()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
options.add_experimental_option('excludeSwitches', ['enable-logging'])
# Assign drivers here.
stealth(driver,
user_agent='DN',
languages=["en-US", "en"],
vendor="Google Inc.",
platform="Win32",
webgl_vendor="Intel Inc.",
renderer="Intel Iris OpenGL Engine",
fix_hairline=True,
) # Before Login, using stealth
login(USERNAME, PASSWORD) # Call login function/method
stealth(driver,
user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.72 Safari/537.36',
languages=["en-US", "en"],
vendor="Google Inc.",
platform="Win32",
webgl_vendor="Intel Inc.",
renderer="Intel Iris OpenGL Engine",
fix_hairline=True,
) # After logging in, revert back user agent to normal.
# Redirecting to Google Meet Web-Page
time.sleep(2)
driver.execute_script("window.open('https://the website that you wanto to go.')")
driver.switch_to.window(driver.window_handles[1]) # Redirecting to required from stackoverflow after logging in
driver.switch_to.window(driver.window_handles[0]) # This switches to stackoverflow website
driver.close() # This closes the stackoverflow website
driver.switch_to.window(driver.window_handles[0]) # Focuses on present website
Click here learn about white_password.

Do this:
Install this python module
pip install selenium-stealth
Add this to your code:
from selenium_stealth import stealth
stealth(driver,
languages=["en-US", "en"],
vendor="Google Inc.",
platform="Win32",
webgl_vendor="Intel Inc.",
renderer="Intel Iris OpenGL Engine",
fix_hairline=True,
)
This worked for me.

Related

Why is Selenium throwing an unable to locate element for LKDN Sales Nav Login

I'm running the code below and keep getting an "Unable to locate Element" response, even though I'm using the correct element ID. Any feedback would be really appreciated.
from selenium import webdriver
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import time
#LOGGING INTO LINKEDIN
driver = webdriver.Chrome("C:\webdriver\chromedriver.exe")
# This instance will be used to log into LinkedIn
# Opening linkedIn's login page
driver.get("https://www.linkedin.com/sales/login")
# waiting for the page to load
# entering username
username = driver.find_element("id","username")
# In case of an error, try changing the element
# tag used here.
# Enter Your Email Address
username.send_keys("*****")
# entering password
pword = driver.find_element("id","password")
# In case of an error, try changing the element
# tag used here.
# Enter Your Password
pword.send_keys("******")
# Clicking on the log in button
# Format (syntax) of writing XPath -->
# //tagname[#attribute='value']
driver.find_element("xpath","//button[#type='submit']").click()
# In case of an error, try changing the
# XPath used here
I'm trying to login to LinkedIn Sales Navigator with Selenium, but I can't get it to work properly. I keep getting an unable to locate element error.
The login fields are inside an iframe, so it is first necessary to switch to that:
iframe = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CSS_SELECTOR, "iframe.authentication-iframe")))
driver.switch_to.frame(iframe)
username = driver.find_element(By.ID,"username")
username.send_keys("*****")
# entering password
pword = driver.find_element(By.ID,"password")
pword.send_keys("******")
driver.find_element(By.XPATH,"//button[#type='submit']").click()
driver.switch_to.default_content()
# Allow time to see impact before closing
time.sleep(5)

Unable to set a cookie of Github using Selenium Webdriver

I tried to set a cookie for GitHub using Selenium, but it always failed. After deeper analysis, I found that it was throwing an exception when setting a cookie with the name __Host-user_session_same_site. This seems very strange and I would like to know the reason for this phenomenon.
from selenium import webdriver
from selenium.webdriver.edge.options import Options
from selenium.webdriver.edge.service import Service
import json
import time
driveroptions = Options()
driveroptions.use_chromium = True
driveroptions.add_argument('–start-maximized')
driveroptions.binary_location = r'C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe'
service = Service(
executable_path=r'C:\Program Files (x86)\Microsoft\Edge\Application\msedgedriver.exe')
driver = webdriver.Edge(options=driveroptions, service=service)
driver.set_page_load_timeout(60)
driver.implicitly_wait(3)
driver.get("https://github.com")
driver.maximize_window()
driver.delete_all_cookies()
with open('cookies.txt', 'r') as f:
cookies_list = json.load(f)
for cookie in cookies_list:
cookie['expiry'] = int(time.time() + 10000)
new_cookie = {k: cookie[k] for k in {'name', 'value', 'domain', 'path', 'expiry'}}
# if cookie['name'] == '__Host-user_session_same_site':
# continue
driver.add_cookie(new_cookie)
Before that, the cookies.txt was exported using f.write(json.dumps(driver.get_cookies())) after I logged into Github. If I turn on the commented code above, everything works fine. Otherwise, the program will throw an exception: selenium.common.exceptions.UnableToSetCookieException: Message: unable to set cookie. I don't quite understand what is so special about cookies with this name (__Host-user_session_same_site).
My runtime environment information is as follows.
MicrosoftEdge=103.0.1264.62
MsEdgeDriver=103.0.1264.62
I would be very grateful if I could get your help.
This cookie is set to ensure that browsers that support SameSite cookies can check to see if a request originates from GitHub.
You will find only this cookie have a different value of Strict in the sameSite attribute, others being Lax. So when you skip this cookie, everything works fine. You can set this cookie separately by adding this code:
driver.add_cookie({'name':'__Host-user_session_same_site','value': 'itsValue','sameSite':'Strict'})

Selenium WebDriver connection to Kameleo browser

So I have figured out how to get started and open a Kameleo browser profile using Python. However, I find the Session ID and Port the chrome browser was started with. I think I have this, but my session ID is throwing an error.
I was expecting the /profiles/{guid}/start endpoint would return a JSON dictionary with the session id and port, also would be nice to have that under the profiles/{guid}/status http calls. I couldn't find it in the swaggerhub documentation.
This is the code I'm using
from kameleo.local_api_client.kameleo_local_api_client import KameleoLocalApiClient
from kameleo.local_api_client.builder_for_create_profile import BuilderForCreateProfile
client = KameleoLocalApiClient()
base_profiles = client.search_base_profiles(
device_type='desktop',
browser_product='chrome'
)
# Create a new profile with recommended settings
# for browser fingerprinting protection
create_profile_request = BuilderForCreateProfile \
.for_base_profile(base_profiles[0].id) \
.set_recommended_defaults() \
.build()
profile = client.create_profile(body=create_profile_request)
# Start the browser
client.start_profile(profile.id)
According to the documentation you don't need to get the port and the sessionID manually as you can make the connection to the browser through Kameleo.CLI.exe port.
If you keep on reading the README you will find an example where they showcase the W3C WebDriver connection.
# Connect to the running browser instance using WebDriver
options = webdriver.ChromeOptions()
options.add_experimental_option("kameleo:profileId", profile.id)
driver = webdriver.Remote(
command_executor=f'{kameleoBaseUrl}/webdriver',
options=options
)
# Use any WebDriver command to drive the browser
# and enjoy full protection from Selenium detection methods
driver.get('https://google.com')
I could also find this code in Kameleo's example repository.

Webdriver(Selenium2) - How to make selenium operate elements without wating for connecting to external AD links?

Environment:
- Selenium 2.39 Standalone Server
- PHP 5.4.11
- PHPUnit 3.7.28
- Chrome V31 & ChromeDriver v2.7
I'm testing a website,which invokes a lot of Advertisement Systems,such as Google AD.
The browser takes a lot of time to connect to external AD links , even all the elements of the page has already been loaded.
If my internet network was not fast when I ran my tests on a webpage,
Selenium would wait for a very long time ,since the AD links responsed slowly.
Under this condition ,Selenium usually waits for over 60 seconds, and throws a timeout exception.
I'm not sure how Senelium works, but it seems that Selenium has to wait for a sign of webpage's full loading, then pulls the DOM to find elements.
I want to make selenium operate elements without waiting for connectiong to external AD links.
Is there a way to do that ? Thank you very much.
I would suggest that you could make use of a proxy. Browsermob integrates well with selenium, very easy to use it:
// start the proxy
ProxyServer server = new ProxyServer(4444);
server.start();
// get the Selenium proxy object
Proxy proxy = server.seleniumProxy();
// This line will automatically return http.200 for any request going to google analytics
server.blacklistRequests("https?://.*\\.google-analytics\\.com/.*", 200);
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, proxy);
// start the browser up
WebDriver driver = new FirefoxDriver(capabilities);
I'm not sure how Senelium works, but it seems that Selenium has to
wait for a sign of webpage's full loading, then pulls the DOM to find
elements.
It is pretty much like this. The default loading strategy is "NORMAL" which means:
NORMAL of type DOMString
The remote end MUST wait until the "document.readyState" of the frame currently handling commands equals "complete", or there are no
more outstanding network requests other than XMLHttpRequests.
I finally found a simple solution for my condition.
I decide to block these Ad requests and tried some firewall and proxy softwares,for example,
comodo,privatefirewall, etc.
comodo is too heavy and complex ,privatefirewall doesn't support wildcards, and firewall would interrupt tests. At last I choosed a proxy software CCproxy. Trial Version is enough.
I create a rule for localhost ,to make it can request my test website domain only, and all other requests are rejected.
Running a test costs about 1-2 minutes before and only 30 seconds now ,it's apparently more stable and fast without connecting to the useless Ad links.
Here're configuration steps:
1.launch CCproxy with Administor privilege( you should set it using Adminisrator in the file property)
2.click Options, select AutoStartup,select AutoDetected for Local IP Address. click OK.
3.create a txt file ,input your domains,like " *.rong360.com*;*.rong360.*; "
4.click Account, select PermitOnly for Permit Category;
click New, input 127.0.0.1 for IP Address/Range;
select WebFilter,click the E button at right side to create a filter;
click the ... button,select the text file you create at Step3,
select PermittedSites. click OK
click OK.
5.click OK to return to the main UI of CCproxy.
6.launch IE and config the local proxy with 127.0.0.1:808
other browsers will use this config automatically too.
now you can run the tests again , you'll feel better if have same condition :)

Unable to set proxy :selenium.common.exceptions.WebDriverException:Access Denied

I want to use Selenium Webdriver and I am unable to do so because when I run my code, I get the following exception.
My code is very basic and as follows.
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://www.google.com.bh")
assert "Google" in driver.title
driver.close()
Exception Message
selenium.common.exceptions.WebDriverException: Message: '<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD>\n<BODY>\n<FONT face="Helvetica">\n<big><strong></strong></big><BR>\n</FONT>\n<blockquote>\n<TABLE border=0 cellPadding=1 width="80%">\n<TR><TD>\n<FONT face="Helvetica">\n<big>Access Denied (authentication_failed)</big>\n<BR>\n<BR>\n</FONT>\n</TD></TR>\n<TR><TD>\n<FONT face="Helvetica">\nYour credentials could not be authenticated: "Credentials are missing.". You will not be permitted access until your credentials can be verified.\n</FONT>\n</TD></TR>\n<TR><TD>\n<FONT face="Helvetica">\nThis is typically caused by an incorrect username and/or password, but could also be caused by network problems.\n</FONT>\n</TD></TR>\n<TR><TD>\n<FONT face="Helvetica" SIZE=2>\n<BR>\nFor assistance, contact your network support team.\n</FONT>\n</TD></TR>\n</TABLE>\n</blockquote>\n</FONT>\n</BODY></HTML>\n'
It opens firefox but after that it is unable to connect to google or any other local sites.
The exception is at driver = webdriver.Firefox()
I googled around and I followed the link on SO.
But unfortunately I still get the same error.
I cannot run as the root user. I changed my proxy settings and set No Proxy element for localhost as well as mentioned in the link.
I am using Python 2.7 and have installed selenium 2.31 version.
I also tried setting proxy.
myProxy = "*********:8080"
proxy = Proxy({
'proxyType': ProxyType.MANUAL,
'httpProxy': myProxy,
'ftpProxy': myProxy,
'sslProxy': myProxy,
'noProxy': 'localhost,127.0.0.1,*.abc'
})
driver = webdriver.Firefox(proxy=proxy)
I also tried to set the proxy to system's proxy i.e., in the above code, 'proxyType': ProxyType.SYSTEM
But it again gives the above exception message.
is there a place where I have to set my username and password?
Any help would be appreciated!
Remove proxy settings from all the browsers on the system manually. I had IE, Firefox and Google Chrome.
When I removed the proxy settings of all the browsers and enabled the proxy only on Firefox,it worked without giving any errors. I do not know the exact reason why this works like this, may be got to do with the registry settings on windows which I am not sure about.
After doing the above said, I ran the basic code and it worked fine.
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://www.google.com.bh")
assert "Google" in driver.title
driver.close()
I didn't set the proxy explicitly also. By default, it had taken the system's proxy settings. Hope this would help others facing similar issue.