Unable to set a cookie of Github using Selenium Webdriver - selenium

I tried to set a cookie for GitHub using Selenium, but it always failed. After deeper analysis, I found that it was throwing an exception when setting a cookie with the name __Host-user_session_same_site. This seems very strange and I would like to know the reason for this phenomenon.
from selenium import webdriver
from selenium.webdriver.edge.options import Options
from selenium.webdriver.edge.service import Service
import json
import time
driveroptions = Options()
driveroptions.use_chromium = True
driveroptions.add_argument('–start-maximized')
driveroptions.binary_location = r'C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe'
service = Service(
executable_path=r'C:\Program Files (x86)\Microsoft\Edge\Application\msedgedriver.exe')
driver = webdriver.Edge(options=driveroptions, service=service)
driver.set_page_load_timeout(60)
driver.implicitly_wait(3)
driver.get("https://github.com")
driver.maximize_window()
driver.delete_all_cookies()
with open('cookies.txt', 'r') as f:
cookies_list = json.load(f)
for cookie in cookies_list:
cookie['expiry'] = int(time.time() + 10000)
new_cookie = {k: cookie[k] for k in {'name', 'value', 'domain', 'path', 'expiry'}}
# if cookie['name'] == '__Host-user_session_same_site':
# continue
driver.add_cookie(new_cookie)
Before that, the cookies.txt was exported using f.write(json.dumps(driver.get_cookies())) after I logged into Github. If I turn on the commented code above, everything works fine. Otherwise, the program will throw an exception: selenium.common.exceptions.UnableToSetCookieException: Message: unable to set cookie. I don't quite understand what is so special about cookies with this name (__Host-user_session_same_site).
My runtime environment information is as follows.
MicrosoftEdge=103.0.1264.62
MsEdgeDriver=103.0.1264.62
I would be very grateful if I could get your help.

This cookie is set to ensure that browsers that support SameSite cookies can check to see if a request originates from GitHub.
You will find only this cookie have a different value of Strict in the sameSite attribute, others being Lax. So when you skip this cookie, everything works fine. You can set this cookie separately by adding this code:
driver.add_cookie({'name':'__Host-user_session_same_site','value': 'itsValue','sameSite':'Strict'})

Related

Selenium WebDriver connection to Kameleo browser

So I have figured out how to get started and open a Kameleo browser profile using Python. However, I find the Session ID and Port the chrome browser was started with. I think I have this, but my session ID is throwing an error.
I was expecting the /profiles/{guid}/start endpoint would return a JSON dictionary with the session id and port, also would be nice to have that under the profiles/{guid}/status http calls. I couldn't find it in the swaggerhub documentation.
This is the code I'm using
from kameleo.local_api_client.kameleo_local_api_client import KameleoLocalApiClient
from kameleo.local_api_client.builder_for_create_profile import BuilderForCreateProfile
client = KameleoLocalApiClient()
base_profiles = client.search_base_profiles(
device_type='desktop',
browser_product='chrome'
)
# Create a new profile with recommended settings
# for browser fingerprinting protection
create_profile_request = BuilderForCreateProfile \
.for_base_profile(base_profiles[0].id) \
.set_recommended_defaults() \
.build()
profile = client.create_profile(body=create_profile_request)
# Start the browser
client.start_profile(profile.id)
According to the documentation you don't need to get the port and the sessionID manually as you can make the connection to the browser through Kameleo.CLI.exe port.
If you keep on reading the README you will find an example where they showcase the W3C WebDriver connection.
# Connect to the running browser instance using WebDriver
options = webdriver.ChromeOptions()
options.add_experimental_option("kameleo:profileId", profile.id)
driver = webdriver.Remote(
command_executor=f'{kameleoBaseUrl}/webdriver',
options=options
)
# Use any WebDriver command to drive the browser
# and enjoy full protection from Selenium detection methods
driver.get('https://google.com')
I could also find this code in Kameleo's example repository.

How to find a browser's command_executor string in Heroku

I've run into the following problem while attempting to emulate a mobile browser with chrome on heroku: Selenium ERR_NAME_NOT_RESOLVED on heroku with mobile testing. Based on Josh Lewis's answer, and the python code sample at https://sites.google.com/a/chromium.org/chromedriver/mobile-emulation
from selenium import webdriver
mobile_emulation = { "deviceName": "Nexus 5" }
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("mobileEmulation", mobile_emulation)
driver = webdriver.Remote(command_executor='http://127.0.0.1:4444/wd/hub',
desired_capabilities = chrome_options.to_capabilities())
It may be that I need to use a command_executor (defined at https://seleniumhq.github.io/selenium/docs/api/py/webdriver_remote/selenium.webdriver.remote.webdriver.html) as
command_executor - Either a string representing URL of the remote server or a
custom
remote_connection.RemoteConnection object. Defaults to
‘http://127.0.0.1:4444/wd/hub'
to get my code working. How would I found this with the Heroku chrome buildpack, https://github.com/heroku/heroku-buildpack-google-chrome
edit:
You can download my project for both windows and heroku use at:
https://github.com/kc1/mobiletest
(keep in mind that if you deploy to heroku you have to set FLASK_CONFIG to production. )

How to interact with Developer tools in chrome using selenium or any other library

I am trying to read or interact with network and console tabs of developer tools in chrome. Could you please guide me how to achieve this.
Thanks
The short answer is no.
How to open Chrome Developer console in Selenium WebDriver using JAVA. As the provided link states you cannot directly access the chrome developer tools.
But if you are interested in access the contents of the browser console and network tab, selenium provides you a way.
System.setProperty("webdriver.chrome.driver", getChromeDriverLocation());
LoggingPreferences loggingprefs = new LoggingPreferences();
loggingprefs.enable(LogType.BROWSER, Level.WARNING);
loggingprefs.enable(LogType.PERFORMANCE, Level.WARNING);
DesiredCapabilities capabilities = DesiredCapabilities.chrome();
capabilities.setCapability(CapabilityType.LOGGING_PREFS, loggingprefs);
driver = new ChromeDriver(capabilities);
You can then print the logs as needed
LogEntries logEntries = SeleniumBaseTest.getWebDriver().manage().logs()
.get(org.openqa.selenium.logging.LogType.BROWSER);
for (LogEntry entry : logEntries) {
System.out.println((String.format("%s %s %s\n", new Date(entry.getTimestamp()), entry.getLevel(),
entry.getMessage())));
}
LogType.BROWSER will give you the browser console.
Logtype.PERFROMANCE will give you the network tab.
Other ways to access the network tab is to use a browser proxy to record the transactions.
http://www.seleniumeasy.com/selenium-tutorials/browsermob-proxy-selenium-example
In Python, Pychrome is working fine as an interface to DevTools Protocol Viewer.
Below is an example I used with a mix of Selenium for the main request and Pychrome as I wanted to get Images without downloading them twice...
import base64
import pychrome
def save_image(file_content, file_name):
try:
file_content=base64.b64decode(file_content)
with open("C:\\Crawler\\temp\\" + file_name,"wb") as f:
f.write(file_content)
except Exception as e:
print(str(e))
def response_received(requestId, loaderId, timestamp, type, response, frameId):
if type == 'Image':
url = response.get('url')
print(f"Image loaded: {url}")
response_body = tab.Network.getResponseBody(requestId=requestId)
file_name = url.split('/')[-1].split('?')[0]
if file_name:
save_image(response_body['body'], file_name)
tab.Network.responseReceived = response_received
# start the tab
tab.start()
# call method
tab.Network.enable()
# get request to target the site selenium
driver.get("https://www.realtor.com/ads/forsale/TMAI112283AAAA")
# wait for loading
tab.wait(50)

Getting CORS headers in a Flask 500 error

I've got a Flask app set up like this
from flask import Flask
from flask.ext.cors import CORS
app = Flask( . . . )
app.debug = True
CORS(app, allow_headers='Content-Type')
CORS works correctly for routes that complete properly. However, if an exception is raised, the debug output is generated without CORS headers. This means I cannot see the debug output in chrome. Can I fix this?
The issue has to do with setting debug mode. You can see the discussion about this in this GitHub issue.
There are two ways you could get around this. The easiest is to set the PROPAGATE_EXCEPTIONS configuration parameter to False, but this deprives you of the debug-mode stack trace page:
app.config['PROPAGATE_EXCEPTIONS'] = False
The second would be to write your own exception handler. I haven't tried this myself, but the flask source can give you guidance.

Unable to set proxy :selenium.common.exceptions.WebDriverException:Access Denied

I want to use Selenium Webdriver and I am unable to do so because when I run my code, I get the following exception.
My code is very basic and as follows.
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://www.google.com.bh")
assert "Google" in driver.title
driver.close()
Exception Message
selenium.common.exceptions.WebDriverException: Message: '<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD>\n<BODY>\n<FONT face="Helvetica">\n<big><strong></strong></big><BR>\n</FONT>\n<blockquote>\n<TABLE border=0 cellPadding=1 width="80%">\n<TR><TD>\n<FONT face="Helvetica">\n<big>Access Denied (authentication_failed)</big>\n<BR>\n<BR>\n</FONT>\n</TD></TR>\n<TR><TD>\n<FONT face="Helvetica">\nYour credentials could not be authenticated: "Credentials are missing.". You will not be permitted access until your credentials can be verified.\n</FONT>\n</TD></TR>\n<TR><TD>\n<FONT face="Helvetica">\nThis is typically caused by an incorrect username and/or password, but could also be caused by network problems.\n</FONT>\n</TD></TR>\n<TR><TD>\n<FONT face="Helvetica" SIZE=2>\n<BR>\nFor assistance, contact your network support team.\n</FONT>\n</TD></TR>\n</TABLE>\n</blockquote>\n</FONT>\n</BODY></HTML>\n'
It opens firefox but after that it is unable to connect to google or any other local sites.
The exception is at driver = webdriver.Firefox()
I googled around and I followed the link on SO.
But unfortunately I still get the same error.
I cannot run as the root user. I changed my proxy settings and set No Proxy element for localhost as well as mentioned in the link.
I am using Python 2.7 and have installed selenium 2.31 version.
I also tried setting proxy.
myProxy = "*********:8080"
proxy = Proxy({
'proxyType': ProxyType.MANUAL,
'httpProxy': myProxy,
'ftpProxy': myProxy,
'sslProxy': myProxy,
'noProxy': 'localhost,127.0.0.1,*.abc'
})
driver = webdriver.Firefox(proxy=proxy)
I also tried to set the proxy to system's proxy i.e., in the above code, 'proxyType': ProxyType.SYSTEM
But it again gives the above exception message.
is there a place where I have to set my username and password?
Any help would be appreciated!
Remove proxy settings from all the browsers on the system manually. I had IE, Firefox and Google Chrome.
When I removed the proxy settings of all the browsers and enabled the proxy only on Firefox,it worked without giving any errors. I do not know the exact reason why this works like this, may be got to do with the registry settings on windows which I am not sure about.
After doing the above said, I ran the basic code and it worked fine.
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://www.google.com.bh")
assert "Google" in driver.title
driver.close()
I didn't set the proxy explicitly also. By default, it had taken the system's proxy settings. Hope this would help others facing similar issue.