I've run into the following problem while attempting to emulate a mobile browser with chrome on heroku: Selenium ERR_NAME_NOT_RESOLVED on heroku with mobile testing. Based on Josh Lewis's answer, and the python code sample at https://sites.google.com/a/chromium.org/chromedriver/mobile-emulation
from selenium import webdriver
mobile_emulation = { "deviceName": "Nexus 5" }
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("mobileEmulation", mobile_emulation)
driver = webdriver.Remote(command_executor='http://127.0.0.1:4444/wd/hub',
desired_capabilities = chrome_options.to_capabilities())
It may be that I need to use a command_executor (defined at https://seleniumhq.github.io/selenium/docs/api/py/webdriver_remote/selenium.webdriver.remote.webdriver.html) as
command_executor - Either a string representing URL of the remote server or a
custom
remote_connection.RemoteConnection object. Defaults to
‘http://127.0.0.1:4444/wd/hub'
to get my code working. How would I found this with the Heroku chrome buildpack, https://github.com/heroku/heroku-buildpack-google-chrome
edit:
You can download my project for both windows and heroku use at:
https://github.com/kc1/mobiletest
(keep in mind that if you deploy to heroku you have to set FLASK_CONFIG to production. )
Related
I have Integrated the Selenium with JMeter to perform Performance Test, For this I have used Firefox driver Cofig and gecko driver. Script is executing successfully in local window machine But when trying to execute it on Linux server it giving the following error.
"The proxy moz-proxy://host:port is requesting a username and password. The site says: “LDAP”:"
Note:
gecko driver is compatible with Firefox version.
username and password passed while setting proxy on server, and also through JMeter command.
It looks like you're behind a corporate proxy server which requires authentication and unfortunately this is not something you can bypass using Firefox Driver Config
You will need to switch to JSR223 Sampler with Groovy language and instantiate the browser manually providing the proxy host/port/username/password/etc. Take a look at Proxy class JavaDoc
Example code:
import org.openqa.selenium.Proxy
import org.openqa.selenium.firefox.FirefoxDriver
import org.openqa.selenium.firefox.FirefoxOptions
Proxy proxy = new Proxy()
proxy.setHttpProxy("http://username:password#proxy-host:proxy.port")
proxy.setSslProxy("http://username:password#proxy-host:proxy.port")
FirefoxOptions options = new FirefoxOptions()
options.setCapability("proxy", proxy)
FirefoxDriver driver = new FirefoxDriver(options)
driver.get('http://example.com')
I am trying to set up sock5 proxy setup with firefox + python + selenium. I am having
IP and PORT of the proxy server. Even i added username and password it is giving pop up windows.
from selenium import webdriver
import time
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.proxy import *
profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference('network.proxy.ssl_port', int(PROXY_PORT))
profile.set_preference('network.proxy.ssl', PROXY_IP)
profile.set_preference("network.proxy.http", PROXY_IP)
profile.set_preference("network.proxy.http_port", int(PROXY_PORT))
profile.set_preference("network.proxy.no_proxies_on", 'localhost, 127.0.0.1')
profile.set_preference("network.proxy.socks_username", PROXY_USERNAME)
profile.set_preference("network.proxy.socks_password", PROXY_PASSWORD)
profile.update_preferences()
driver = webdriver.Firefox(firefox_profile=profile, executable_path="./geckodriver")
driver.get("https://www.whatismyip.com/")
time.sleep(10)
driver.close()
System Information:
Python 3.8.10
Ubuntu 20.04
Mozilla Firefox 99.0
That's just your proxy service, I had that same problem. I use a service called smartproxy.com. You can write your computer's IP address on the website for the authentication process. If the IP address that is running the code matches the IP address you listed on their site, there will be no pop up.
I do not know the proxy service you are using but it may have an authenticated method similar to the one I explained.
So I have figured out how to get started and open a Kameleo browser profile using Python. However, I find the Session ID and Port the chrome browser was started with. I think I have this, but my session ID is throwing an error.
I was expecting the /profiles/{guid}/start endpoint would return a JSON dictionary with the session id and port, also would be nice to have that under the profiles/{guid}/status http calls. I couldn't find it in the swaggerhub documentation.
This is the code I'm using
from kameleo.local_api_client.kameleo_local_api_client import KameleoLocalApiClient
from kameleo.local_api_client.builder_for_create_profile import BuilderForCreateProfile
client = KameleoLocalApiClient()
base_profiles = client.search_base_profiles(
device_type='desktop',
browser_product='chrome'
)
# Create a new profile with recommended settings
# for browser fingerprinting protection
create_profile_request = BuilderForCreateProfile \
.for_base_profile(base_profiles[0].id) \
.set_recommended_defaults() \
.build()
profile = client.create_profile(body=create_profile_request)
# Start the browser
client.start_profile(profile.id)
According to the documentation you don't need to get the port and the sessionID manually as you can make the connection to the browser through Kameleo.CLI.exe port.
If you keep on reading the README you will find an example where they showcase the W3C WebDriver connection.
# Connect to the running browser instance using WebDriver
options = webdriver.ChromeOptions()
options.add_experimental_option("kameleo:profileId", profile.id)
driver = webdriver.Remote(
command_executor=f'{kameleoBaseUrl}/webdriver',
options=options
)
# Use any WebDriver command to drive the browser
# and enjoy full protection from Selenium detection methods
driver.get('https://google.com')
I could also find this code in Kameleo's example repository.
I am trying to read or interact with network and console tabs of developer tools in chrome. Could you please guide me how to achieve this.
Thanks
The short answer is no.
How to open Chrome Developer console in Selenium WebDriver using JAVA. As the provided link states you cannot directly access the chrome developer tools.
But if you are interested in access the contents of the browser console and network tab, selenium provides you a way.
System.setProperty("webdriver.chrome.driver", getChromeDriverLocation());
LoggingPreferences loggingprefs = new LoggingPreferences();
loggingprefs.enable(LogType.BROWSER, Level.WARNING);
loggingprefs.enable(LogType.PERFORMANCE, Level.WARNING);
DesiredCapabilities capabilities = DesiredCapabilities.chrome();
capabilities.setCapability(CapabilityType.LOGGING_PREFS, loggingprefs);
driver = new ChromeDriver(capabilities);
You can then print the logs as needed
LogEntries logEntries = SeleniumBaseTest.getWebDriver().manage().logs()
.get(org.openqa.selenium.logging.LogType.BROWSER);
for (LogEntry entry : logEntries) {
System.out.println((String.format("%s %s %s\n", new Date(entry.getTimestamp()), entry.getLevel(),
entry.getMessage())));
}
LogType.BROWSER will give you the browser console.
Logtype.PERFROMANCE will give you the network tab.
Other ways to access the network tab is to use a browser proxy to record the transactions.
http://www.seleniumeasy.com/selenium-tutorials/browsermob-proxy-selenium-example
In Python, Pychrome is working fine as an interface to DevTools Protocol Viewer.
Below is an example I used with a mix of Selenium for the main request and Pychrome as I wanted to get Images without downloading them twice...
import base64
import pychrome
def save_image(file_content, file_name):
try:
file_content=base64.b64decode(file_content)
with open("C:\\Crawler\\temp\\" + file_name,"wb") as f:
f.write(file_content)
except Exception as e:
print(str(e))
def response_received(requestId, loaderId, timestamp, type, response, frameId):
if type == 'Image':
url = response.get('url')
print(f"Image loaded: {url}")
response_body = tab.Network.getResponseBody(requestId=requestId)
file_name = url.split('/')[-1].split('?')[0]
if file_name:
save_image(response_body['body'], file_name)
tab.Network.responseReceived = response_received
# start the tab
tab.start()
# call method
tab.Network.enable()
# get request to target the site selenium
driver.get("https://www.realtor.com/ads/forsale/TMAI112283AAAA")
# wait for loading
tab.wait(50)
I want to use Selenium Webdriver and I am unable to do so because when I run my code, I get the following exception.
My code is very basic and as follows.
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://www.google.com.bh")
assert "Google" in driver.title
driver.close()
Exception Message
selenium.common.exceptions.WebDriverException: Message: '<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD>\n<BODY>\n<FONT face="Helvetica">\n<big><strong></strong></big><BR>\n</FONT>\n<blockquote>\n<TABLE border=0 cellPadding=1 width="80%">\n<TR><TD>\n<FONT face="Helvetica">\n<big>Access Denied (authentication_failed)</big>\n<BR>\n<BR>\n</FONT>\n</TD></TR>\n<TR><TD>\n<FONT face="Helvetica">\nYour credentials could not be authenticated: "Credentials are missing.". You will not be permitted access until your credentials can be verified.\n</FONT>\n</TD></TR>\n<TR><TD>\n<FONT face="Helvetica">\nThis is typically caused by an incorrect username and/or password, but could also be caused by network problems.\n</FONT>\n</TD></TR>\n<TR><TD>\n<FONT face="Helvetica" SIZE=2>\n<BR>\nFor assistance, contact your network support team.\n</FONT>\n</TD></TR>\n</TABLE>\n</blockquote>\n</FONT>\n</BODY></HTML>\n'
It opens firefox but after that it is unable to connect to google or any other local sites.
The exception is at driver = webdriver.Firefox()
I googled around and I followed the link on SO.
But unfortunately I still get the same error.
I cannot run as the root user. I changed my proxy settings and set No Proxy element for localhost as well as mentioned in the link.
I am using Python 2.7 and have installed selenium 2.31 version.
I also tried setting proxy.
myProxy = "*********:8080"
proxy = Proxy({
'proxyType': ProxyType.MANUAL,
'httpProxy': myProxy,
'ftpProxy': myProxy,
'sslProxy': myProxy,
'noProxy': 'localhost,127.0.0.1,*.abc'
})
driver = webdriver.Firefox(proxy=proxy)
I also tried to set the proxy to system's proxy i.e., in the above code, 'proxyType': ProxyType.SYSTEM
But it again gives the above exception message.
is there a place where I have to set my username and password?
Any help would be appreciated!
Remove proxy settings from all the browsers on the system manually. I had IE, Firefox and Google Chrome.
When I removed the proxy settings of all the browsers and enabled the proxy only on Firefox,it worked without giving any errors. I do not know the exact reason why this works like this, may be got to do with the registry settings on windows which I am not sure about.
After doing the above said, I ran the basic code and it worked fine.
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://www.google.com.bh")
assert "Google" in driver.title
driver.close()
I didn't set the proxy explicitly also. By default, it had taken the system's proxy settings. Hope this would help others facing similar issue.