I am trying to Crawl this site and get the links of each Job posting. I am using the Selenium and PhantomJS to render the webpage, but I am not able to find the links in the rendered content. This is the script I am using
public static void main(String[] args) {
// TODO Auto-generated method stub
String url="https://www.paycomonline.net/v4/ats/web.php/jobs?clientkey=D25120971391831BA4315C705AA7ABF1&jpt=";
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setJavascriptEnabled(true);
capabilities.setPlatform(Platform.LINUX);
RemoteWebDriver driver = getRemoteDriver("http://localhost:9515", capabilities);
driver.manage().window().maximize();
driver.get(url);
new WebDriverWait(driver, 360)
.until(RemoteWebDriver -> ((JavascriptExecutor) RemoteWebDriver)
.executeScript("return document.readyState").equals("complete"));
byte[] content = driver.getPageSource().getBytes();
String html = new String(content);
System.out.println(html);
}
I am running PhantomJS in my local machine at port 9515.Can someone help?
I'm trying to automate the download of an XML file with Google Chrome.
I'm using:
Google Chrome v73.0.3683.75 (64 bits)
Chromedriver v73
Selenium WebDriver v3.14.0
C#
The problem comes up when a message of harmful file appears:
As I'm using Chromedriver, I can not interact with this message, so I tried to accept the download from the chrome://downloads page.
Once I open the chrome://downloads page, I click on the Keep button, but an Alert comes up again to confirm the download.
This popup is not a popup Selenium and Chromedriver can Handle with the Dismiss()/Accept()/SendKeys()/... methods. When I try to SwitchTo() it, Chromedriver crashes.
I tried to directly send the keystrokes of {TAB} and {SPACE}/{RIGHT} and {ENTER}, but Chrome seems not to catch them...
The full code is:
String currentWindow = this.Drivers[Navegador].CurrentWindowHandle;
String popupHandle = "";
((IJavaScriptExecutor)this.Drivers[Navegador]).ExecuteScript("window.open('about:blank','_blank')");
ReadOnlyCollection<String> tabs = this.Drivers[Navegador].WindowHandles;
foreach (string handle in tabs){
if (handle != currentWindow){
popupHandle = handle;
break;
}
}
this.Drivers[Navegador].SwitchTo().Window(popupHandle);
this.Drivers[Navegador].Navigate().GoToUrl("chrome://downloads");
String script = "return document.querySelector('body > downloads-manager').shadowRoot.querySelector('#downloadsList > downloads-item').shadowRoot.querySelector('#dangerous > paper-button:nth-child(2)');";
//String script = "return document.querySelector('body > downloads-manager').shadowRoot.querySelector('#downloadsList > downloads-item:nth-child(2)').shadowRoot.querySelector('#url').textContent;";
IWebElement boton = (IWebElement) ((IJavaScriptExecutor) this.Drivers[Navegador]).ExecuteScript(script);
boton.Click();
Thread.Sleep(2000);
SendKeys.Send("{TAB}{SPACE}");
Thread.Sleep(1000);
this.Drivers[Navegador].Close();
this.Drivers[Navegador].SwitchTo().Window(currentWindow);
this.Drivers[Navegador].SwitchTo().DefaultContent();
result = true;
IMPORTANT NOTE:
I tried to launch Chrome with all the flags/options/experimental_options/user_preferences/... possible and it doesn't work. These options/arguments seem to be deprecated in the latest versions of Chrome or Chromedriver.
As discussed with OP, answering the question in Java.
Encountered the same problem few months back, so this is how it worked for me, might work for you as well.
Map<String, Object> chromePreferences = new Hashtable<String, Object>();
// Below preferences will disable popup dialog while downloading the file
chromePreferences.put("profile.default_content_settings.popups", 0);
chromePreferences.put("download.prompt_for_download", "false");
// Set the customised path for downloading the file, if required
chromePreferences.put("download.default_directory", "path");
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.setExperimentalOption("prefs", chromePreferences);
DesiredCapabilities cap = DesiredCapabilities.chrome();
cap.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
cap.setCapability(ChromeOptions.CAPABILITY, chromeOptions);
//Now initiate ChromeDriver with the capabilities set above and continue with your automation
ChromeDriver chromeDriver = new ChromeDriver(cap);
I recently ran into this issue, and due to the deprecation of some of the methods in the ChromeDriver, the above solution didn't work.
After a lot of research I decided to switch to IE and explore an alternate option with the inspiration from this article - https://sqa.stackexchange.com/questions/3169/downloading-a-file-in-internet-explorer-through-selenium/3520 I came up with this solution in JAVA.
It is not as 'clean' but it worked for me.
public static void main(String[] args) throws NoAlertPresentException,InterruptedException {
System.setProperty("webdriver.ie.driver","C:\\selenium-java-3.141.59\\IEDriverServer.exe");
String url ="myfileurl";
WebDriver driver = new InternetExplorerDriver();
driver.get(url);
try {
Robot robot = new Robot();
Thread.sleep(2000);
//press alt+s key to save
robot.keyPress(KeyEvent.VK_ALT);
robot.keyPress(KeyEvent.VK_S);
robot.keyRelease(KeyEvent.VK_S);
robot.keyRelease(KeyEvent.VK_ALT);
Thread.sleep(2000);
}
catch (AWTException e) {
e.printStackTrace();
}
driver.close();
}
I'm trying integrate selenium with spring. The code below works perfect if no spring dependencies added in pom.xml. but if I add spring boot dependency like below(I didn't add cucumber spring here and I removed all springcontext xml, cusumber xml from eclipse workspace) and run same test without modifying anything, It opens a IE test window http://localhost:36359/ then error out in console like below "org.openqa.selenium.remote.SessionNotFoundException: session null does not exist (WARNING: The server did not provide any stacktrace information)".
I even tried adding all spring annotation replacing #Before method by initizing webdriver in springcontext; but still the same behavior. can someone help me to fix this issue. I'm using selenium 3.4.0 version
<parent>
<artifactId>spring-boot-starter-parent</artifactId>
<groupId>org.springframework.boot</groupId>
<version>1.4.3.RELEASE</version>
<relativePath />
</parent>
public class ScenarioOutlineStepDef {
WebDriver driver;
#Before()
public void setUp() {
System.setProperty("webdriver.ie.driver", "C:/IEDriverServer-64.exe");
DesiredCapabilities dc = new DesiredCapabilities();
dc.setCapability(InternetExplorerDriver.ENABLE_PERSISTENT_HOVERING, false);
dc.setCapability(InternetExplorerDriver.IE_ENSURE_CLEAN_SESSION, true);
dc.setCapability(InternetExplorerDriver.INTRODUCE_FLAKINESS_BY_IGNORING_SECURITY_DOMAINS, true);
dc.setCapability(InternetExplorerDriver.NATIVE_EVENTS, false);
dc.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
dc.setJavascriptEnabled(true);
dc.setBrowserName("internet explorer");
driver = new InternetExplorerDriver(dc);
}
#Given("^user navigates to Pricing Portal$")
public void goToPricingPortal() {
driver.navigate().to(
"xyz.com;
}
#When("^I enter Username as \"([^\"]*)\" and Password as \"([^\"]*)\"$")
public void I_enter_Username_as_and_Password_as(String arg1, String arg2) {
driver.findElement(By.id("txtUserDefault")).sendKeys(arg1);
driver.findElement(By.id("txtPassDefault")).sendKeys(arg2);
driver.findElement(By.cssSelector("input[type=\"submit\"]")).click();
}
#Then("^login should be unsuccessful$")
public void validateRelogin() {
if (driver.getCurrentUrl().equalsIgnoreCase(
"xyz.com")) {
System.out.println("Test Pass");
} else {
System.out.println("Test Failed");
}
// driver.close();
}
Do it manually
Set same Security level for all zones. Try this steps
Open Internet Explorer Browser
Go to menu and open Tools -> Internet Options -> Security
Set all values of zones (Internet, Local intranet, Trusted sites, Restricted sites) to the same protected mode, enabled or disabled should not matter
click on OK.
OR use DesiredCapabilities like this
DesiredCapabilities IEcaps = DesiredCapabilities.internetExplorer();
IEcaps .setCapability(
InternetExplorerDriver.INTRODUCE_FLAKINESS_BY_IGNORING_SECURITY_DOMAINS,
true);
File SrcFile= new File("E:\\Sankalp\\IE\\IEDriverServer.exe");
System.setProperty("webdriver.ie.driver", SrcFile.getAbsolutePath());
WebDriver driver = new InternetExplorerDriver(IEcaps );
driver.get("Your URL");
driver.quit();
I need to implement Headless Browser for HTTPS (validate certificate). For this I need to write extra line of code.
I have written for browser HTTP and it is working fine.
public class Headless {
public static void main(String[] args)
{
File src=new File("C:\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe");
System.setProperty("phantomjs.binary.path",src.getAbsolutePath());
WebDriver driver=new PhantomJSDriver();
driver.get("https://www.google.co.in/");
System.out.println(driver.getTitle());
}
}
I google it and get some info:-
phantomjs --ignore-ssl-errors=yes;
DesiredCapabilities dcap = new DesiredCapabilities();
String[] phantomArgs = new String[] {
"--ssl-protocol=any",
"--ignore-ssl-errors=true"
};
dcap.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, phantomArgs);
PhantomJSDriver driver = new PhantomJSDriver(dcap);
You can read more about the command-line options on this link: Command Line Interface | PhantomJS
I need to test a JS geolocation functionality with Selenium and I am using chromedriver to run the test on the latest Chrome.
The problem is now that Chrome prompts me to enable Geolocation during the test and that I don't know how to click that little bar on runtime, so I am desperately looking for a way to start the chromedriver and chrome with some option or trigger to enable this by default. All I could find here was however how I can disable geolocation altogether.
How can I solve this issue?
In the known issues section of the chromedriver wiki they said that you Cannot specify a custom profile
This is why it seems to me that #Sotomajor answer about using profile with Chrome as you would do with firefox will not work.
In one of my integration tests I faced the same issue. But because I was not bothering about the real geolocation values, all I had to do is to mock window.navigator.gelocation
In you java test code put this workaround to avoid Chrome geoloc permission info bar.
chromeDriver.executeScript("window.navigator.geolocation.getCurrentPosition = function(success) {
var position = {
"coords": {
"latitude": "555",
"longitude": "999"
}
};
success(position);
}");
latitude (555) and longitude (999) values here are just test value
Approach which worked for me in Firefox was to visit that site manually first, give those permissions and afterwards copy firefox profile somewhere outside and create selenium firefox instance with that profile.
So:
cp -r ~/Library/Application\ Support/Firefox/Profiles/tp3khne7.default /tmp/ff.profile
Creating FF instance:
FirefoxProfile firefoxProfile = new FirefoxProfile(new File("/tmp/ff.profile"));
FirefoxDriver driver = new FirefoxDriver(firefoxProfile);
I'm pretty sure that something similar should be applicable to Chrome. Although api of profile loading is a bit different. You can check it here: http://code.google.com/p/selenium/wiki/ChromeDriver
Here is how I did it with capybara for cucumber tests
Capybara.register_driver :selenium2 do |app|
profile = Selenium::WebDriver::Chrome::Profile.new
profile['geolocation.default_content_setting'] = 1
config = { :browser => :chrome, :profile => profile }
Capybara::Selenium::Driver.new(app, config)
end
And there is link to other usefull profile settings: pref_names.cc
Take a look at "Tweaking profile preferences" in RubyBindings
As for your initial question:
You should start Firefox manually once - and select the profile you use for Selenium.
Type about:permissions in the address line; find the name of your host - and select share location : "allow".
That's all. Now your Selenium test cases will not see that dreaded browser dialog which is not in the DOM.
I took your method but it can not working. I did not find "BaseUrls.Root.AbsoluteUri" in Chrome's Preferences config. And use a script for test
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument("proxy-server=http://127.0.0.1:1087")
prefs = {
'profile.default_content_setting_values':
{
'notifications': 1,
'geolocation': 1
},
'devtools.preferences': {
'emulation.geolocationOverride': "\"11.111698#-122.222954:\"",
},
'profile.content_settings.exceptions.geolocation':{
'BaseUrls.Root.AbsoluteUri': {
'last_modified': '13160237885099795',
'setting': '1'
}
},
'profile.geolocation.default_content_setting': 1
}
chromeOptions.add_experimental_option('prefs', prefs)
chromedriver_path = os.path.join(BASE_PATH, 'utils/driver/chromedriver')
log_path = os.path.join(BASE_PATH, 'utils/driver/test.log')
self.driver = webdriver.Chrome(executable_path=chromedriver_path, chrome_options=chromeOptions, service_log_path=log_path)
We just found a different approach that allows us to enable geolocation on chrome (currently 65.0.3325.181 (Build officiel) (64 bits)) without mocking the native javascript function.
The idea is to authorize the current site (represented by BaseUrls.Root.AbsoluteUri) to access to geolocation information.
public static void UseChromeDriver(string lang = null)
{
var options = new ChromeOptions();
options.AddArguments(
"--disable-plugins",
"--no-experiments",
"--disk-cache-dir=null");
var geolocationPref = new JObject(
new JProperty(
BaseUrls.Root.AbsoluteUri,
new JObject(
new JProperty("last_modified", "13160237885099795"),
new JProperty("setting", "1")
)
)
);
options.AddUserProfilePreference(
"content_settings.exceptions.geolocation",
geolocationPref);
WebDriver = UseDriver<ChromeDriver>(options);
}
private static TWebDriver UseDriver<TWebDriver>(DriverOptions aDriverOptions)
where TWebDriver : RemoteWebDriver
{
Guard.RequiresNotNull(aDriverOptions, nameof(UITestsContext), nameof(aDriverOptions));
var webDriver = (TWebDriver)Activator.CreateInstance(typeof(TWebDriver), aDriverOptions);
Guard.EnsuresNotNull(webDriver, nameof(UITestsContext), nameof(WebDriver));
webDriver.Manage().Window.Maximize();
webDriver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(10);
webDriver.NavigateToHome();
return webDriver;
}
ChromeOptions options = new ChromeOptions();
HashMap<String, Integer> contentSettings = new HashMap<String, Integer>();
HashMap<String, Object> profile = new HashMap<String, Object>();
HashMap<String, Object> prefs = new HashMap<String, Object>();
contentSettings.put("geolocation", 1);
contentSettings.put("notifications", 2);
profile.put("managed_default_content_settings", contentSettings);
prefs.put("profile", profile);
options.setExperimentalOption("prefs", prefs);
The simplest to set geoLocation is to just naviaget on that url and click on allow location by selenium. Here is the code for refrence
driver.navigate().to("chrome://settings/content");
driver.switchTo().frame("settings");
WebElement location= driver.findElement(By.xpath("//*[#name='location' and #value='allow']"));
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
((JavascriptExecutor) driver).executeScript("arguments[0].click();", location);
WebElement done= driver.findElement(By.xpath(""));
driver.findElement(By.xpath("//*[#id='content-settings-overlay-confirm']")).click();
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
driver.navigate().to("url");
Cypress:
I encountered with same problem but mocking was not worked for me.
My case:
Application use geo location for the login purpose. So with mock it not worked for me.
Solution:
I used below package and followed the steps and its works for me.
https://www.npmjs.com/package/cypress-visit-with-custom-geolocation
Working sample example:
https://github.com/gaikwadamolraj/cypress-visit-with-custom-geolocation