I am looking for automated way to capture HTML after I do some actions on the web page.
For example I select some item in dropdown and HTML has been changed, i want to capture that HTML and dump into file. As result, I will end up with many different HTML files on my hdd.
I was thinking it might be possible to achieve that by using Selenium, maybe some other plugin which would give me possibility so save HTML in automatic manner to file.
do you mean source code?
for Python:
driver.page_source
for Java:
driver.getPageSource();
you can run these code after each step where page is changed
I think what you are asking to do will not be that easy. There are other questions on SO (e.g. this one) asking the same thing and there aren't really any good answers. I tried googling for a few mins to find a way to do this. I would think something more like a browser plug-in would exist that would do this for you.
If I were forced to code this using Selenium I would do something like the following...
Create a script that launches the browser and navigates to the page you want to track. In a user-defined interval, the script would grab the page source and compare it to the last capture. If the source is not the same, it would diff the two pages and write the diff to disk. I'm sure there are a number of diffing libraries you could find and use.
The problems with this approach...
If you made too many changes within the defined interval, you would get a glob of changes and not be able to differentiate what action made what change.
If you make the interval too small, you may run into perf issues.
Probably the most significant issue is going to be the fact that you run several tests and then go back and look at the diffs... but you won't have any way to tell what changes correspond to what action since you can't tie the two together other than order of occurrence.
What might be cool is if you could inject a button into the page that when clicked would popup an input dialog that you could type some text into and use that as a label for the upcoming action diff. For example, you click the button and type "choose price" - OK. Now you select a price from a dropdown. The next time you click the button, the script detects the button click and does a quick diff and writes it to disk using the "choose price" label... or something like that.
I found answer for my own question.
Start a selenium chrome driver server.
Use selenium client connect to and all changes could be captured by using code example below:
Code:
WebDriver driver = new RemoteWebDriver(new URL("http://127.0.0.1:9515"), DesiredCapabilities.chrome());
driver.get("http://google.com");
By by = new By.ByTagName("div");
List<WebElement> oldDivs = driver.findElements(by);
while(true){
try {
List<WebElement> newDivs = driver.findElements(by);
if (oldDivs != newDivs) {
for (WebElement element : newDivs) {
String a = element.getAttribute("a");
String b = element.getAttribute("b");
System.out.println(a + " :" + b);
}
}
}catch (Exception e){
System.err.println(e);
}
}
Related
I was working to scrape links to articles on a website. But normally when site was loaded it list only 5 articles then it requires to click load more button to display more articles list.
Html source has only links to first five articles.
I used selenium python to automate clicking load more button to completely load webpage with all article listings.
Question is now how can i extract links to all those articles.
After loading site completely with selenium i tried to get html source with driver.page_source and printed it but still it has only link to first 5 articles.
I want to get links to all those articles that were loaded in webpage after clicking load more button.
Please someone help to provide solution.
Maybe the links take some time to show up and your code is doing driver.source_code before the source code is updated. You can select the links with Selenium after an explicit wait so that you can make sure that the links that are dinamically added to the web page are fully loaded. It is difficult to boil down exactly what you need without a link to your source, but (in Python) it should be something similar to:
from selenium.webdriver.support.ui import WebDriverWait
def condition(driver):
"""If the selector defined in the function retrieves 10 or more results, return the results.
Else, return None.
"""
selector = 'a.my_class' # Selects all <a> tags with the class "my_class"
els = driver.find_elements_by_css_selector(selector)
if len(els) >= 10:
return els
# Making an assignment only when the condition returns a truthy value when called (waiting until 2 min):
links_elements = WebDriverWait(driver, timeout=120).until(condition)
# Getting the href attribute of the links
links_href = [link.get_attribute('href') for link in links_elements]
In this code, you are:
Constantly looking for the elements you want until there are 10 or more of them. You can do this by CSS Selector (as in the example), XPath or other method. This gives you a list of Selenium objects as soon as the wait condition returns an object with a True value, until a certain timeout. See more on explicit waits in the documentation. You should make the appropriate condition for your case - maybe expecting a certain number of links is not good if you are not sure of how many links there will be in the end.
Extracting what you want from the Selenium object. For that, use the appropriate method over the elements in the list you got from the step above.
I've a test case where after creation of an item, notification is shown, but that notification disappear s within 2-3 seconds.
I want to identify the element for that notification, but when I try to inspect element for that in firebug, it's HTML snippet disappears very quickly since notification itself disappears. Hence I couldn't identify element for it and finding it very difficult to automate.
Can anyone suggest how to deal with such situation?
Since you say "I've a test case", I'm thinking you are testing an application of yours. The plainest way to get around your problem is to know your application. Even if a third-party library is providing the notification code, you can read the doc to see if you can increase the delay or you can read the source code to figure out how it creates the element and where.
If the above fails, then if you can get together a sequence of operations in Selenium that triggers a notification, you should be able to get a serialization of body quickly enough that you can then examine at your leisure. Using Selenium for Python, it would go:
print driver.execute_script("return document.body.outerHTML;")
I would run the code above with redirection to save the output of the print statement to a file that I'd then examine at my leisure. You could make it narrower if you wish by getting the outerHTML of a descendant of body. I like to have a good bit of context so that I know where exactly the element is being created. I've used libraries and configurations that create such notifications as children of body, and some that put them as children of other elements.
Open the previous page before the notification
Press "Ctrl+Shift+c", Navigate to "sources" tab
Do the manual step to generate the notification, which is going to hide in few seconds
Press "F8". Page load scripts will be paused
Then inspect the element as usual and fetch the xpath at your convenience
You can use the below JAVA code to wait for 20 seconds till any element with text 'your notification' appears in the page:
try{
WebDriverWait wait = new WebDriverWait(driver,20);
wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//*[contains(text(),'Your notification here')]")));
}catch(Throwable e){
System.err.println("Error while waiting for the notification to appear: "+ e.getMessage());
}
How to do the Automation testing of any Google map. I have a map in my project/application, now I want to click on each markers.
Since you have no experience with WebDriver, I'm going to give you the answer you want (and not the one you need which is "Go look at WebDriver manual and tutorials.").
Java example:
// opens up Chrome, but you can use any other browser
WebDriver driver = new ChromeDriver();
// goes to GMaps page and searches for "Washington"
driver.get("https://maps.google.com/maps?q=Washington");
// clicks the only marker on the page
driver.findElement(By.cssSelector("img[src*='markerTransparent.png']")).click();
// don't forget to kill the browser or else you'll have neverending chromedriver.exe processes
driver.quit();
Now, you need to take a step back, look at WebDriver, choose a language in which you want to write your tests, go through the API and some examples, then try to implement your tests and if something goes astray, feel free to post another question with a particular issue (just make sure to search for it first).
You can click on each marker by locating that marker using ID.
Here is one script which I wrote to click on marker of google maps
d = Selenium::WebDriver.for :firefox
d.get 'http://maps.google.com'
d.find_element(:id, 'gbqfq').click
d.find_element(:id, 'gbqfq').send_keys 'hdfc bank pune'
d.find_element(:id, 'gbqfb').click
d.find_element(:id, 'mtgt_J.1000').click
Previously I have been using chrome Auto Refresh plug in. However, now my code has multiple ChromeDriver instances opening and closing and I cannot use Auto Refresh. Also, it is quite a hassle to install Auto Refresh on new computers.
Is there any way to refresh driver (simulate F5 say every 15 seconds if driver does not change remains motionless) with Selenium similar to Google Auto Refresh?
refresh is a built in command.
driver = webdriver.Chrome()
driver.get("http://www.google.com")
driver.refresh()
If you don't have the chrome driver it can be found here:
https://code.google.com/p/chromedriver/downloads/list
Put the binary in the same folder as the python script you're writing. (Or add it to the path or whatever, more information here: https://code.google.com/p/selenium/wiki/ChromeDriver)
edit:
If you want to refresh ever 10 seconds or something, just wrap the refresh line with a loop and a delay. For example:
import time
while(True):
driver.refresh()
time.sleep(refresh_time_in_seconds)
If you only want to refresh if the page hasn't changed in the meantime, keep track of the page that you're on. driver.current_url is the url of the current page. So putting it all together it would be:
import time
refresh_time_in_seconds = 15
driver = webdriver.Chrome()
driver.get("http://www.google.com")
url = driver.current_url
while(True):
if url == driver.current_url:
driver.refresh()
url = driver.current_url
time.sleep(refresh_time_in_seconds)
Well there are two ways of doing this.
1. We can use refresh method
driver.get("some website url");
driver.navigate().refresh();
We can use actions class and mimic F5 press
Actions act = new Actions(driver);
act.SendKeys(Keys.F5).perform();
If you write unit tests that must be run like if you had to open/refresh a new browser session each time, you can use a method with before annotations:
#Before
public void refreshPage() {
driver.navigate().refresh();
}
If all tests are individually successful (green) but fail all together, the reason might also been that you need to wait for some resources to be available on the page, so you also need to handle it, setting the timeout like this:
public WebElement getSaveButton() {
return findDynamicElementByXPath(By.xpath("//*[#id=\"form:btnSave\"]"), 320);
}
320 is a long time, but you must make sure that you give enough time to get all that it takes to test.
I'm using Selenium to ease my testing burden and I have about 1,000 different drop down list combinations (spread across multiple pages and drop down lists) that need to be tested. Basically, what I would like to do is select each <option> inside of a <select>, click the Submit button, select an item (first, second, third, etc.) in the drop down list on the resulting page, click submit, and then go back and select the next item, in sequence. Each time, it should assert that a certain value (related to the drop down list value selected) is present on the final page. Does anybody know if this kind of logic is possible in Selenium?
I'm having a hard time explaining this, so hopefully this pseudo code clears things up
foreach option in select
select option
submit form
foreach option in select
select option
submit form
assert that page contains text that matches selected values
Edit: I have selected values from the drop down list while the recorder is playing, but it seems like the recorder isn't picking up the selected drop down list values. Nor have I been able to figure out how to perform the operation for each <option> in a <select>.
The first question I have is whether or not it's even possible. If it is, could somebody please point me in the right direction to get me started?
Edit 2: I'm not opposed to using another web automated testing utility. If anybody has any recommendations for a free alternative, please feel free to make that recommendation.
What language are using Selenium in? If you're just using Selenium by writing HTML, I'd recommend switching to a programming language and using Selenium RC -- bindings are available for a wide variety of languages, such as Java and Python. In Java, I believe the following would do what you want:
void test(Selenium browser, String startPageUrl,
String firstFormLocator, String firstSelectLocator,
String secondFormLocator, String secondSelectLocator) {
browser.open(startPageUrl);
for (String option : browser.getSelectOptions(firstSelectLocator)) {
browser.open(startPageUrl);
browser.select(firstSelectLocator, "label=" + option);
browser.submit(firstFormLocator); // Or click the submit button
for (String subOption : browser.getSelectOptions(secondSelectLocator) {
browser.open(startPageUrl);
browser.select(firstSelectLocator, "label=" + option);
browser.submit(firstFormLocator); // Or click the submit button
browser.select(secondSelectLocator, "label=" + subOption);
browser.submit(secondFormLocator); // Or click the submit button
// Do your assertions
}
}
}
The code isn't exactly readable, so it might be worth some time abstracting the page away slightly using the Page Object pattern. This also helps make the code more maintainable, for instance when you change the ID of an element, you only need to change it in the page object rather than every test.
Also bear in mind that doing this 1000 times isn't going to be quick. It might be worth seeing if you do similar testing just below the web interface to allow quicker feedback from tests, and then test the web interface is using the lower layer correctly. Also, do you really need 1000 tests? It seems that there's some redundancy in testing here -- is the 1000th test going to fail if the last 999 have passed?