Testing Service Worker and Cache in Selenium - selenium

I have an app that uses Service Workers and Cache API. The cache's name used in Service Worker contains GIT revision (hash) so that with every release, new cache is used and the old one removed. This is to make 100% sure users will always get latest sources (this case is fine for intended release schedule).
I'm using Selenium Java (WebDriver) tests that run few times a day (let's say when there is a change in GIT repo) and I'd like to make sure that the worker's cache is being correctly wiped with every new GIT revision.
By default, Selenium is creating a new temporary profile directory for each browser session, which means there is always no cache when the tests start. I'd like to use cache though, my idea is (let's talk about Firefox for now):
keep some model profile directory
when Selenium starts Firefox, pass the model profile to the new temporary profile, e.g. new FirefoxProfile(new File(PROFILE_MODEL))
this would ensure that some cache exists prior starting browser
once tests are finished, back up the temporary profile (as it contains updated cache) and replace the (now older) PROFILE_MODEL with the latest profile so that next test run would use this updated cache again
The "model profile update step" at the end of test run would look like this in pseudocode
File modelDir = new File(PROFILE_MODEL);
// copy current model for backup
bckPath = Files.createTempDirectory("bckProfile");
Util.copy(modelDir, bckPath);
// remove the now obsolete PROFILE_MODEL
Util.deleteDirectory(modelDir);
// copy current profile to the model dir
File profileSnapshot = currentProfile.layoutOnDisk();
Util.copy(profileSnapshot.toPath(), modelPath);
The problem is though that the call currentProfile.layoutOnDisk(); doesn't seem to contain any cache related information and I have no idea how to get path of the currently being used temporary profile (in Linux, typically something like /tmp/anonymous6209009552324183685webdriver-profile
So basically my question is, how to preserve and control browser (Firefox) cache across multiple test runs? I know it is often desired to start browser with fresh cache in tests but I think for this particular case of testing Service Workers it is quite useful to have some control over it.
Please note that I'm running the tests in CI server, so some manual solution could be quite difficult (but hey, at least it could point me to some direction...)
Update: I can initiate Firefox profile with
profile = new ProfilesIni().getProfile("test")
and then knowing the actual path to "test" model profile (model, not the copy created by getProfile call), I could replace all its files with the ones from calling ProfilesIni().getProfile("test") at the end of tests, but this does not seem to e.g. copy history of visited pages.
Update 2: This is how I start browser:
caps = DesiredCapabilities.firefox();
FirefoxProfile sampleProfile = new ProfilesIni().getProfile("test");
caps.setCapability(FirefoxDriver.PROFILE, sampleProfile);
browser = new FirefoxDriver(caps);
and I can see it creates 2 profile temporary folders:
one is created by the call new ProfilesIni().getProfile("test") and it won't get updated "ever" during the test
second one is created from FirefoxDriver.startClient method which calls NewProfileExtensionConnection.start which then calls FirefoxProfile.layoutOnDisk
Now during the tests, only the second one profile is being updated which I guess makes kind of sense. The problem is how to get the actual profile dir that is created from the capabilities...

You could override the FirefoxProfile.layoutOnDisk method to always use the same folder for the profile:
public class FirefoxProfileEx extends FirefoxProfile {
File profileDir;
public FirefoxProfileEx(File profileDir){
super(profileDir);
this.profileDir = profileDir;
}
public File layoutOnDisk() {
try {
File userPrefs = new File(profileDir, "user.js");
installExtensions(profileDir);
deleteLockFiles(profileDir);
deleteExtensionsCacheIfItExists(profileDir);
updateUserPrefs(userPrefs);
return profileDir;
} catch (IOException e) {
throw new UnableToCreateProfileException(e);
}
}
}
Usage:
FirefoxProfileEx profile = new FirefoxProfileEx(new File("C:\\temp\\profile"));
WebDriver driver= new FirefoxDriver(profile);

Known layoutOnDisk() will used to call current profile and we write or save to disk. We need to use this saved profile or dir mostly early as possible because once execution is completed then saved profile by using layoutOnDisk() is not useful. They don't work.
For more information please see here
So for firefox only thing is just saved all cookies by using "driver.Manage().Cookies.AllCookies;" use these in next execution.
I hope it is prefer to use named profile instead of taking profile from directory. Here the way to create profile
ProfilesIni profile = new ProfilesIni();
FirefoxProfile myprofile = profile.getProfile("myProfilewhichIsSaved");
WebDriver driver = new FirefoxDriver(myprofile);
I hope IE helps by not clearing before starting. here is link helps you
Also you can try by chrome profile
Thank You,
Murali

Related

Using Selenium with an existing session and sign in details [duplicate]

For some unknown reasons ,my browser open test pages of my remote server very slowly. So I am thinking if I can reconnect to the browser after quitting the script but don't execute webdriver.quit() this will leave the browser opened. It is probably kind of HOOK or webdriver handle.
I have looked up the selenium API doc but didn't find any function.
I'm using Chrome 62,x64,windows 7,selenium 3.8.0.
I'll be very appreciated whether the question can be solved or not.
No, you can't reconnect to the previous Web Browsing Session after you quit the script. Even if you are able to extract the Session ID, Cookies and other session attributes from the previous Browsing Context still you won't be able to pass those attributes as a HOOK to the WebDriver.
A cleaner way would be to call webdriver.quit() and then span a new Browsing Context.
Deep Dive
There had been a lot of discussions and attempts around to reconnect WebDriver to an existing running Browsing Context. In the discussion Allow webdriver to attach to a running browser Simon Stewart [Creator WebDriver] clearly mentioned:
Reconnecting to an existing Browsing Context is a browser specific feature, hence can't be implemented in a generic way.
With internet-explorer, it's possible to iterate over the open windows in the OS and find the right IE process to attach to.
firefox and google-chrome needs to be started in a specific mode and configuration, which effectively means that just
attaching to a running instance isn't technically possible.
tl; dr
webdriver.firefox.useExisting not implemented
Yes, that's actually quite easy to do.
A selenium <-> webdriver session is represented by a connection url and session_id, you just reconnect to an existing one.
Disclaimer - the approach is using selenium internal properties ("private", in a way), which may change in new releases; you'd better not use it for production code; it's better not to be used against remote SE (yours hub, or provider like BrowserStack/Sauce Labs), because of a caveat/resource drainage explained at the end.
When a webdriver instance is initiated, you need to get the before-mentioned properties; sample:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://www.google.com/')
# now Google is opened, the browser is fully functional; print the two properties
# command_executor._url (it's "private", not for a direct usage), and session_id
print(f'driver.command_executor._url: {driver.command_executor._url}')
print(f'driver.session_id: {driver.session_id}')
With those two properties now known, another instance can connect; the "trick" is to initiate a Remote driver, and provide the _url above - thus it will connect to that running selenium process:
driver2 = webdriver.Remote(command_executor=the_known_url)
# when the started selenium is a local one, the url is in the form 'http://127.0.0.1:62526'
When that is ran, you'll see a new browser window being opened.
That's because upon initiating the driver, the selenium library automatically starts a new session for it - and now you have 1 webdriver process with 2 sessions (browsers instances).
If you navigate to an url, you'll see it is executed on that new browser instance, not the one that's left from the previous start - which is not the desired behavior.
At this point, two things need to be done - a) close the current SE session ("the new one"), and b) switch this instance to the previous session:
if driver2.session_id != the_known_session_id: # this is pretty much guaranteed to be the case
driver2.close() # this closes the session's window - it is currently the only one, thus the session itself will be auto-killed, yet:
driver2.quit() # for remote connections (like ours), this deletes the session, but does not stop the SE server
# take the session that's already running
driver2.session_id = the_known_session_id
# do something with the now hijacked session:
driver.get('https://www.bing.com/')
And, that is it - you're now connected to the previous/already existing session, with all its properties (cookies, LocalStorage, etc).
By the way, you do not have to provide desired_capabilities when initiating the new remote driver - those are stored and inherited from the existing session you took over.
Caveat - having a SE process running can lead to some resource drainage in the system.
Whenever one is started and then not closed - like in the first piece of the code - it will stay there until you manually kill it. By this I mean - in Windows for example - you'll see a "chromedriver.exe" process, that you have to terminate manually once you are done with it. It cannot be closed by a driver that has connected to it as to a remote selenium process.
The reason - whenever you initiate a local browser instance, and then call its quit() method, it has 2 parts in it - the first one is to delete the session from the Selenium instance (what's done in the second code piece up there), and the other is to stop the local service (the chrome/geckodriver) - which generally works ok.
The thing is, for Remote sessions the second piece is missing - your local machine cannot control a remote process, that's the work of that remote's hub. So that 2nd part is literally a pass python statement - a no-op.
If you start too many selenium services on a remote hub, and don't have a control over it - that'll lead to resource drainage from that server. Cloud providers like BrowserStack take measures against this - they are closing services with no activity for the last 60s, etc, yet - this is something you don't want to do.
And as for local SE services - just don't forget to occasionally clean up the OS from orphaned selenium drivers you forgot about :)
OK after mixing various solutions shared on here and tweaking I have this working now as below. Script will use previously left open chrome window if present - the remote connection is perfectly able to kill the browser if needed and code functions just fine.
I would love a way to automate the getting of session_id and url for previous active session without having to write them out to a file during hte previous session for pick up...
This is my first post on here so apologies for breaking any norms
#Set manually - read/write from a file for automation
session_id = "e0137cd71ab49b111f0151c756625d31"
executor_url = "http://localhost:50491"
def attach_to_session(executor_url, session_id):
original_execute = WebDriver.execute
def new_command_execute(self, command, params=None):
if command == "newSession":
# Mock the response
return {'success': 0, 'value': None, 'sessionId': session_id}
else:
return original_execute(self, command, params)
# Patch the function before creating the driver object
WebDriver.execute = new_command_execute
driver = webdriver.Remote(command_executor=executor_url, desired_capabilities={})
driver.session_id = session_id
# Replace the patched function with original function
WebDriver.execute = original_execute
return driver
remote_session = 0
#Try to connect to the last opened session - if failing open new window
try:
driver = attach_to_session(executor_url,session_id)
driver.current_url
print(" Driver has an active window we have connected to it and running here now : ")
print(" Chrome session ID ",session_id)
print(" executor_url",executor_url)
except:
print("No Driver window open - make a new one")
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=myoptions)
session_id = driver.session_id
executor_url = driver.command_executor._url
Without getting into why do you think that leaving an open browser windows will solve the problem of being slow, you don't really need a handle to do that. Just keep running the tests without closing the session or, in other words, without calling driver.quit() as you have mentioned yourself. The question here though framework that comes with its own runner? Like Cucumber?
In any case, you must have some "setup" and "cleanup" code. So what you need to do is to ensure during the "cleanup" phase that the browser is back to its initial state. That means:
Blank page is displayed
Cookies are erased for the session

How do I clear local storage on remote webdriver session

I am running selenium on remotewebdriver that connects to browser stack to run tests on different browsers. the thing is i have three #Test in my code and at the end of each test the URL is loaded again so I land on the home page again. In Chrome, the first test passes, the second test fails as the previous selection from the first test is remembered in the local storage. Then the 3rd test passes again as the local storage is cleared. In firefox all tests are passing.
I have tried with the code i have pasted in this ticket - i have put it #AfterMethod class in my setup. it clears the local storage in all the tests except no.2 in chrome. without this code, all of the tests are failing except the first one. someone please advise and sorry if it doesn't make sense.
final RemoteExecuteMethod executeMethod = new RemoteExecuteMethod((RemoteWebDriver) driver);
final RemoteWebStorage webStorage = new RemoteWebStorage(executeMethod);
final LocalStorage storage = webStorage.getLocalStorage();
storage.removeItem("sessionState");
I am not sure how to make the code more stable. Any ideas.

Selenium Firefox Webdriver doesn't run when launched through a scheduled task with option "Run whether user is logged on or not""

I would like to execute tests using Selenium WebDriver with a variety of browsers (at least Firefox and Chrome). In order to run these tests regularly, I created a scheduled task which launches a simple command:
node screenshot.js >log.txt 2>&1
Where screenshot.js is:
var webdriver = require('selenium-webdriver'),
...
// Import drivers here
...
var driver = new webdriver.Builder()
.forBrowser('firefox')
.build();
driver.get('http://www.google.com/ncr');
driver.takeScreenshot().then(
function(image, err) {
require('fs').writeFile('out.jpg', image, 'base64', function(err) {
console.log(err);
});
}
);
driver.quit();
If I run the scheduled task with the option "Run only when user is logged on", everything wells well and the screenshot is saved. If however I configure the task with "Run whether user is logged in or not" it doesn't.
Interesting note is that if I replace "firefox" with "chrome" in my script, the screenshot is correctly is saved in both cases.
The only reference to this issue that I found was this: https://stackoverflow.com/a/47283428/2847079 but it doesn't provide any solution.
As the asker/answerer of the referenced issue:
Try the workaround. Change the task to run only when the user is logged in, and leave the user logged in at all times - which was a good enough solution for me, since the machine I ran the task on did nothing else.
I realise that this is less than optimal solution, so perhaps you can work out some separate task to do the logging in and out at the appropriate times before and after you run your main task. See this question which explains how to go about this: https://superuser.com/questions/616206/run-interactive-task-even-if-user-is-not-logged-on-windows
An option that was not available to me - but may be to you - is to not run the task on windows. Windows (somewhat bafflingly) forces you to be logged in a priori to run an interactive task, but you should be able to run firefox interactively in linux with a cronjob without being logged in.
Unfortunately for you (but very fortunately for me!) I no longer work at the company where this environment/code problem arose, so I can't test the above suggestions.
At any rate, I suggest you submit a bug report to the selenium project, as there doesn't appear to be a proper solution at present - though happy to be proven wrong.

robot framework: Reuse existing Browser Window between Tests

I would like to run tests with pybot, then run more tests with pybot using the same browser window that the first pybot opened.
So…
pybot test1.txt
#opens browser window and runs test1.txt and doesn't close the window
#pybot completes execution
pybot test2.txt
#uses the same browser window from test1
#pybot completes execution
pybot test3.txt
#uses the same browser window from test1
#pybot completes execution
can't figure out how to do that….
I've tried Open Browser www.mysite.com alias=1 in the first test and then Switch Browser 1 in the others, but they just error with No browser is open
A selenium driver instance has two properties characterizing its connection to a selenium webdriver - a connection url, and session id. By setting those to the values of an already running one, you effectively "hijack" it, and can use freely.
Disclaimer - the solution uses internal SE structures, so can break on newer versions. Also, as you are connecting to the running webdriver as to aRemote one, you cannot close it even if you want to - thus this can lead to resources leakage on the machine that it runs on; e.g. that webdriver has to be eventually terminated manually by a task manager.
So first things first - you have a running browser instance, and you need to get its properties for future connections. They are 2 - driver.command_executor._url, and driver.session_id where driver is the object name of the running instance. This python code will do just that:
from robot.libraries.BuiltIn import BuiltIn
def return_driver_props()
seLib = BuiltIn().get_library_instance('SeleniumLibrary')
# the driver is instantiated in the SeleniumLibrary, but not provided publicly, thus accessing it through this py code
remote_url = seLib.driver.command_executor._url # for local instance, this is a value in the form 'http://localhost:57856'
session_id = seLib.driver.session_id
return remote_url, session_id
Importing that file as a Library, by calling the function/method you'll have the 2 props:
${conn_url} ${session_id}= Return Driver Props
# and do whatever is needed to make them known - log, store in a file, DB, etc.
Now in the second run that needs to reattach, and with the 2 values in hand, you just use the keyword Open Browser and specify a remote connection:
Open Browser about:about remote_url=${that_known_url} browser=${the_used_browser_type} # the last args - chrome|firefox|edge - whatever you're connecting two
The tricky part - the moment you connect to a remote server, selenium automatically starts a new session - which is a second browser instance (this started somewhere around selenium3, though I'm not sure on the exact timing). E.g. if you start using it right now - that is not the browser you wanted, but a brand new. That's also the reason why I gave as target address "about:about" - so it loads a dummy page, very fast.
Two things must happen at this point - a) you have to get rid of the "dummy" SE session, and b) switch to the previous one:
def set_driver_session_id(sesion_id):
""" Sets the sessoin_id of the current driver insance to the provided one. """
seLib = BuiltIn().get_library_instance('SeleniumLibrary')
if seLib.driver.session_id != sesion_id: # this is pretty much guaranteed to be the case
seLib.driver.close() # this closes the session's window
seLib.driver.quit() # for remote connections (like ours), this deletes the session, but doesn't stop the SE
# set to the session that's already running
seLib.driver.session_id = sesion_id
This function/keyword is called with the known session id:
Set Driver Session ID ${session_id}
,and voilà, you are now in control of the previous browser, with its full state - the url it was at, cookies, localStorage, etc.
I'm leaving as an exercise to the reader how to automatically pass the url and the session id.
What I myself am doing is storing them in a file in a temp folder after running the first piece, and reading from there in the follow-up runs, with some error handling around it - missing or bad file, the connection cannot happen, and so on, with fallbacks to new instance creation.
Actually it is absolute possible to do however you have to organise your tests in testsuites
testsuites/
__init__.robot
test1.robot
test2.robot
test3.robot
in __init__.robot you have to open browser in suite setup and destroy it in suite teardown f.e.
*** Settings ***
Suite Setup Open Browser
Suite Teardown Close Browser
and run tests pybot ./testsuites
I don't know how effcient it is but I needed the same process and I used code like this to complete this...
open browser ${HOMEPAGE} ${BROWSER}
#opens the browser to the homepage using the browser of my choice
maximize browser window
#maximizes the window
(DOES STUFF)
#completes some processes
select window title=calendar
#selects the window with the name calendar
(DOES MORE STUFF)
This worked for me, try it out.
If you need any information on 'select window' look here: http://rtomac.github.io/robotframework-selenium2library/doc/Selenium2Library.html#Select%20Window
Or you can always try
switch browser 1
I'm new to robotframework, but I hope this helped.
There's an open issue for new functionality to re-use existing browser session. There are some workaround mentions and custom code modifications that seem to achieve this.
In my case, I was setting off warning emails from the target website, about logging in from a new device, every time I logged in during testing. Instead, I wanted to log in a single time and then resume the same (authenticated) browser session during subsequent runs.
To do this, the key is that you need to point the webdriver to the same user data directory so that subsequent runs can resume the browser session. For my needs, I created a local directory called user-data.
I then defined:
Open Browser Profiled
[Arguments] ${url}
${options}= Evaluate sys.modules['selenium.webdriver'].ChromeOptions() sys, selenium.webdriver
Call Method ${options} add_argument --user-data-dir\=${./user-data}
Create WebDriver Chrome chrome_options=${options}
Go To ${url}
And:
Log In
Open Browser Profiled ${LOGIN_URL}
Input Text email ${EMAIL}
Input Password password ${password}
Click Button Log in
Then, on the first execution, I run:
Do The Things
Log In
# Open Browser Profiled ${AFTER_LOGIN_URL}
...
And, on subsequent executions:
Do The Things
# Log In
Open Browser Profiled ${AFTER_LOGIN_URL}
...
For extra points, you can also make this more dynamic and detect whether a log in is needed so that you do not have to modify the script, e.g.:
Do The Things
Log In or Open
...
Log In or Open
Open Browser Profiled ${AFTER_LOGIN_URL}
Sleep 2 # Wait for page to settle
${logged_in}= Run Keyword And Return Status Title Should Be ${AFTER_LOGIN_TITLE}
Run Keyword Unless ${logged_in} Log In
Special thanks to this post, which outlined the method I used to resume the browser session.

NAnt, MbUnit, CruiseControl, Selenium - passing settings to the test assembly

I am putting together some ideas for our automated testing platform and have been looking at Selenium for the test runner.
I am wrapping the recorded Selenium C# scripts in an MbUnit test, which is being triggered via the MbUnit NAnt task. The Selenium test client is created as follows:
selenium = new DefaultSelenium("host", 4444, "*iexplore", "http://[url]/");
How can I pass the host, port and url settings into the test so their values can be controlled via the NAnt task?
For example, I may have multiple Selenium RC servers listening and I want to use the same test code passing in each server address instead of embedding the settings within the tests themselves.
I have an approach mocked up using a custom NAnt task I have written but it is not the most elegant solution at present and I wondered if there was an easier way to accomplish what I want to do.
Many thanks if anyone can help.
Thanks for the responses so far.
Environment variables could work, however, we could be running parallel tests via a single test assembly so I wouldn't want settings to be overwritten during execution, which could break another test. Interesting line of thought though, thanks, I reckon I could use that in other areas.
My current solution involves a custom NAnt task build on top of the MbUnit task, which allows me to specify the additional host, port, url settings as attributes. These are then saved as a config file within the build directory and then read in by the test assemblies. This feels a bit "clunky" to me as my tests need to inherit from a specific class. Not too bad but I'd like to have less dependencies and concentrate on the testing.
Maybe I am worrying too much!!
I have a base class for all test fixtures which has the following setup code:
[FixtureSetUp]
public virtual void TestFixtureSetup ()
{
BrowserType = (BrowserType) Enum.Parse (typeof (BrowserType),
System.Configuration.ConfigurationManager.AppSettings["BrowserType"],
true);
testMachine = System.Configuration.ConfigurationManager.AppSettings["TestMachine"];
seleniumPort = int.Parse (System.Configuration.ConfigurationManager.AppSettings["SeleniumPort"],
System.Globalization.CultureInfo.InvariantCulture);
seleniumSpeed = System.Configuration.ConfigurationManager.AppSettings["SeleniumSpeed"];
browserUrl = System.Configuration.ConfigurationManager.AppSettings["BrowserUrl"];
targetUrl = new Uri (System.Configuration.ConfigurationManager.AppSettings["TargetUrl"]);
string browserExe;
switch (BrowserType)
{
case BrowserType.InternetExplorer:
browserExe = "*iexplore";
break;
case BrowserType.Firefox:
browserExe = "*firefox";
break;
default:
throw new NotSupportedException ();
}
selenium = new DefaultSelenium (testMachine, seleniumPort, browserExe, browserUrl);
selenium.Start ();
System.Console.WriteLine ("Started Selenium session (browser type={0})",
browserType);
// sets the speed of execution of GUI commands
if (false == String.IsNullOrEmpty (seleniumSpeed))
selenium.SetSpeed (seleniumSpeed);
}
I then simply supply the test runner with a config. file:
For MSBuild I use environment variables, I create those in my CC.NET config then they would be available in the script. I think this would work for you too.
Anytime I need to integrate with an external entity using NAnt I either end up using the exec task or writing a custom task. Given the information you posted it would seem that writing your own would indeed be a good solution, However you state you're not happy with it. Can you elaborate a bit on why you don't think you current solution is an elegant one?
Update
Not knowing internal details it seems like you've solved it pretty well with a custom task. From what I've heard, that's how I would have done it.
Maybe a new solution will show itself in time, but for now be light on yourself!