we are trying to run Selenium-Tests on Browserstack against an AWS-Vaadin-App on several Jenkins slaves in parallel.
Companies-Jenkins -> Browserstack -> AWS-Vaadin-App
Our test framework uses the Vaadin Testbench with a valid license key.
All tests start as expected with a login (at the app) and the business workflow. But after a while there is a connection closed on all tests, the Vaadin framework shows "server connection lost".
T0 -> T1
-> T2
-> Tn
If we run the same on just one Jenkins slave in a sequence, it runs ok (also here we can see sometimes "server connection lost" but the selenium tests tries to wait and go on when the warning disappears, usually that works. In parallel it never works).
T0 -> T1 -> T2 -> Tn
Do you have an idea, why this happens? Could it be a problem with our Vaadin license?
It sounds like a problem with the server, not the tests. Is your server running out of memory by doing too much in parallel? Check the server logs, that is where you should find the reason. You can most likely see the same "server connection lost" if you manually open a session in your browser while the tests are running
We got help from the browserstack support:
eg. Initiate a Binary connection as:
./BrowserStackLocal --key $BROWSERSTACK_ACCESS_KEY --local-identifier test123
This initiates a unique Binary connection with a unique modifier "test123". This can then be used when executing the tests by setting the below capabilities:
caps.setCapability("browserstack.local","true");
caps.setCapability("browserstack.localIdentifier", "test123");
The same details are also mentioned in the link: https://www.browserstack.com/local-testing/app-automate#multiple-local-testing-connections
Related
I need to test our mail server in a clojure project. To do that I thought I would open a mock up server and send emails using this mock up server and check if they are sent. For that I found for example this server.
To be able to execute
lein test
and have each test tested, I need to run the SMTP server once before each test and once at the end. I can also run the server in a fixture and exit it after each test. Since i am running about 100 tests, it does not make sense to always start and shutdown the server.
My approaches that i thought are the following:
1 - I write a bash script that starts the (mockup) mail server, runs lein test, then shuts down the server.
(Here I lose the ease of executing lein test in the IDE)
2- I could have a fixture checking if the server is started and start it if its not. However after the test finished the server will still be running which is not desired.
What is the correct way to solve this problem?
Can I order the tests in clojure such that the last test file shutsdown the mail server ?
One solution is to use the Java GreenMail library. It allows you to start up an SMPT mail server in your JVM, which makes it easy to start, stop and inspect.
Here are a few snippets from my testing with GreenMail. First you create mail server:
(def mail-setup (ServerSetup. ServerSetup/PORT_SMTP nil ServerSetup/PROTOCOL_SMTP))
(def green-mail (GreenMail. mail-setup))
(.start green-mail)
; Now the server listens on localhosts SMPT port for emails. Run your test code
; Run the code under test. Then you can receive the emails
(doseq [m (.getReceivedMessages green-mail)]
; Verify the emails etc
(println "Subject: " (.getSubject m)
" TO: " (str/join "," (.getRecipients m Message$RecipientType/TO)))
)
; When done
(.stop green-mail)
Depending on you your tests you can start and stop it per test. Or might keep a test server running for a whole test suite.
Check the GreeMail documentation for more details. It supports tons of scenarious.
I have a test system (various web pages / web applications), that is hosted in an environment accessible only via machines with IP addresses that are white listed. I control the white list.
Our CI system is cloud hosted (Gitlab), so VMs are spun up dynamically as needed to run automated integration tests as a part of the build pipeline.
The tests in question use BrowserStack automation to run Selenium based tests, which means the source IP addresses of the BrowserStack automation driven requests that hit the test environment are dynamic, as BS is cloud hosted. Also the IP addresses of our test runner machines that call / invoke the BrowserStack automation are dynamic as well.
The whole system worked fine before the intro of IP white listing on the test environment. Since white listing was enabled, the BrowserStack tests can no longer access the environment URLs (due to not being able to white list the dynamic IPs).
I have been trying to get the CI driven tests working again using BS "Local Testing" feature, outlined here https://www.browserstack.com/local-testing.
I have set-up a dedicated Linux VM with a static IP address (cloud hosted). I have installed and am running the BrowserStackLocal.exe binary, using our BS key. It starts up fine and says it has connected to BrowserStack via a web socket. My understanding is this should cause all http(s) etc requests that come from my CI / BrowserStack automation driven tests to be routed through that stand-alone machine (via BS cloud), resulting in it's static IP address being the source of the requests seen at the test environment. This IP addr is white listed.
This is the command that is running on the dedicated / static IP machine:
BrowserStackLocal.exe --{access key} --verbose 3
I have also tried the below, but it made no apparent difference:
BrowserStackLocal.exe --{access key} --force-local --verbose 3
However, this does not seem to work? Either through "live" testing if I try and access the test env directly through BrowserStack, or through BS automate. In both cases the http(s) requests all time out and cannot access our test environment URLs. Also even with --verbose 3 logging level enabled on the BrowserStackLocal.exe process, I never see any request being logged on the stand-alone / static IP machine when I try to run the tests in various ways.
So I am wondering if this is the correct way to solve this problem? Am I misunderstanding how to do this? Do I need to run the BrowserStackLocal.exe perhaps on the same CI runner machine that is invoking the BS automation? This would be problematic as these have dynamic IPs as well (currently).
Thanks in advance for any help!
EDIT/UPDATE: I managed to get this to work!! (Sort of) - it's just a bit slow. If I run the following command on my existing dedicated / static IP server:
BrowserStackLocal.exe --key {mykey} --force-local --verbose 3
Then on another machine (like my dev laptop) if I hit the BS web driver server http://hub-cloud.browserstack.com/wd/hub, and access the site http://www.whatsmyip.org/ to see what IP address comes back, and it did (eventually) come back with my static IP machines address! The problem though is it was quite slow - 20-30 secs for that one site hit, so still looking at alternative solutions. Note for this to work your test code must set the "local" browserstack capability flag to 'true' - eg for Node.js:
// Input capabilities
var capabilities = {
'browserstack.local' : 'true'
}
UPDATE 2: Turning down the --verbose logging level on the local binary (or leaving that flag off completely) seemed to improve things - I am getting 5-10 sec response times now for each request. That might have to do. But this does work as described.
SOLUTION: I managed to get this to work - it's just a bit slow. If I run the following command on my existing dedicated / static IP server (note adding verbose logging seems to slow things down more, so no --verbose flag used now):
BrowserStackLocal.exe --key {mykey} --force-local
Then on another machine (like my dev laptop) if I hit the BS web driver server http://hub-cloud.browserstack.com/wd/hub, and access the site http://www.whatsmyip.org/ to see what IP address comes back, and it did come back with my static IP machines address. Note for this to work your test code must set the "local" browserstack capability flag to 'true' - eg for Node.js:
// Input capabilities
var capabilities = {
'browserstack.local' : 'true'
}
So while a little slow, that might have to do. But this does work as described.
I want to sync automation result from jenkins to testlink. I tried with Testlink -jenking plugin and testlink-api-client but not worked getting error.
Pre-setup :
$tlCfg-> api-> enabled
$tlCfg-> exec_cfg-> enable_test_automation
From Testlink UI enable automation for the project.
Test code :
TestLinkAPIClient testlinkAPIClient = new TestLinkAPIClient(APIKEY, "http://localhost/testlink/lib/api/xmlrpc/v1/xmlrpc.php");
testlinkAPIClient.reportTestCaseResult(Project, TestPlan, TEST_CASE, Build, notes/comments, teststatus);
output :
"testlink.api.java.client.TestLinkAPIException: The call to the xml-rpc client failed.".
References used :satishjohn.wordpress.com
2. softwaretestinghelp.com
and other stackoverflow threads.
I browsed and try out defined steps from some of the blogs but still facing same issue?. Can anyone help me to resolve this issue or other approach on sync result with testlink ?.
I believe you should follow the documentation(1) written by kino who wrote the plugin.We recently managed to sync automation results from Jenkins to Testlink by following above doc.Our auto tests were written based on testng framework, Hence we used "testng-results.xml" and TestNg method name based result seeking strategy.
We didn't come across an issue as you mentioned. From (2) and (3) you can get the plugin source .My advice is to debug the code after enabling the debug on Jenkins hosted tomcat server. So you can find the actual cause of the issue by yourself.
Reference:
(1) https://wiki.jenkins-ci.org/download/attachments/753702/jenkins.pdf
(2) https://github.com/jenkinsci/testlink-plugin
(3) https://github.com/kinow/testlink-java-apienter code here
You can run wireshark and filter on port "tcp port http" to see exact error you get from the server. When it was not working for us we were getting 200 OK with text "XML-RPC server accepts POST requests only."
You can also check /var/log/apache2/error.log for testlink errors.
We fixed the issue by setting following config in config.inc.php and restarting apache.
$tlCfg->api->enabled = TRUE;
$tlCfg->exec_cfg->enable_test_automation = ENABLED;
Environment:
- Selenium 2.39 Standalone Server
- PHP 5.4.11
- PHPUnit 3.7.28
- Chrome V31 & ChromeDriver v2.7
I'm testing a website,which invokes a lot of Advertisement Systems,such as Google AD.
The browser takes a lot of time to connect to external AD links , even all the elements of the page has already been loaded.
If my internet network was not fast when I ran my tests on a webpage,
Selenium would wait for a very long time ,since the AD links responsed slowly.
Under this condition ,Selenium usually waits for over 60 seconds, and throws a timeout exception.
I'm not sure how Senelium works, but it seems that Selenium has to wait for a sign of webpage's full loading, then pulls the DOM to find elements.
I want to make selenium operate elements without waiting for connectiong to external AD links.
Is there a way to do that ? Thank you very much.
I would suggest that you could make use of a proxy. Browsermob integrates well with selenium, very easy to use it:
// start the proxy
ProxyServer server = new ProxyServer(4444);
server.start();
// get the Selenium proxy object
Proxy proxy = server.seleniumProxy();
// This line will automatically return http.200 for any request going to google analytics
server.blacklistRequests("https?://.*\\.google-analytics\\.com/.*", 200);
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, proxy);
// start the browser up
WebDriver driver = new FirefoxDriver(capabilities);
I'm not sure how Senelium works, but it seems that Selenium has to
wait for a sign of webpage's full loading, then pulls the DOM to find
elements.
It is pretty much like this. The default loading strategy is "NORMAL" which means:
NORMAL of type DOMString
The remote end MUST wait until the "document.readyState" of the frame currently handling commands equals "complete", or there are no
more outstanding network requests other than XMLHttpRequests.
I finally found a simple solution for my condition.
I decide to block these Ad requests and tried some firewall and proxy softwares,for example,
comodo,privatefirewall, etc.
comodo is too heavy and complex ,privatefirewall doesn't support wildcards, and firewall would interrupt tests. At last I choosed a proxy software CCproxy. Trial Version is enough.
I create a rule for localhost ,to make it can request my test website domain only, and all other requests are rejected.
Running a test costs about 1-2 minutes　before and only 30 seconds now ,it's apparently more stable and fast without connecting to the useless Ad links.
Here're configuration steps:
1.launch CCproxy with Administor privilege( you should set it using Adminisrator in the file property)
2.click Options, select AutoStartup,select AutoDetected for Local IP Address. click OK.
3.create a txt file ,input your domains,like " *.rong360.com*;*.rong360.*; "
4.click Account, select PermitOnly for Permit Category;
click New, input 127.0.0.1 for IP Address/Range;
select WebFilter,click the E button at right side to create a filter;
click the ... button,select the text file you create at Step3,
select PermittedSites. click OK
click OK.
5.click OK to return to the main UI of CCproxy.
6.launch IE and config the local proxy with 127.0.0.1:808
other browsers will use this config automatically too.
now you can run the tests again , you'll feel better if have same condition :)
I wrote some junit test cases using seleniumrc . I am trying to integrate the test cases with Jenkins.
I installed seleniumrc-plugin for eclipse.I am unable to add the test cases to jenkins.
In Jenkins--> COnfigureProject --> Build Environment --> Create Seleniumrc instance ,I entered all the details for
Host , Port ,Browser ,OS
After that,I don't know what to choose for Build. I tried going through this source
https://wiki.jenkins-ci.org/display/JENKINS/SeleniumRC+Plugin
but couldnt figure out a way. Can anyone please explain me in detailed what to do next?
Try to configure Selenium manager and Remote Agents according to pt. 1 and 2.
After that on Jenkins CI configure Host and Port which must point to machine where "Selenium manager instance" is launched (it can be the same or another machine)