Jenkins return job status SUCCESS when the test failed - selenium

I am automating my protractor integration tests using Jenkins. When some of the tests fails Jenkins weeps saying that the result was successful. This is my output:
24 specs, 1 failure
Finished in 64.079 seconds
Shutting down selenium standalone server.
[launcher] 0 instance(s) of WebDriver still running
[launcher] chrome #1 failed 1 test(s)
[launcher] overall: 1 failed spec(s)
[launcher] Process exited with error code 1
Test failed but keep the grunt process alive.
Done, without errors.[39m
Terminating xvnc.
$ vncserver -kill :28
Killing Xtightvnc process ID 24369
Finished: SUCCESS
I don't know if my problem is selenium or grunt related.
Does any one know what can be the problem? Thanks in advance!

Looks like this Jenkins job is not set up properly, I suggest you review your Job Configuration.

After some research I found out the problem.
I am using "grunt-protractor-runner" library. It has an option call "keepAlive" and it was set to true. This caused the process to continue even is something was wrong and not passing the error code to grunt.
After set it up to false everything behave as expected.

Related

Mocha, Electron: Errors in testing

I was testing my electron app with mocha and got this error several times (and I'm still getting it!). Every time I run the program it returns with the error
> Executing task: npm run test <
> forte#1.0.0 test PATH TO PROJECT
> mocha ./tests/test.js
0 passing (3ms)
Test failed Failed to create session.
unknown error: Chrome failed to start: was killed.
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location \node_modules\spectron\lib\launcher.bat is no longer running, so ChromeDriver is
assuming that Chrome has crashed.)
Just in case this is important, the part which says (The process started from chrome location \node_modules... actually had the whole path, not just that.
In my test.js file I have the code from the spectron home page, which you can view here. What is wrong with the test files?

Unspecified Protractor error at the end of a test run

I have upgraded to Protractor 6.0.0 and Jasmine 3.3.1, and had to redo all my tests to run asynchronously.
When my Protractor test finishes running, I have an unexplained error at the end. No tests fail, but the process seems to abort irregularly. I am running against Chrome 73 in the latest webdriver, where I had to specifically downgrade to Chrome 73 webdriver because 74 is still in Beta.
Here is a snapshot of a test run:
Test suite 1 Tests
√ should check first thing
√ should check second thing
√ should display another thing
Executed 3 of 94 specs INCOMPLETE (89 SKIPPED).
C:\Users\Joon\AppData\Roaming\npm\node_modules\protractor\node_modules\jasmine-core\lib\jasmine-core\jasmine.js:3190
throw arguments[0];
^
NoSuchSessionError: invalid session id (Driver info:
chromedriver=73.0.3683.68
(47787ec04b6e38e22703e856e101e840b65afe72),platform=Windows NT
10.0.17134 x86_64)
at Object.checkLegacyResponse (C:\Users\Joon\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\error.js:585:15)
at parseHttpResponse (C:\Users\Joon\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\http.js:533:13)
at Executor.execute (C:\Users\Joon\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\http.js:468:26)
at
at process._tickCallback (internal/process/next_tick.js:188:7)
There is no additional error message, the only other SO post I found that had the same error message also had an additional promise rejected error afterwards.
How can I debug / troubleshoot what is causing this error?
I figured it out - there were some calls to browser.wait that did not have an await before them.
One of these was causing a synchronization error which led to that unspecified state management error at the end of the test run.
For anyone else experiencing WebDriver state errors after you upgrade from the control flow to the async flow - be sure to put await calls in front of all Protractor browser interaction calls. The one I had missed was browser.wait but it could be any other call to Protractor that causes this issue

Process unexpectedly closed with status 1

I am running selenium tests with selenium 3.8.0 and geckodriver 0.19.1. During the run some tests pass fine, while for other tests I get the error:
Process unexpectedly closed with status 1
My question: In which code file is this error thrown?
I am not asking for a solution of my problem as I doubt anyone can help. All I am asking is the source code which throws the error so I can debug it myself.

Selenium Exception with no stack trace (grunt / protractor / selenium)

I try to run grunt test, and Selenium exits without closing the browser. I am unable to see where the exception is coming from.
grunt test
Running "protractor_webdriver:start" (protractor_webdriver) task
Starting Selenium server
Started Selenium server: http://127.94.0.1:4444
Running "protractor:projecttest" (protractor) task
[10:16:33] I/hosted - Using the selenium server at http://localhost:4444/wd/hub
[10:16:33] I/launcher - Running 1 instances of WebDriver
Session created: count=1, browserName=chrome
......Exception thrown: Going to shut down the Selenium server
Shutting down Selenium server: http://127.94.0.1:4444
Shut down Selenium server: http://127.94.0.1:4444 (OKOK)
Fatal error: 10:16:45.529 WARN - Exception thrown
All of my tests are passing. I even added logs to prove the final test passes and exits without exception.
I tried to up my logging from "dots" to "progress" but it doesn't change anything. Any ideas?

Can I set a longer timeout for protractor to connect to selenium driver?

Running my protractor tests remotely (jenkins) leads to timeout error sometimes. That is not deterministic.
Starting selenium standalone server...
[launcher] Running 1 instances of WebDriver
[launcher] Process exited with error code 1
/opt/jenkins.dir/workspace/my-jenkins-job/integration-test/ui/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:1761
throw error;
^
Error: Timed out waiting for the WebDriver server at http://10.97.193.53:4455/wd/hub
at Error (<anonymous>)
at onResponse (/opt/jenkins.dir/workspace/my-jenkins-job/integration-test/ui/node_modules/protractor/node_modules/selenium-webdriver/http/util.js:87:11)
at /opt/jenkins.dir/workspace/my-jenkins-job/integration-test/ui/node_modules/protractor/node_modules/selenium-webdriver/http/util.js:42:21
at /opt/jenkins.dir/workspace/my-jenkins-job/integration-test/ui/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/http/http.js:96:5
at ClientRequest.<anonymous> (/opt/jenkins.dir/workspace/my-jenkins-job/integration-test/ui/node_modules/protractor/node_modules/selenium-webdriver/http/index.js:145:7)
at ClientRequest.emit (events.js:95:17)
at Socket.socketErrorListener (http.js:1548:9)
at Socket.emit (events.js:95:17)
at net.js:441:14
at process._tickCallback (node.js:448:13)
However when I run the tests locally in my mac there is no problem and the tests run perfectly.
I have tried to start the selenium servers manually in the remote machines and I have realised that sometimes it works immediately and sometimes I have to wait up to one minute.
My question is: Is there any way to tell protractor to wait longer for the webdriver to connect?
Environment details
Machine: Red Hat 4.4.7-11
Protractor version: 1.8.0
Selenium Server Standalone: 2.45.0
You can specify it by using driver.wait function.
var webdriver = require('selenium-webdriver');
var protractor = require('protractor');
var driver = new webdriver.Builder().usingServer("seleniumAddress").build();
var browser = protractor.wrapDriver(driver);
browser.driver.wait(driver.getWindowHandle(), 5000, 'Server should start within 5 seconds');
References :
http://angular.github.io/protractor/#/api?view=webdriver.WebDriver.prototype.wait
http://angular.github.io/protractor/#/api?view=webdriver.WebDriver.prototype.getWindowHandle
Yes, and it should solve your issue. Use the seleniumServerStartTimeout option in your protractor.conf.js file to bump up the timeout from the default 30 seconds to something longer like 90 seconds:
exports.config = {
seleniumServerStartTimeout: 90000
};
I experienced the same issue on a CentOS 7 VM. For whatever reason the selenium server seems to take wildly different amounts of time to start up each time, and can sometimes exceed the default timeout.