We have a release definition setup within VSTS with a Visual Studio Test task to run our automated test suite. The problem we're experiencing is that the release is succeeding regardless of whether there are failing tests or not. There doesn't appear to be any control within the task configuration over the behaviour of error handling/test failures.
Does anybody have any workarounds or solutions for what we're trying to achieve?
Thanks.
UPDATE
Release Logs:
2018-04-10T07:50:31.6993593Z ##[section]Starting: Execute Automated Tests
2018-04-10T07:50:31.6999650Z ==============================================================================
2018-04-10T07:50:31.6999764Z Task : Visual Studio Test
2018-04-10T07:50:31.6999919Z Description : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test runner. Test frameworks that have a Visual Studio test adapter such as xUnit, NUnit, Chutzpah, etc. can also be run. Tests can be distributed on multiple agents using this task (version 2).
2018-04-10T07:50:31.7000100Z Version : 2.3.28
2018-04-10T07:50:31.7000183Z Author : Microsoft Corporation
2018-04-10T07:50:31.7000292Z Help : [More Information](https://go.microsoft.com/fwlink/?LinkId=835764)
2018-04-10T07:50:31.7000391Z ==============================================================================
2018-04-10T07:50:32.4137415Z Run the tests locally using vstest.console.exe
2018-04-10T07:50:32.4137720Z ========================================================
2018-04-10T07:50:32.4137953Z Test selector : Test assemblies
2018-04-10T07:50:32.4138095Z Test assemblies : **\V6TestAutomation.dll
2018-04-10T07:50:32.4138483Z Test filter criteria : null
2018-04-10T07:50:32.4138620Z Search folder : D:\a\r1\a
2018-04-10T07:50:32.4138759Z Run settings file : D:\a\r1\a
2018-04-10T07:50:32.4138932Z Run in parallel : false
2018-04-10T07:50:32.4139050Z Run in isolation : false
2018-04-10T07:50:32.4139740Z Path to custom adapters : D:\a\r1\a\Test Automation\packages
2018-04-10T07:50:32.4139868Z Other console options : null
2018-04-10T07:50:32.4140002Z Code coverage enabled : false
2018-04-10T07:50:32.4140402Z Rerun failed tests: true
2018-04-10T07:50:32.4140572Z Rerun failed tests threshold: 30
2018-04-10T07:50:32.4140686Z Rerun maximum attempts: 3
2018-04-10T07:50:32.4140862Z VisualStudio version selected for test execution : latest
2018-04-10T07:50:33.9447744Z ========================================================
2018-04-10T07:50:34.1691886Z SystemVssConnection exists true
2018-04-10T07:50:34.1694146Z [command]D:\a\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.3.28\Modules\DTAExecutionHost.exe
2018-04-10T07:50:36.9541002Z ##########################################################################
2018-04-10T07:50:37.1168390Z Starting TestExecution Model...
2018-04-10T07:50:46.8510175Z **************** Starting test execution *********************
2018-04-10T07:50:46.8510636Z [command]C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe #C:\Users\VSSADM~1\AppData\Local\Temp\d9500801-3c93-11e8-bd28-bd9645a0d16e.txt
2018-04-10T07:50:51.7078146Z Microsoft (R) Test Execution Command Line Tool Version 15.6.0
2018-04-10T07:50:51.7078766Z Copyright (c) Microsoft Corporation. All rights reserved.
2018-04-10T07:50:51.8463454Z vstest.console.exe
2018-04-10T07:50:51.8463809Z "D:\a\r1\a\Test Automation\bin\V6TestAutomation.dll"
2018-04-10T07:50:51.8464039Z /logger:"trx"
2018-04-10T07:50:51.8464255Z /TestAdapterPath:"D:\a\r1\a\Test Automation\packages"
2018-04-10T07:50:55.2887935Z Starting test execution, please wait...
2018-04-10T07:50:58.9435629Z SpecFlow+Runner execution started
2018-04-10T07:50:59.9152811Z SpecRun: running tests in D:\a\r1\a\Test Automation\bin\V6TestAutomation.dll
2018-04-10T07:50:59.9153503Z Profile: Results
2018-04-10T07:51:00.3834652Z Discovering target: Default
2018-04-10T07:51:00.3834868Z Discovering tests from assembly: D:\a\r1\a\Test Automation\bin\V6TestAutomation.dll
2018-04-10T07:51:04.1966900Z Starting ChromeDriver 2.37.543627 (63642262d9fb93fb4ab52398be4286d844092a5e) on port 1675
2018-04-10T07:51:04.1967270Z Only local connections are allowed.
2018-04-10T07:51:13.5791284Z Scenario: Create Record in TestTableExplorer -> Failed on thread #0
2018-04-10T07:51:13.5863835Z ##[error] [ERROR] no such element: Unable to locate element: {"method":"id","selector":"serialNumber"}
2018-04-10T07:51:13.5915120Z ##[error] (Session info: chrome=65.0.3325.181)
2018-04-10T07:51:13.5917595Z ##[error] (Driver info: chromedriver=2.37.543627 (63642262d9fb93fb4ab52398be4286d844092a5e),platform=Windows NT 10.0.14393 x86_64)
2018-04-10T07:51:13.5919630Z Failed Create Record
2018-04-10T07:51:13.5919860Z Error Message:
2018-04-10T07:51:13.5920023Z no such element: Unable to locate element: {"method":"id","selector":"serialNumber"}
2018-04-10T07:51:13.5920218Z (Session info: chrome=65.0.3325.181)
2018-04-10T07:51:13.5920384Z (Driver info: chromedriver=2.37.543627 (63642262d9fb93fb4ab52398be4286d844092a5e),platform=Windows NT 10.0.14393 x86_64)
2018-04-10T07:51:13.5920571Z Stack Trace:
2018-04-10T07:51:13.5920736Z OpenQA.Selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"id","selector":"serialNumber"}
2018-04-10T07:51:13.5920924Z (Session info: chrome=65.0.3325.181)
2018-04-10T07:51:13.5921110Z (Driver info: chromedriver=2.37.543627 (63642262d9fb93fb4ab52398be4286d844092a5e),platform=Windows NT 10.0.14393 x86_64)
2018-04-10T07:51:13.5921294Z at OpenQA.Selenium.Remote.RemoteWebDriver.UnpackAndThrowOnError(Response errorResponse)
2018-04-10T07:51:13.5921463Z at OpenQA.Selenium.Remote.RemoteWebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
2018-04-10T07:51:13.5921666Z at OpenQA.Selenium.Remote.RemoteWebDriver.FindElement(String mechanism, String value)
2018-04-10T07:51:13.5921829Z at OpenQA.Selenium.Remote.RemoteWebDriver.FindElementById(String id)
2018-04-10T07:51:13.5922006Z at OpenQA.Selenium.By.<>c__DisplayClass16_0.<Id>b__0(ISearchContext context)
2018-04-10T07:51:13.5922164Z at OpenQA.Selenium.By.FindElement(ISearchContext context)
2018-04-10T07:51:13.5922735Z at OpenQA.Selenium.Remote.RemoteWebDriver.FindElement(By by)
2018-04-10T07:51:13.5923027Z at V6TestAutomation.Areas.TestTableExplorer.TestTableExplorerSteps.WhenICreateARecord() in D:\a\1\s\V6TestAutomation\Areas\TestTableExplorer\TestTableExplorerSteps.cs:line 52
2018-04-10T07:51:13.5923250Z at lambda_method(Closure , IContextManager )
2018-04-10T07:51:13.5923463Z at TechTalk.SpecFlow.Bindings.BindingInvoker.InvokeBinding(IBinding binding, IContextManager contextManager, Object[] arguments, ITestTracer testTracer, TimeSpan& duration)
2018-04-10T07:51:13.5924073Z at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStepMatch(BindingMatch match, Object[] arguments)
2018-04-10T07:51:13.5924298Z at TechTalk.SpecRun.SpecFlowPlugin.Runtime.RunnerTestExecutionEngine.ExecuteStepMatch(BindingMatch match, Object[] arguments)
2018-04-10T07:51:13.5924540Z at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(IContextManager contextManager, StepInstance stepInstance)
2018-04-10T07:51:13.5924743Z at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnAfterLastStep()
2018-04-10T07:51:13.5924929Z at TechTalk.SpecFlow.TestRunner.CollectScenarioErrors()
2018-04-10T07:51:13.5925145Z at V6TestAutomation.Areas.TestTableExplorer.TestTableExplorerFeature.ScenarioCleanup()
2018-04-10T07:51:13.5925381Z at V6TestAutomation.Areas.TestTableExplorer.TestTableExplorerFeature.CreateRecord() in D:\a\1\s\V6TestAutomation\Areas\TestTableExplorer\TestTableExplorer.feature:line 10
2018-04-10T07:51:13.5925639Z at TechTalk.SpecRun.Framework.TaskExecutors.StaticOrInstanceMethodExecutor.ExecuteInternal(ITestThreadExecutionContext testThreadExecutionContext)
2018-04-10T07:51:13.5926381Z at TechTalk.SpecRun.Framework.TaskExecutors.StaticOrInstanceMethodExecutor.Execute(ITestThreadExecutionContext testThreadExecutionContext)
2018-04-10T07:51:13.5926594Z at TechTalk.SpecRun.Framework.TestAssemblyExecutor.ExecuteTestNodeTask(TestNode testNode, ITaskExecutor task, TraceEventType eventType)
** OMITTED FOR SIMPLICITY **
2018-04-10T07:51:16.0337874Z Standard Output Messages:
2018-04-10T07:51:16.0338025Z Given the test table explorer page is displayed
2018-04-10T07:51:16.0338210Z -> done: TestTableExplorerSteps.GivenTheTestTableExplorerPageIsDisplayed() (0.7s)
2018-04-10T07:51:16.0338368Z
2018-04-10T07:51:16.0338510Z And a record exists in the records table
2018-04-10T07:51:16.0338697Z -> error: no such element: Unable to locate element: {"method":"id","selector":"serialNumber"}
2018-04-10T07:51:16.0338864Z -> (Session info: chrome=65.0.3325.181)
2018-04-10T07:51:16.0339035Z -> (Driver info: chromedriver=2.37.543627 (63642262d9fb93fb4ab52398be4286d844092a5e),platform=Windows NT 10.0.14393 x86_64)
2018-04-10T07:51:16.0339209Z
2018-04-10T07:51:16.0339345Z When I delete a record
2018-04-10T07:51:16.0339491Z -> skipped because of previous errors
2018-04-10T07:51:16.0339647Z
2018-04-10T07:51:16.0339793Z Then the record is not displayed in the records table
2018-04-10T07:51:16.0339955Z -> skipped because of previous errors
2018-04-10T07:51:16.0340118Z Standard Error Messages:
2018-04-10T07:51:16.0340454Z no such element: Unable to locate element: {"method":"id","selector":"serialNumber"}
2018-04-10T07:51:16.0340705Z (Session info: chrome=65.0.3325.181)
2018-04-10T07:51:16.0340935Z (Driver info: chromedriver=2.37.543627 (63642262d9fb93fb4ab52398be4286d844092a5e),platform=Windows NT 10.0.14393 x86_64) OpenQA.Selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"id","selector":"serialNumber"}
2018-04-10T07:51:16.0341397Z (Session info: chrome=65.0.3325.181)
2018-04-10T07:51:16.0341607Z (Driver info: chromedriver=2.37.543627 (63642262d9fb93fb4ab52398be4286d844092a5e),platform=Windows NT 10.0.14393 x86_64)
2018-04-10T07:51:16.0342023Z at OpenQA.Selenium.Remote.RemoteWebDriver.UnpackAndThrowOnError(Response errorResponse)
2018-04-10T07:51:16.0342779Z at OpenQA.Selenium.Remote.RemoteWebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
2018-04-10T07:51:16.0343045Z at OpenQA.Selenium.Remote.RemoteWebDriver.FindElement(String mechanism, String value)
2018-04-10T07:51:16.0343244Z at OpenQA.Selenium.Remote.RemoteWebDriver.FindElementById(String id)
2018-04-10T07:51:16.0343450Z at OpenQA.Selenium.By.<>c__DisplayClass16_0.<Id>b__0(ISearchContext context)
2018-04-10T07:51:16.0343659Z at OpenQA.Selenium.By.FindElement(ISearchContext context)
2018-04-10T07:51:16.0343844Z at OpenQA.Selenium.Remote.RemoteWebDriver.FindElement(By by)
2018-04-10T07:51:16.0344068Z at V6TestAutomation.Areas.TestTableExplorer.TestTableExplorerSteps.WhenICreateARecord() in D:\a\1\s\V6TestAutomation\Areas\TestTableExplorer\TestTableExplorerSteps.cs:line 52
2018-04-10T07:51:16.0344295Z at lambda_method(Closure , IContextManager )
2018-04-10T07:51:16.0344530Z at TechTalk.SpecFlow.Bindings.BindingInvoker.InvokeBinding(IBinding binding, IContextManager contextManager, Object[] arguments, ITestTracer testTracer, TimeSpan& duration)
2018-04-10T07:51:16.0344900Z at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStepMatch(BindingMatch match, Object[] arguments)
2018-04-10T07:51:16.0345120Z at TechTalk.SpecRun.SpecFlowPlugin.Runtime.RunnerTestExecutionEngine.ExecuteStepMatch(BindingMatch match, Object[] arguments)
2018-04-10T07:51:16.0345350Z at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(IContextManager contextManager, StepInstance stepInstance)
2018-04-10T07:51:16.0345574Z at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnAfterLastStep()
2018-04-10T07:51:16.0345762Z at TechTalk.SpecFlow.TestRunner.CollectScenarioErrors()
2018-04-10T07:51:16.0345972Z at V6TestAutomation.Areas.TestTableExplorer.TestTableExplorerFeature.ScenarioCleanup()
2018-04-10T07:51:16.0346352Z at V6TestAutomation.Areas.TestTableExplorer.TestTableExplorerFeature.DeleteRecord() in D:\a\1\s\V6TestAutomation\Areas\TestTableExplorer\TestTableExplorer.feature:line 17
2018-04-10T07:51:16.0346583Z at TechTalk.SpecRun.Framework.TaskExecutors.StaticOrInstanceMethodExecutor.ExecuteInternal(ITestThreadExecutionContext testThreadExecutionContext)
2018-04-10T07:51:16.0347340Z at TechTalk.SpecRun.Framework.TaskExecutors.StaticOrInstanceMethodExecutor.Execute(ITestThreadExecutionContext testThreadExecutionContext)
2018-04-10T07:51:16.0347580Z at TechTalk.SpecRun.Framework.TestAssemblyExecutor.ExecuteTestNodeTask(TestNode testNode, ITaskExecutor task, TraceEventType eventType)
2018-04-10T07:51:25.4504774Z Result: 2 failed
2018-04-10T07:51:25.4505328Z Total: 39
2018-04-10T07:51:25.4506188Z Succeeded: 0
2018-04-10T07:51:25.4510416Z Ignored: 37
2018-04-10T07:51:25.4511014Z Pending: 0
2018-04-10T07:51:25.4511308Z Skipped: 0
2018-04-10T07:51:25.4511584Z Failed: 2
2018-04-10T07:51:25.4511843Z Execution Time: 00:00:14.4039087
2018-04-10T07:51:25.4512174Z Report file: file:///D:\a\r1\a\TestResults\V6TestAutomation_Results_2018-04-10T075100.html
2018-04-10T07:51:25.4513035Z Log file: file:///D:\a\r1\a\TestResults\V6TestAutomation_Results_2018-04-10T075100.log
2018-04-10T07:51:25.4513325Z SpecFlow+Runner execution finished
2018-04-10T07:51:26.7860424Z Attachments:
2018-04-10T07:51:26.7861290Z C:\Users\VssAdministrator\AppData\Local\Temp\gxkaqztp.ujc\V6TestAutomation_Results_2018-04-10T075100.log
2018-04-10T07:51:26.7861672Z C:\Users\VssAdministrator\AppData\Local\Temp\gxkaqztp.ujc\V6TestAutomation_Results_2018-04-10T075100.html
2018-04-10T07:51:26.7862366Z Total tests: 39. Passed: 0. Failed: 2. Skipped: 37.
2018-04-10T07:51:26.7863031Z ##[error]Test Run Failed.
2018-04-10T07:51:26.7865279Z Test execution time: 28.6017 Seconds
2018-04-10T07:51:27.7157854Z Results File: D:\a\r1\a\TestResults\VssAdministrator_factoryvm-az261_2018-04-10_07_51_13.trx
2018-04-10T07:51:27.7162409Z **************** Completed test execution *********************
2018-04-10T07:51:27.7163247Z Test results files: D:\a\r1\a\TestResults\VssAdministrator_factoryvm-az261_2018-04-10_07_51_13.trx
2018-04-10T07:51:27.7789456Z Created test run: 450
2018-04-10T07:51:27.7790908Z Publishing test results: 39
2018-04-10T07:51:27.7846686Z Publishing test results to test run '450'.
2018-04-10T07:51:27.7875023Z Test results remaining: 39. Test run id: 450.
2018-04-10T07:51:28.8790810Z Published test results: 39
2018-04-10T07:51:28.8791160Z Publishing Attachments: 3
2018-04-10T07:51:29.8023722Z Failed tests: 2; Total tests: 39;
2018-04-10T07:51:29.9086633Z **************** Rerunning failed tests for Test run 450 *********************
2018-04-10T07:51:29.9086887Z Max attempts: 3; Current attempt: 1; Failed test cases threshold: 30; Failed test cases max limit: 0
2018-04-10T07:51:29.9087136Z [command]C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe #C:\Users\VssAdministrator\AppData\Local\Temp\tmp3668.tmp
2018-04-10T07:51:30.0872809Z Microsoft (R) Test Execution Command Line Tool Version 15.6.0
2018-04-10T07:51:30.0874243Z Copyright (c) Microsoft Corporation. All rights reserved.
2018-04-10T07:51:30.0907958Z vstest.console.exe /TestCaseFilter:"FullyQualifiedName=V6TestAutomation.TestTableExplorer.Delete Record|FullyQualifiedName=V6TestAutomation.TestTableExplorer.Create Record"
2018-04-10T07:51:30.0908308Z "D:\a\r1\a\Test Automation\bin\V6TestAutomation.dll"
2018-04-10T07:51:30.0908475Z /logger:"trx"
2018-04-10T07:51:30.0908664Z /TestAdapterPath:"D:\a\r1\a\Test Automation\packages"
2018-04-10T07:51:30.3707778Z Starting test execution, please wait...
2018-04-10T07:51:30.6698475Z SpecFlow+Runner execution started
2018-04-10T07:51:30.7564008Z SpecRun: running tests in D:\a\r1\a\Test Automation\bin\V6TestAutomation.dll
2018-04-10T07:51:30.7564629Z Profile: Results
2018-04-10T07:51:30.8100191Z Discovering target: Default
2018-04-10T07:51:30.8100808Z Discovering tests from assembly: D:\a\r1\a\Test Automation\bin\V6TestAutomation.dll
2018-04-10T07:51:33.2194874Z Result: tests executed with warnings
2018-04-10T07:51:33.2195303Z Total: 0
2018-04-10T07:51:33.2198974Z Succeeded: 0
2018-04-10T07:51:33.2204195Z Ignored: 0
2018-04-10T07:51:33.2204453Z Pending: 0
2018-04-10T07:51:33.2204660Z Skipped: 0
2018-04-10T07:51:33.2204862Z Failed: 0
2018-04-10T07:51:33.2205031Z Execution Time: 00:00:00.1429995
2018-04-10T07:51:33.2205259Z Report file: file:///D:\a\r1\a\TestResults\V6TestAutomation_Results_2018-04-10T075130.html
2018-04-10T07:51:33.2241215Z Log file: file:///D:\a\r1\a\TestResults\V6TestAutomation_Results_2018-04-10T075130.log
2018-04-10T07:51:33.2264814Z SpecFlow+Runner execution finished
2018-04-10T07:51:33.3474231Z No test is available in D:\a\r1\a\Test Automation\bin\V6TestAutomation.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
2018-04-10T07:51:33.3550733Z Attachments:
2018-04-10T07:51:33.3551210Z C:\Users\VssAdministrator\AppData\Local\Temp\ktqh3alc.o31\V6TestAutomation_Results_2018-04-10T075130.log
2018-04-10T07:51:33.3551627Z C:\Users\VssAdministrator\AppData\Local\Temp\ktqh3alc.o31\V6TestAutomation_Results_2018-04-10T075130.html
2018-04-10T07:51:33.3737235Z **************** Completed test execution *********************
2018-04-10T07:51:33.3742156Z Test results files: D:\a\r1\a\TestResults\VssAdministrator_factoryvm-az261_2018-04-10_07_51_13.trx
2018-04-10T07:51:33.3778622Z Updating test results: 39
2018-04-10T07:51:33.3806561Z Publishing test results to test run '450'.
2018-04-10T07:51:33.5064845Z Test results remaining: 39. Test run id: 450.
2018-04-10T07:51:34.1698821Z Updated test results: 39
2018-04-10T07:51:34.1699499Z Publishing Attachments: 3
2018-04-10T07:51:34.5508376Z Failed tests: 2; Total tests: 39;
2018-04-10T07:51:34.6399978Z **************** Rerunning failed tests for Test run 450 *********************
2018-04-10T07:51:34.6400353Z Max attempts: 3; Current attempt: 2; Failed test cases threshold: 30; Failed test cases max limit: 0
2018-04-10T07:51:34.6400646Z [command]C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe #C:\Users\VssAdministrator\AppData\Local\Temp\tmp48E7.tmp
2018-04-10T07:51:34.8356823Z Microsoft (R) Test Execution Command Line Tool Version 15.6.0
2018-04-10T07:51:34.8357741Z Copyright (c) Microsoft Corporation. All rights reserved.
2018-04-10T07:51:34.8394018Z vstest.console.exe /TestCaseFilter:"FullyQualifiedName=V6TestAutomation.TestTableExplorer.Delete Record|FullyQualifiedName=V6TestAutomation.TestTableExplorer.Create Record"
2018-04-10T07:51:34.8394448Z "D:\a\r1\a\Test Automation\bin\V6TestAutomation.dll"
2018-04-10T07:51:34.8394775Z /logger:"trx"
2018-04-10T07:51:34.8395040Z /TestAdapterPath:"D:\a\r1\a\Test Automation\packages"
2018-04-10T07:51:35.1104820Z Starting test execution, please wait...
2018-04-10T07:51:35.3962945Z SpecFlow+Runner execution started
2018-04-10T07:51:35.4856434Z SpecRun: running tests in D:\a\r1\a\Test Automation\bin\V6TestAutomation.dll
2018-04-10T07:51:35.4857610Z Profile: Results
2018-04-10T07:51:35.5388466Z Discovering target: Default
2018-04-10T07:51:36.2348959Z Discovering tests from assembly: D:\a\r1\a\Test Automation\bin\V6TestAutomation.dll
2018-04-10T07:51:37.8740951Z Result: tests executed with warnings
2018-04-10T07:51:37.8741119Z Total: 0
2018-04-10T07:51:37.8741196Z Succeeded: 0
2018-04-10T07:51:37.8741271Z Ignored: 0
2018-04-10T07:51:37.8741367Z Pending: 0
2018-04-10T07:51:37.8741441Z Skipped: 0
2018-04-10T07:51:37.8741515Z Failed: 0
2018-04-10T07:51:37.8741594Z Execution Time: 00:00:00.1390451
2018-04-10T07:51:37.8741720Z Report file: file:///D:\a\r1\a\TestResults\V6TestAutomation_Results_2018-04-10T075135.html
2018-04-10T07:51:37.8795841Z Log file: file:///D:\a\r1\a\TestResults\V6TestAutomation_Results_2018-04-10T075135.log
2018-04-10T07:51:37.8799980Z SpecFlow+Runner execution finished
2018-04-10T07:51:37.9989204Z No test is available in D:\a\r1\a\Test Automation\bin\V6TestAutomation.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
2018-04-10T07:51:38.0066895Z Attachments:
2018-04-10T07:51:38.0070117Z C:\Users\VssAdministrator\AppData\Local\Temp\qj2sdzi0.2zo\V6TestAutomation_Results_2018-04-10T075135.log
2018-04-10T07:51:38.0070466Z C:\Users\VssAdministrator\AppData\Local\Temp\qj2sdzi0.2zo\V6TestAutomation_Results_2018-04-10T075135.html
2018-04-10T07:51:38.0255222Z **************** Completed test execution *********************
2018-04-10T07:51:38.0260460Z Test results files: D:\a\r1\a\TestResults\VssAdministrator_factoryvm-az261_2018-04-10_07_51_13.trx
2018-04-10T07:51:38.0286385Z Updating test results: 39
2018-04-10T07:51:38.0286821Z Publishing test results to test run '450'.
2018-04-10T07:51:38.1088816Z Test results remaining: 39. Test run id: 450.
2018-04-10T07:51:38.8928593Z Updated test results: 39
2018-04-10T07:51:38.8929427Z Publishing Attachments: 3
2018-04-10T07:51:39.3643307Z Failed tests: 2; Total tests: 39;
2018-04-10T07:51:39.4770115Z **************** Rerunning failed tests for Test run 450 *********************
2018-04-10T07:51:39.4770400Z Max attempts: 3; Current attempt: 3; Failed test cases threshold: 30; Failed test cases max limit: 0
2018-04-10T07:51:39.4771471Z [command]C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe #C:\Users\VssAdministrator\AppData\Local\Temp\tmp5BC4.tmp
2018-04-10T07:51:39.6540115Z Microsoft (R) Test Execution Command Line Tool Version 15.6.0
2018-04-10T07:51:39.6540767Z Copyright (c) Microsoft Corporation. All rights reserved.
2018-04-10T07:51:39.6575246Z vstest.console.exe /TestCaseFilter:"FullyQualifiedName=V6TestAutomation.TestTableExplorer.Delete Record|FullyQualifiedName=V6TestAutomation.TestTableExplorer.Create Record"
2018-04-10T07:51:39.6575652Z "D:\a\r1\a\Test Automation\bin\V6TestAutomation.dll"
2018-04-10T07:51:39.6576107Z /logger:"trx"
2018-04-10T07:51:39.6576343Z /TestAdapterPath:"D:\a\r1\a\Test Automation\packages"
2018-04-10T07:51:39.9186431Z Starting test execution, please wait...
2018-04-10T07:51:40.2173519Z SpecFlow+Runner execution started
2018-04-10T07:51:40.3177982Z SpecRun: running tests in D:\a\r1\a\Test Automation\bin\V6TestAutomation.dll
2018-04-10T07:51:40.3178533Z Profile: Results
2018-04-10T07:51:40.3718045Z Discovering target: Default
2018-04-10T07:51:40.3728894Z Discovering tests from assembly: D:\a\r1\a\Test Automation\bin\V6TestAutomation.dll
2018-04-10T07:51:42.7354487Z Result: tests executed with warnings
2018-04-10T07:51:42.7358260Z Total: 0
2018-04-10T07:51:42.7361861Z Succeeded: 0
2018-04-10T07:51:42.7366617Z Ignored: 0
2018-04-10T07:51:42.7366911Z Pending: 0
2018-04-10T07:51:42.7367075Z Skipped: 0
2018-04-10T07:51:42.7367232Z Failed: 0
2018-04-10T07:51:42.7367423Z Execution Time: 00:00:00.1520062
2018-04-10T07:51:42.7367608Z Report file: file:///D:\a\r1\a\TestResults\V6TestAutomation_Results_2018-04-10T075140.html
2018-04-10T07:51:42.7400258Z Log file: file:///D:\a\r1\a\TestResults\V6TestAutomation_Results_2018-04-10T075140.log
2018-04-10T07:51:42.7400615Z SpecFlow+Runner execution finished
2018-04-10T07:51:42.8538664Z No test is available in D:\a\r1\a\Test Automation\bin\V6TestAutomation.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
2018-04-10T07:51:42.8612084Z Attachments:
2018-04-10T07:51:42.8612798Z C:\Users\VssAdministrator\AppData\Local\Temp\yetfa5qk.yec\V6TestAutomation_Results_2018-04-10T075140.log
2018-04-10T07:51:42.8613054Z C:\Users\VssAdministrator\AppData\Local\Temp\yetfa5qk.yec\V6TestAutomation_Results_2018-04-10T075140.html
2018-04-10T07:51:42.8800667Z **************** Completed test execution *********************
2018-04-10T07:51:42.8806717Z Test results files: D:\a\r1\a\TestResults\VssAdministrator_factoryvm-az261_2018-04-10_07_51_13.trx
2018-04-10T07:51:42.8852645Z Updating test results: 39
2018-04-10T07:51:42.8852918Z Publishing test results to test run '450'.
2018-04-10T07:51:42.9670889Z Test results remaining: 39. Test run id: 450.
2018-04-10T07:51:44.0187591Z Updated test results: 39
2018-04-10T07:51:44.0187889Z Publishing Attachments: 3
2018-04-10T07:51:44.2739520Z ##[warning]Maximum number of attempts running the failed tests reached: 3
2018-04-10T07:51:44.3511297Z Completed TestExecution Model...
2018-04-10T07:51:44.4323867Z ##[section]Finishing: Execute Automated Tests
Related
I'm running Selenium tests on CentOS 7 machine through Jenkins pipeline hence using Xvfb. When use screen argument with any valid value like following -
wrap([$class: 'Xvfb', autoDisplayName: true, screen: '1440x800x8']) {
<code to run Selenium test here>
}
It results into following error -
[testng] org.openqa.selenium.WebDriverException: chrome not reachable
[testng] (Session info: chrome=80.0.3987.132)
[testng] (Driver info: chromedriver=80.0.3987.16 (320f6526c1632ad4f205ebce69b99a062ed78647-refs/branch-heads/3987#{#185}),platform=Linux 3.10.0-862.el7.x86_64 x86_64) (WARNING: The server did not provide any stacktrace information)
[testng] Command duration or timeout: 0 milliseconds
[testng] Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:13:22.693Z'
If we run it without screen argument it runs successfully.
One more observation - with screen argument, it perfectly works on Ubuntu 18
Please help me how I can run Selenium tests with screen argument on CentOS 7.
After trying with different values for screen parameters, I realized this is not something which is wrong with Chrome browser or driver but with value passed to screen
When I changed depth value from 8 to 16, Selenium execution started successfully - screen: '1440x800x16'
wrap([$class: 'Xvfb', autoDisplayName: true, screen: '1440x800x16']) {
<code to run Selenium test here>
}
But technically, I did not understand why this error occurs and how depth value helped to resolve it.
My team is using Jenkins for running test suits and getting report. Our regression test suit runs well on local machines while running the same test on Jenkins throws several errors, most of which, nearly 99%, is about selenium unable to locate element.
What would be cause of the issue?
org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":"[data-testid='regular-shipment-form-step-shipment-scale-button-proceed']"}
(Session info: chrome=80.0.3987.106)
(Driver info: chromedriver=2.36.540471 (9c759b81a907e70363c6312294d30b6ccccc2752),platform=Linux 4.15.0-1077-azure x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds
I have a c-sharp solution (say sample.sln), but to run sonarqube analysis, as per the docs, i have run the following command:
".\3rdparty\sonarqube\msbuild\SonarScanner.MSBuild.exe" begin /k:"comm"
/n:"comm" /v:"2.0"
/d:sonar.cs.dotcover.reportsPaths=
"D:\Jenkins\workspace\comm\TEST\CoverageReport.html"
/d:sonar.host.url="http://101.149.24.128"
/d:sonar.sources="D:\Jenkins\workspace\comm\source\*.cs"
msbuild comm.sln /t:Rebuild
".\3rdparty\sonarqube\msbuild\SonarScanner.MSBuild.exe" end
but getting this error :
SONAR_SCANNER_OPTS is not configured. Setting it to the default value of -Xmx1024m
Calling the SonarQube Scanner...
INFO: Scanner configuration file: D:\Jenkins\workspace\comm\3rdparty\sonarqube\msbuild\sonar-scanner-3.2.0.1227\bin\..\conf\sonar-scanner.properties
INFO: Project root configuration file: D:\Jenkins\workspace\comm\.sonarqube\out\sonar-project.properties
INFO: SonarQube Scanner 3.2.0.1227
INFO: Java 1.8.0_151 Oracle Corporation (64-bit)
INFO: Windows Server 2016 10.0 amd64
INFO: SONAR_SCANNER_OPTS=-Xmx1024m
INFO: User cache: C:\Users\jenkins\.sonar\cache
INFO: SonarQube server 6.7.5
INFO: Default locale: "en_US", source code encoding: "windows-1252" (analysis is platform dependent)
INFO: Publish mode
INFO: Load global settings
INFO: Load global settings (done) | time=278ms
INFO: Server id: 7950B4AD-AWYRoMpRpurWib7aM-qP
INFO: User cache: C:\Users\jenkins\.sonar\cache
INFO: Load plugins index
INFO: Load plugins index (done) | time=84ms
INFO: Process project properties
INFO: ------------------------------------------------------------------------
INFO: EXECUTION FAILURE
INFO: ------------------------------------------------------------------------
INFO: Total time: 13.565s
INFO: Final Memory: 6M/84M
INFO: ------------------------------------------------------------------------
ERROR: Error during SonarQube Scanner execution
ERROR: Unable to load component class org.sonar.scanner.scan.ProjectLock
ERROR: Caused by: Unable to load component class org.sonar.scanner.scan.DefaultInputModuleHierarchy
ERROR: Caused by: Unable to load component class org.sonar.scanner.scan.ProjectBuildersExecutor
ERROR: Caused by: Unable to load component class com.talanlabs.sonar.plugins.gitlab.CommitProjectBuilder
ERROR: Caused by: Unable to load component class com.talanlabs.sonar.plugins.gitlab.GitLabPluginConfiguration
ERROR: Caused by: Unable to load component class org.sonar.scanner.scan.MutableProjectSettings
ERROR: Caused by: Unable to load component class org.sonar.api.batch.bootstrap.ProjectReactor
**ERROR: Caused by: Illegal char <*> at index 43: D:\Jenkins\workspace\comm\source\*.cs**
ERROR:
The SonarQube Scanner did not complete successfully
so i thought of adding, all the sonar related properties into sonar-project.properties, and kept inside the project root folder, and ran the analysis. but got an error below.
sonar-project.properties files are not understood by the SonarScanner for MSBuild. Remove those files from the following folders: D:\Jenkins\workspace\comm
have looked at this question
what i could infer from this post is, we cannot set sonar-project.properties for msbuild ? is it ?
So then how to do include/exclude files/folders from code coverage in sonar analysis ? I don't have admin access, to set administrative global exclusions
Any help ?
P.S using sonarqube LTS 6.7.6
After installing Taurus on Win 10 machine and creating a new file for performance testing my API:
perf-test-config.yml
execution:
- concurrency: 100
ramp-up: 1m
hold-for: 2m
scenario: helloworld-api-perf-test
scenarios:
quick-test:
requests:
- https://helloworld-api.cfapps.io
I get following error in log:
Error Log:
> bzt perf-test-config.yml
15:32:18 INFO: Taurus CLI Tool v1.11.0
15:32:18 INFO: Starting with configs: ['perf-test-config.yml']
15:32:18 INFO: Configuring...
15:32:18 INFO: Artifacts dir: C:\Users\chandeln\MY-WORK\helloworld-api\2018-03-29_15-32-18.609453
15:32:18 WARNING: at path 'execution.0.scenario': scenario 'helloworld-api-perf-test' is used but isn't defined
15:32:18 INFO: Preparing...
15:32:19 WARNING: Could not find location at path: helloworld-api-perf-test
15:32:19 ERROR: Config Error: Scenario 'helloworld-api-perf-test' not found in scenarios: dict_keys(['quick-test'])
15:32:19 INFO: Post-processing...
15:32:19 INFO: Artifacts dir: C:\Users\chandeln\MY-WORK\helloworld-api\2018-03-29_15-32-18.609453
15:32:19 WARNING: Done performing with code: 1
As in taurus User Manual scenario in execution should be also under scenarios for example:
execution:
- concurrency: 100
ramp-up: 1m
hold-for: 2m
scenario: helloworld-api-perf-test
scenarios:
helloworld-api-perf-test:
requests:
- https://helloworld-api.cfapps.io
You have only quick-test under "scenarios" dictionary and in "execution" block you're trying to call something named helloworld-api-perf-test
So the options are in:
Change scenario: helloworld-api-perf-test to scenario: quick-test
Change quick-test: to helloworld-api-perf-test:
More information:
Taurus Configuration Syntax
Navigating your First Steps Using Taurus
I use selenium web driver with junit in Eclipse. Advice me please how can I report test results?
I try make it in such way:
http://testerinyou.blogspot.com/2011/03/how-to-create-reports-using-junit.html?showComment=1342535781861#c6211334222001490686
But get errors during gereration:
Buildfile: E:\eclipse\WebDriver\build.xml
build-subprojects:
init:
build-project:
[echo] WebDriver: E:\eclipse\WebDriver\build.xml
[javac] E:\eclipse\WebDriver\build.xml:43: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
build:
Sector:
[mkdir] Created dir: E:\eclipse\WebDriver\junit
[junit] Running Sector
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
[junit] Test Sector FAILED
junitreport:
[junitreport] Processing E:\eclipse\WebDriver\junit\TESTS-TestSuites.xml to E:\DOCUME~1\CONSUL~1\LOCALS~1\Temp\null255203125
[junitreport] Loading stylesheet jar:file:/E:/eclipse/plugins/org.apache.ant_1.8.2.v20120109-1030/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs /optional/junit/xsl/junit-frames.xsl
[junitreport] : Error! The first argument to the non-static Java function 'replace' is not a valid object reference.
[junitreport] : Error! Cannot convert data-type 'void' to 'reference'.
[junitreport] : Fatal Error! Could not compile stylesheet
[junitreport] Failed to process E:\eclipse\WebDriver\junit\TESTS-TestSuites.xml
BUILD FAILED
E:\eclipse\WebDriver\build.xml:135: Errors while applying transformations: Fatal error during transformation
Total time: 4 seconds
There are different possibilities to create reports for Selenium WebDriver.
A user developed a small library to create a html report with screenshots and so on.
For further information look here: https://github.com/vorburger/webdriver-reporting
Furthermore you could use another framework based on Selenium called Thucydides. It generates really nice looking html-reports and improves Selenium WebDriver. You can visit the official website for more information (www.thucydides.info)
You could use Ant oder TestNG to create a report for your tests.
Look here for more information: http://selftechy.com/2011/06/29/selenium-use-ant-to-generate-html-reports
I like UnitTH to generate test reports.
You can start it via ant or maven
Further questions?