If I am withing a test case in VSTS is there any way I can mark this test "Passed" or "Failed" without going back out to the Test Hub?
No, we cannot mark the outcome to "Passed" or "Failed" within test case itself.
Test case is a work item type, a test case is just a work item and the work item itself doesn't have a field with the value "Passed" or "Failed".
The outcome "Passed" or "Failed" is just for test result, so you have to make a test run to pass or fail the test case. You can check that a new test run will be generated when you mark a test case to passed or failed in Test Hub. (Just check that under Test --> Runs)
Related
I am generating cypress report using mochawesome, which is default UI.
I need to make it custom.
Heading - Test Summary
Number of Total Pass case
Number of Total Fail Case
On click on Total Number of pass test case , accordion will be open and will display Test Name and time which took to run test case
On click on Total Number of failed test case , accordion will be open and will display Test Name and time which took to run test case
and reason of failed
In Robot Automation, how to re-run the failed test case immediately if it is failed, before going to another test case execution.
For instance,
*** Test Cases ***
Login User And Create Another User
Login User ....
Create Another User ...
Login With New User
Login User..
Test Function ABC
.....
.....
Since one test has a dependency on another test, I need to re-run the failed case immediately after it is failed. Before executing another test.
In one word, you can't, and you shouldn't; a case is a case, with binary outcome. And if you have dependencies between tests, that's a smelly design; try to change it to a pre-condition (env setup) for the second case, so it is atomic.
Disclaimer: this rant is for the automatic re-execution in a single run. After a run has finished, RF has baked-in functionality to re-execute just the failed ones (so flaky tests are given the chance to succeed); but as I understood your question, you are not asking for the latter.
In two words, if you really need to do it, you can; extract the whole test case in a keyword, and call it inside Wait Until Keyword Succeeds, giving it 2 (or more?) attempts:
*** Test Cases ***
Test Function ABC
Wait Until Keyword Succeeds 2 times 100ms The Actual Test For Function ABC
*** Keywords ***
The Actual Test For Function ABC
.....
.....
Karate step execution stops when any one of the step fails.
Example:
Scenario : verify user details.
Given url "this is my webservice"
When method post
Then status 200
*assert 1==2
Then response
Then match XXXXXXX
The match XXXX
The steps fails Assert , remain steps does not execute. Is there any way even my assert fails remaining steps can continue the process
This is the expected behavior.
But you can use the karate.match() function to perform the assert manually. Then you can use conditional logic to decide if you want to continue next steps or not. But I totally don't recommend this.
For example:
* def temp = karate.match(actual, expected)
* print 'some step'
* assert temp.pass
Is it possible to handle exceptions from the test case? I have 2 kinds of failure I want to track: a test failed to run, and a test ran but received the wrong output. If I need to raise an exception to fail my test, how can I distinguish between the two failure types? So say I have the following:
*** Test Cases ***
Case 1
Login 1.2.3.4 user pass
Check Log For this log line
If I can't log in, then the Login Keyword would raise an ExecutionError. If the log file doesn't exist, I would also get an ExecutionError. But if the log file does exist and the line isn't in the log, I should get an OutputError.
I may want to immediately fail the test on an ExecutionError, since it means my test did not run and there is some issue that needs to be fixed in the environment or with the test case. But on an OutputError, I may want to continue the test. It may only refer to a single piece of output and the test may be valuable to continue to check the rest of the output.
How can this be done?
Robot has several keywords for dealing with errors, such as Run keyword and ignore error which can be used to run another keyword that might fail. From the documentation:
This keyword returns two values, so that the first is either string
PASS or FAIL, depending on the status of the executed keyword. The
second value is either the return value of the keyword or the received
error message. See Run Keyword And Return Status If you are only
interested in the execution status.
That being said, it might be easier to write a python-based keyword which calls your Login keyword, since it will be easier to deal with multiple exceptions.
You can use something like this
${err_msg}= Run Keyword And Expect Error * <Your keyword>
Should Not Be Empty ${err_msg}
There are couple of different variations you could try like
Run Keyword And Continue On Failure, Run Keyword And Expect Error, Run Keyword And Ignore Error for the first statement above.
Option for the second statement above are Should Be Equal As Strings, Should Contain, Should Match.
You can explore more on Robot keywords
I need to call two teardown keywords in test case but must not create new keyword for that.
I am interesting if there is a such syntax for keywords as for documentation or loops for example:
[Documentation] line1
... line2
... line3
Use the "Run Keywords" keyword.
From doc "This keyword is mainly useful in setups and teardowns when they need to take care of multiple actions and creating a new higher level user keyword would be an overkill"
Would look like that:
Test Case
[Teardown] Run Keywords Teardown 1 Teardown 2
or also
Test Case
[Teardown] Run Keywords Teardown 1
... Teardown 2
and with arguments
Test Case
[Teardown] Run Keywords Teardown 1 arg1 arg2
... AND Teardown 2 arg1
For executing multiple keywords in Test Teardown method use the following trick:
Firstly, define a new keyword containing the set of keywords you want to execute.
E.g. here Failed Case Handle is a new definition of the other two keywords take screenshot and close application. Consider this is to take a screenshot and then close the running application.
*** Keywords ***
Failed Case Handle
take screenshot
close application
Basically, when you call the Failed Case Handle keyword, take screenshot and close application will be executed respectively.
Then, in the ***Settings*** section define the Test Teardown procedure by the following example.
*** Settings ***
Test Teardown run keyword if test failed Failed Case Handle
or,
*** Settings ***
Test Teardown run keyword Failed Case Handle
So, in the first case Failed Case Handle keyword will be called if any test case fails. On the other-hand in the second case Failed Case Handle keyword will be called after each test cases.