After executing our test automation we are sending the results into Rally. Here are the steps that we are following:
Create new Test Set
Add Test Case to Test set - The test case
already exists in a test folder
Query for the test case within the test set - We added this additional read since we were receiving concurrency issues trying to immediately post results after adding the test case to the testset
Post the results to the testcase within the testset.
The issue we are running into is in step 3 above. Sometimes, when a read is done to find the test case in the test set, it is unable to find it and so the results cannot be posted.
We have actually adjusted the code so it attempts 4 times with a 20 second delay between the third and 4th attempt and there are still several where results cannot be posted to Rally.
If anyone else has run into this issue and could provide guidance on a resolution that would be great
Related
I have the xUnitFileImport scheduled job configured in my polarion project (as described in Polarion documentation) to import e2e test results (formatted to JUnit test results)
<job cronExpression="0 0/5 * * * ? *" id="xUnitFileImport" name="Import e2e Tests Results" scope="system">
<path>D:\myProject\data\import-test-results\e2e-gitlab</path>
<project>myProject</project>
<userAccountVaultKey>myKey</userAccountVaultKey>
<maxCreatedDefects>10</maxCreatedDefects>
<maxCreatedDefectsPercent>5</maxCreatedDefectsPercent>
<templateTestRunId>xUnit Build Test</templateTestRunId>
<idRegex>(.*).xml</idRegex>
<groupIdRegex>(.*)_.*.xml</groupIdRegex>
</job>
This works and I get my test results imported into a new test run and new test cases are created. But if I run the import job multiple times (for each test run) it creates duplicate test case work items even though they have the same name, which leads to this situation:
Is there some way to tell the import job to reference the existing testcases to the
newly created test run, instead of creating new ones?
What i have done so far:
yes I checked that the "custom field for test case id" in the "testing > configuration" is configured
yes I checked that the field value is really set in the created test case
The current value in this field is e.g. ".Login" as i don't want the classnames in the report.
YES I still get the same behaviour with the classname set
In the scheduler I have changed the job parameter for the group id because it wasn't filled. New value is: <groupIdRegex>e2e-results-(.*).xml</groupIdRegex>
I checked that no other custom fields are interfering, only the standard fields are set
I checked that no readonly fields are present
I do use a template for the testcases as supported by the xUnitFileImport. The testcases are successfully created and i don't see anything that would interfere
However I do have a hyperlink set in the template (I'll try removing this soon™)
I changed the test run template from "xUnit Build test" to "xUnit Manual Test Upload" this however did not lead to any visible change
I changed the template status from draft to active. Had no change in behaviour.
I tripple checked all the fields in the created test cases. They are literally the same, which leads to the conclusion that no fields in the testcases interfere with referencing to them
After all this time i have invested now, researching on my own and asking on different forums, I am ready to call this a polarion bug unless someone proves me this functionality is working.
I believe you have to set a custom field that identifies the testcase with the xUnit file you're importing, for the importer to identify the testcase.
Try adding a custom field to the TestCase workitem and selecting it here.
Custom Field for Test Case ID option in settings
If you're planning on creating test cases beforehand, note that the ID is formatted form the {classname}.{name} for a given case.
I have multiple specs so i created a suite for different specs.
Let's take the below scenario.
this is my suite structure in the conf file.
suites:{
forms:['specs/requestE.js'],
search:['specs/findaSpec.js'],
offers:['specs/offersPrograms.js','specs/destinationsSpec.js'],
headerfooterlinks:['specs/footerlinksSpec.js','specs/headerMenuSpec.js']
},
When I run each spec individually it works correctly and generates the test results, but when I run the whole suite only the first one is working, others are not getting executed. As a result it gives timeout error.
Do you have any test cases in first spec with fit('', function(){}) instead of it('', function(){}) ?
if thats the case, it'll just execute one spec while ignoring the rest
What will be the test case for the condition:
On some criteria i.e 'resolution still not taken' ,the user with this criteria is blocked to visibility of an issue information?
Test case can be like this:
Steps:
1) Open the application.
Expected result: Application should get opened successfully.
2) Open the issue resolution status module.
Expected Result: Issue resolution status module should get opened successfully.
3) Check the status of issue.
Expected Result:
If status of issue is 'resolution still not taken' then user with this criteria is blocked to visibility of this issue.
If status is something else then do whatever your application says to do.
Note: I just mentioned raw structure of test case. You can write this in format which you use for your test case.
I am trying to build an automation framework on Rally. I am using Rally RESTful API and Pyral. The ranking method of the workspace I am working in is DnD method. I am able to copy the test set and its test cases but unable to maintain the original ranking of test cases within test set.
I tried to GET the test cases of a specific test set with order = 'DragAndDropRank' but it only gives me the same order as FormattedID of the test case. Ex:
query_criteria = 'TestSets = "%s"' % str(ts._ref)
response = self.rally.get('TestCase', fetch=True, query=query_criteria, order='DragAndDropRank')
for tc in response:
....
Let's say there are three test cases TC1, TC2, TC3 within the test set TS1. In the "Iteration Test Case Status" view of Rally, the order of test cases in TS1 is: TC3, TC1, TC2. However, the order of tc generated from code above is always TC1, TC2, TC3.
How can I get the original relative order of test cases within the test set in the "Iteration Test Case Status" view of Rally?
It is a defect that was first mentioned in this post.
Unfortunately the defect is still open.
I am using clj-webdriver to fill out a form in my test case.
(quick-fill-submit {"#firstname" "Firstnam"}
{"#lastname" "Lastnaem"}
{"#username" "Username"}
{"#password" "Password"}
{"#password2" "Password"}
{"#roles" "user"})
(click "button#add-user")
Every time I run this code in my test case the third value is filled in blank.
I moved the fields around and verified it. It is always the third field.
When execute my test case step by step in a repl it works fine but when
running it through lein test it fails.
This seems to be some kind of timing issue. When I for example stall the
execution by adding a
(wait-until #(= 1 2) 10000)
between the two functions the field gets filled. A simple
(Thread/sleep n)
does not work in this case. Why is Selenium not filling in the form correctly?
WebDriver and AJAX calls usually require tweaking the wait settings. You should try setting implicit-wait to something bigger than 0 (which is the default). Another option would be to use wait-until and a predicate that checks for the presence of the elements.