I am trying to create a smoke test for a webserver using a webtest and tfs build 2015. I have a webtest that uses a Datasource(n rows) to run n times. I also have a Context Parameter used to configure the test to various web urls. I used an orderedtest to contain the webtest. I can run the orderedtest using the Visual Studio Test Task. Unfortunately, I needed to create multiple file copies of each webtest and orderedtest because I cannot figure out how to pass a Context Parameter to an orderedtest/webtest.
Environment variables whose name starts Test. are added to the web test as context parameters. If environment variable Test.Abcd has the value efgh then the context parameter Abcd will to be set to efgh. See here for more details.
Related
Trying to set up Azure Devops pipeline for test automation run (Java + Maven + Selenium) tests are running on Browserstack
To view Browser Stack Results I added two tasks BrowserStackConfig and BrowserStackResults to pipiline YAML file
(according to instructions from BrowserStack)
But I am getting unexpected error on BrowserStackResults step : 'Build not found by name ..'
Now I am trying to get results for "old" automation run on BS (Set BROWSERSTACK_BUILD_NAME to existing BS results = 20220114_666666)
Setup:
Using Azure Devops with installed Browserstack extension
Steps:
Prepared YML file
Add pipeline for YML
Run pipeline
Result:
BS Configuration step works
Unexpected error on BrowserStackResults step "##[error] Build not found by name: "20220114_666666" error
The following only fixes BrowserStackConfig task:
variables:
- name: BROWSERSTACK_BUILD_NAME
value: "test-001"
But not for BrowserStackResults task. Is there another environment variable that controls the later? Or a list of variables used by browserstack?
According to the Browserstack
Note: Ensure to set the build capability in your test script using the environment variable BROWSERSTACK_BUILD_NAME. The extension will fail to embed test reports in your pipeline if this capability is missing.
Set variables for BrowserStack Azure DevOps Extension
The BrowserStack Azure DevOps extension, by default, sets the following environment variables:
BROWSERSTACK_USERNAME
BROWSERSTACK_ACCESS_KEY
BROWSERSTACK_LOCAL
BROWSERSTACK_LOCAL_IDENTIFIER
BROWSERSTACK_BUILD_NAME
When you create a service connection, the BROWSERSTACK_USERNAME and the BROWSERSTACK_ACCESS_KEY are automatically added as variables during the configuration step.
The extension also adds the BROWSERSTACK_BUILD_NAME variable that autogenerates a build name for your test runs.
Edit your test script to call environment variables
Edit your test script to add the environment variables for setting capabilities using the following code snippets:
String buildName = System.getenv("BROWSERSTACK_BUILD_NAME");
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability("build", buildName); // CI/CD job name using BROWSERSTACK_BUILD_NAME env variable
Seems you need to check your code if BROWSERSTACK_BUILD_NAME environment variable value is passed to buildName and added to capabilities. Otherwise, the report won't be generated
Locally, I have successfully implemented the interface ITestEventListener within a C# netcoreapp3.1 csproj. However, when the tests are an within a Jenkins Pipeline, things appear to not be working(?).
I am using version 3.12.0 of NUnit.Engine.
By locally I am referring to using 1) Visual Studio Version 16.9.2 to run the tests and 2) command line dotnet test -c devint --test-adapter-path:. --logger:nunit to run the tests. I am getting successful test runs.
Success is my [Extension]public class ReportTestListener : ITestEventListener {...} generates an html file. I am to see the html file is created locally whereas from the Jenkins Pipeline the html file is not generated.
Within the Jenkins Pipeline, I am using the command sh "dotnet test -c ${env.TARGET_ENV} --test-adapter-path:. --logger:nunit" where env.TARGET_ENV resolves to devint. I know tests successfully run within the Jenkins Pipeline since the NUnit test results file is generated/published.
What I am not sure of is how to test/validate that the ReportTestListener is being called within the Jenkins Pipeline. I know that testing frameworks such as NUnit uses refection to identify test classes and methods. I am presuming that also happens with my implementation of [Extension]public class ReportTestListener : ITestEventListener {...}. Ideas/Suggestions on how to validate that ITestEventListener's method void OnTestEvent(string report) is being called besides writing out to disk?
Changed file writing text path to use / instead of \.
Was looking for a solution to run a feature file at the end of the suite
My workflow (In parallel Run)
karate.callSingle('Login.feature') so at the beginning i do one
login and then use the cookies/token for the whole suite
Run tests in Parallel
Runs the Logout.feature file
There is no direct support for this currently. By the way no one has ever requested this. If this is so important, kindly open a feature request.
One workaround is to set a singleton / Java static variable from callSingle and then in your JUnit / Java parallel runner, call the feature to logout using the Java API (search the docs for this) and you can pass arguments / access the static variable.
EDIT: just realized that the #AfterClass JUnit annotation may be more than sufficient for your needs.
Been scratching my head for a while with this and I thought I would ask here. I am trying to run automated tests with tcm via command prompt. I have set up and configured a test controller and agent. Tcm is recognizing all the tests from tfs when I list etc. Created a test run and the results came back:
Total:128
Passed:0
Failed:128
Inconclusive:128
Does this have something to do with the Controller or Agent not being set up properly? Would tcm recognize the tests if they were not set up properly? I have set up a lab environment and I can access the controller PC.
Any help would be greatly appreciated.
Thanks
Russ
Based on the information that "The build directory of the test run either does not exist or access permission is required", from steps 6 and 7 of this MSDN documentation:
Create and Define a build definition
Copy a build directory with the relevant information into this solution
This may also have relevant information if the above doesn't work
I am trying to set up two specific PHPUnit test environments for Symfony2.
I am using Ant to run PHPUnit, so when I run the two commands below I would expect the following results:
ant test
Runs the test suite using MySQL databse. (matching staging and production envs)
ant ramdisk
Runs the test suite using a SQLite ramdisk. (super fast!)
I can figure out how to set up the MySQL & SQLite for PHPUnit individually, quite easily.
How do I get PHPUnit to specify a particular environment to use?
Our base class for testing sets up the environment, but I cannot figure out how to get a argument passed into here from PHPUnit so that I can conditionally set the environment.
I have tried a different bootstrap for ramdisk, but since the base test class sets the environment I could not make much progress.
Any ideas?
I eventually solved this.
PHPUnit can pass variables through to Symfony: http://phpunit.de/manual/3.7/en/appendixes.configuration.html#appendixes.configuration.php-ini-constants-variables