Is it possible to generate auto API test report using Datadog? :)
I'm creating a project in Java using Rest Assured and Hamcrest. I have the project on gitlab and I would like the tests to be automatically run. Based on them, I would like to receive a report :).
At first I through I could use a allure but my supervisor asked me if I could do it in Datadog.
I tried to find background material on this subject, but failed :(
I will be grateful for every link with supporting materials :)
What do you use for build tool maven or gradle?
I think this should work for you:
https://docs.datadoghq.com/continuous_integration/tests/java
Related
We have QA and DEV environment in our automation repo. We are using karate as our framework. We have TestParallel class and integrated allure report.
How could we run all tests in QA first then in DEV back to back using TestParallel Class and see the results in the same report?
Thanks for such a great tool btw.
We are going to try and make this easier in the next version.
For now, you have to aggregate the reports yourself. Can you try this and let us know how it goes.
use the Runner class 2 times to run your tests with different settings and karate.env set for QA and then DEV
the important part is using a different value for the workingDir, e.g. target/reports/qa and then target/reports/dev - else the second run will overwrite the first
now when generating the HTML report, you can provide target/reports as the source folder. this should work for the Maven Cucumber Reports, for Allure, please figure this out on your own
if the above approach does not work well enough for your needs, please figure out a way to manually aggregate the Results object you get from each instance of the Runner, this should not be too complicated as Java code
I'm new to DevOps work and I've been tasked to setup a build for one of our .net Standard Library projects. I used the build template for this, which works great! But I want to add a code coverage into build output (there's a test project in the solution).
In the "VsTest - testAssemblies" task, I've checked the "Enable Code Coverage" option, as shown below:
I added a Publish code coverage task and it's asking for something called a "Summary File" - see below:
I'm not sure how to set this option? Has anyone done this?
I seem to get test results published in the build results without config, see below:
And I'd like the same for the Code Coverage tab in the build results, which is currently blank:
Also (may be a completely separate question), I was wondering - can I add a "Quality Gate" to say if Coverage is below 80% then fail the build?
Thanks for any advice in advance - it's a case of knowing what I want, but not sure how to achieve it!
Hurrah!!!
There is a solution for it. :)
You have to install a an Add-on to your organization in VSTS (Azure DevOps).
The name of the addon is Build Quality Checks
Here is the link:
https://marketplace.visualstudio.com/items?itemName=mspremier.BuildQualityChecks
It works perfectly. And this is what you want
By default, the VsTest task will publish the Code Coverage result to Build Summary page directly. So we don't need Publish Code Coverage result task to publish the coverage result. Just make sure you have selected "Code Coverage enable" section in VsTest task.
>> Also (may be a completely separate question), I was wondering - can I add a "Quality Gate" to say if Coverage is below 80% then fail the build?
In current VSTS, we could not set "Quality Gate" to set the build failed if it doesn't match the configured % value. There has other communities also have this requirement and have submit a user voice. Please feel free to add your comments or vote it from below link:
https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services/suggestions/3817520-fail-build-on-insufficient-code-coverage
Where can i save my testNG reports for analysis later.
My regression suite will be running daily and I want to store the reports for anaysis and comparitive study later.
What can i use for the same?
Allure, ReportPortal or maybe some other tools could help you.
There is no such repository. You can go ahead and build a customized org.testng.IReporter which persists the test results to a database which you can fetch and run your analysis.
As suggested you can go database..
Generally i used custom HTML reports for suite and links to Tests HTML reports. I will create dir with data and time stamp for every run and move to shared location to keep and share.
As I see in version 3.3 labels feature were provided. I would really love to use them but it's currenty impossible because lack of handling labels in ant database tasks. Is it handled in some other way ?? Additionally i would like to know how could I add an issue to liquibase jira.
To add something to the Liquibase Jira, you just need to sign up for an account here:
https://liquibase.jira.com/login?
After your account is created, you will be able to create new issues.
As far as the Ant tasks go, it appears that you should be able to use labels and contexts in the Ant tasks. If you could include an Ant build file that shows any problems you are having along with the command you are using and the output, we can help address those issues.
I have a process for running automated functional tests which is external to Microsoft Team Foundation Server (TFS) 2010. Test cases are tracked as Test Case work items within TFS, however. After running these tests, how can I publish the results to TFS using the TFS API? Can someone point me to sample code that demonstrates this?
Please note that I expressly want to avoid a solution that requires transformation of my test results into the .trx file format. Searches have turned up dead links, or solutions that rely on this method.
It sounds like the following blogs may almost be what you're after
http://blogs.msdn.com/b/jpricket/archive/2010/02/23/creating-fake-builds-in-tfs-build-2010.aspx
http://msmvps.com/blogs/vstsblog/archive/2011/04/26/creating-fake-builds-in-tfs-build-2010-using-the-command-line.aspx
It doesn't actually have code for adding test results, however it does say this:
"In order to associate test results and the like, you have to create build project nodes with the fake build."
You should be able to create a Microsoft.TeamFoundation.Build.Client.TestSummary with a summary of your test results.
There's a couple of internal classes that look interesting, specifically Microsoft.TeamFoundation.Build.Controls.TestRunDetails, which could potentially be useful if you don't mind using some reflection.
However what I would recommend, is using the API to look at the nodes in a standard TFS build to see how they are built up.