I'm trying to get reporting working for Karate DSL, and it's proven a challenge because my team uses Circle CI instead of Jenkins. Cucumber reporting seems to only work for Jenkins.
I've had a look at this documentation, here:
https://github.com/intuit/karate/tree/master/karate-demo#example-report
https://github.com/jenkinsci/cucumber-reports-plugin
I was wondering if there is a circle friendly equivalent you could recommend? It'd be even better if the reports could be generated in the terminal. It's going to be a hard sell to convince my team to change CI tools just so I can implement a test framework.
Thanks!
Here's what I suggest:
If you follow the demo / doc instructions - you will get the HTML reports in say target/cucumber-html-reports, and this is "pure Maven and Java", no dependency on CircleCI at all so far.
Now all you need to do is somehow make these HTML reports accessible via the web. In Jenkins, there is an HTML Publisher Plugin. I am not familiar with CircleCI but a quick search suggests that there is a way to expose links to build artifacts.
Also note that when you follow the demo, Java JUnit XML reports would also be output to target/cucumber-reports. It looks like CircleCI has support for these which means that it should be able to derive the build pass/fail status and stats if configured right.
Also note that Karate now enables you to write custom reports: https://stackoverflow.com/a/66773839/143475
Related
We are building a shop for a customer on Shopware 6.3.5.2 and want to use tests to
ensure that core functionality is not broken by our customizations (static plugins)
write new tests for new functionality
There is Running End-to-End Tests but this seems to be for core development and uses psh.phar which is not available in the production template.
How should this be done?
edit
This question is meant a bit broader and concerns also Unit Tests.
Actually, you can use the E2E tests of the platform project - as Cypress itself doesn't care where to run the test against. However, as you already noticed you cannot use psh commands to run them. You may run the tests though the basic Cypress commands, setting your shop's url as baseUrl of the tests, for example via this command:
./node_modules/.bin/cypress run --config baseUrl="<your-url>"
It works with cypress open as well.
The only thing what may become troublesome is the setToInitialState command in most of the tests which takes care about the clean up of shopware's database using psh scripts, unfortunately. You may need to adjust it by overriding the command in order to reset the database of the Production template.
I hope I was able to help a bit. 🙏
There are actually two parts here:
ensure that core functionality is not broken by our customizations (static plugins)
write new tests for new functionality
re 1: For regression tests like this I would suggest end-to-end tests. Either test through the UI with tools like selenium or through the HTTP API (I don't know if the shopware API is sufficient for extensive regression tests).
re 2: Since plugins do not run on their own I would extract all relevant functionality into plain old PHP classes that are independent of shopware and test those in isolation. Explore if some of that functionality can be made visible through an API and test the plugin integration through this. Depending on the actual plugin you might have to resort to UI tests again.
Is there any good way to get Karate tests to test a Spring-Boot microservice in the "integration-test" phase of a Maven build? What I mean is: is there an anti-pattern/workaround that works well OR am I out of luck because this was an afterthought of the Karate development?
Facts I have gathered so far:
It says here: "the surefire plugin is not hardcoded into Karate"
Is there a way to run Karate test during maven's integration test phase?
Running Karate tests in context of maven-failsafe-plugin does not work at all. Easy to reproduce on my test project url below. Two problems foremost:
When running from maven-failsafe, Karate will generate .txt and .html files but does not generate .json Cucumber output files.
Therefore, you can get no HTML report from the cucumber-reporting plugin (which I require)
I created a project to demonstrate these facts: https://github.com/djangofan/karate-api-sample
Running integration tests requires that I start Spring Boot before the test suite and then stop the spring boot server afterwards. Maven SureFire does not have this capability BUT the maven failsafe plugin does: the maven build helper plugin has a pre-integration-test hook for it. Limited only by Maven surefire, I am out of luck.
I see an example in the karate/karate-demo project of starting the spring boot server from code. This is not easy to accomplish when I am trying to test a web service that is already established. Is this my only option: to use surefire with a class filter and code to bootstrap?
Given no good answer, I will just revert to using cucumber-java + resteasy, which I am pretty sure will work with maven failsafe: https://maven.apache.org/surefire/maven-failsafe-plugin/examples/cucumber.html
First, this is not Karate.js not sure where that came from :)
When running from maven-failsafe, Karate will generate .txt and .html files but does not generate .json Cucumber output files.
Sorry, cannot replicate.
For failsafe, just use the naming conventions and it will work, e.g. **/IT*.java
Maven SureFire does not have this capability BUT the maven failsafe plugin does:
So use failsafe as mentioned above. If you are not able to do that, please change your project to demonstrate the issue and we promise to fix it. Also refer these instructions: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue
Given no good answer, I will just revert to using cucumber-java + resteasy
That is of course your choice. At least in my (biased :) opinion you will miss all the JSON assertion value that Karate adds, and the parallel execution + aggregation of reports. I am sure you will help as far as possible so that we can make the experience better for all who need this solution - but else, hey - no worries :P
As part of testing in the cloud, I plan to copy my test results (I happen to be using gradle) as xml out of a short lived container to an s3 bucket.
I am wondering what is the best approach for making use of the bucket full of test results and if anyone has ideas for how to get the most of these results with the least custom code, etc.
In the past I have handled this with XSLT to make the XML into HTML but that approach feels like more work than I'd like to have to do.
Has anyone solved this issue for their own purposes that can share your approach?
As I side note... I could also copy out to my s3 bucket the HTML reports from gradle if someone has an idea about how to use the HTML instead.
Ideally, I would be able to see a sorted list of results and click on the most recent to see the test run details.
Disclaimer: I'm not a Devops guy, I'll present a "developer's" point of view here
The answer IMO depends on what exactly would like to do with all these reports.
In a nutshell, you can have text/xml reports from the test execution (I can't say for gradle but in maven surefire plugin creates these reports, I believe gradle does the same).
You can also generate a good looking site with the reports information, there are tools for this as well, for example Allure Reports. Bottom line, some stuff to be shown can be generated in the build directory.
Copying the reports is as simple as copying a file, but the question is what would like to do with them when they're in S3?
In my understanding, its a Job for CI tools (like Jenkins) to run the tests during the build, or maybe a suit of automation tests against a deployed environment (in the cloud).
These CI Tools run gradle to build the project, and run tests, then they show test results per build memorizing last N builds. It can also be integrated with tools like aforementioned allure and show the HTML report per build as long as the build is stored in CI tool.
So I'm not sure I understand why would you like to "run tests in the cloud" (from which I assume you're running tests suite when the artifact is being deployed).
As for possible ideas of what can be done with all these results:
You can create a site that will show the results across builds (again, this is
already solved by CI tool like Jenkins)
You can store the results of running the tests in some database and provide some
clever statistics (I personally don't see much benefit if usually the build is
green and you're not in the process of "recuperation" of project from flaky tests or
something)
I want to know, if we can use karate with JS based UI automation framework. And if yes, what are the requisite to follow. Do we require maven in that case?
We have well matured UI automation Selenium based framework. We want to know, if we can integrate Karate DSL without using maven.
You might if you use the CLI (command line interface) provided by the stand-alone JAR / executable. Please read the documentation here: https://github.com/intuit/karate/tree/master/karate-netty#standalone-jar
Which means if you can call an OS command from node / JS - you can use Karate. But there are no guarantees - and you can assume the answer to your question is NO.
That said - we plan to have our own competitor to Selenium - but it will take a little more time: https://github.com/intuit/karate/tree/master/karate-core
I am trying to integrate Allure2 reports with Jira plugin called Xray. So far i managed to generate those reports on local machine and on Jenkins server in pipeline job with Allure plugin.
Also i see that there is an actual Allure plugin called xray-plugin on Allure Framework GitHub, however instructions how to use this plugin are not clear enough for me (for example, where one should put those Jira env variables?).
Can someone here explain in more clear manner how to integrate Allure2 with x-ray?