I'm trying to get allure reporting working as part of an NUnit project in C# using Selenium WebDriver. Following the documentation for installing allure seems to work fine on a local machine but I'm trying to get the Team City plugin to also work this the project. I've uploaded the allure team city plugin and added a build step that will generate a report even if the build fails or is stopped. The project has all the required allure nunit tags added and again this generates results locally that a report can be generated from. However, when running tests from Team City the build hangs and does not even begin to run tests. At the step that it should start running tests it just sits there. The Team City build logs do not show anything wrong. I'm using NUnit 3.6, Team City 2018.1, allure 2,7, allure team city plugin 2.9.
Does anyone have any experience with allure reporting? The documentation is a little out of date but I've done as much as I can with it.
I'm not sure if this is allowed, answering your own question, but in light of little on the ground of existing responses given the Allure developers actually suggest posting to this site as opposed to posting to their own Git hub site, I find it rather frustrating that there has been little response.
Anyway, having spent further hours looking into this I believe that the hanging of the builds is purely down to the allure-config.json file and the path to the allure-results setting within it. Any attempt to change that setting from its default appears to instantly cause the TC builds containing Allure Reports to hang.
hopefully this may help others having similar issues in future.
Related
On running Selenium Test through Gitlab CI/CD Pipeline, Screenshots are not displaying within Extent Test Report. Test Report and Screenshots folder are getting generated separately as an artifact there but on running in local, it is displaying within test report.
I have tried base 64 encoded also for taking screenshots but it is not working. As path will change as it runs on different server when we run automation suite on Gitlab CI/CD pipeline, therefore I guess screenshots are not getting displayed within extent report.
Kindly please suggest what we can do here.
See GitLab 13.12 (May 2021)
Failed test screenshots in test report
GitLab makes it easy for teams to set up end-to-end testing with automation tools like Selenium that capture screenshots of failed tests as artifacts. This is great until you have to sort through a huge archive of screenshots looking for the specific one you need to debug a failing test. Eventually, you may give up due to frustration and just re-run the test locally to try and figure out the source of the issue instead of wasting more time.
Now, you can link directly to the captured screenshot from the details screen in the Unit Test report on the pipeline page. This lets you quickly review the captured screenshot alongside the stack trace to identify what failed as fast as possible.
See Documentation and Issue.
As part of testing in the cloud, I plan to copy my test results (I happen to be using gradle) as xml out of a short lived container to an s3 bucket.
I am wondering what is the best approach for making use of the bucket full of test results and if anyone has ideas for how to get the most of these results with the least custom code, etc.
In the past I have handled this with XSLT to make the XML into HTML but that approach feels like more work than I'd like to have to do.
Has anyone solved this issue for their own purposes that can share your approach?
As I side note... I could also copy out to my s3 bucket the HTML reports from gradle if someone has an idea about how to use the HTML instead.
Ideally, I would be able to see a sorted list of results and click on the most recent to see the test run details.
Disclaimer: I'm not a Devops guy, I'll present a "developer's" point of view here
The answer IMO depends on what exactly would like to do with all these reports.
In a nutshell, you can have text/xml reports from the test execution (I can't say for gradle but in maven surefire plugin creates these reports, I believe gradle does the same).
You can also generate a good looking site with the reports information, there are tools for this as well, for example Allure Reports. Bottom line, some stuff to be shown can be generated in the build directory.
Copying the reports is as simple as copying a file, but the question is what would like to do with them when they're in S3?
In my understanding, its a Job for CI tools (like Jenkins) to run the tests during the build, or maybe a suit of automation tests against a deployed environment (in the cloud).
These CI Tools run gradle to build the project, and run tests, then they show test results per build memorizing last N builds. It can also be integrated with tools like aforementioned allure and show the HTML report per build as long as the build is stored in CI tool.
So I'm not sure I understand why would you like to "run tests in the cloud" (from which I assume you're running tests suite when the artifact is being deployed).
As for possible ideas of what can be done with all these results:
You can create a site that will show the results across builds (again, this is
already solved by CI tool like Jenkins)
You can store the results of running the tests in some database and provide some
clever statistics (I personally don't see much benefit if usually the build is
green and you're not in the process of "recuperation" of project from flaky tests or
something)
I am having an issue getting selenium end to end tests to work after an automated deployment using visual studio team services (VSTS).
I have a build working that generates a build artefact. This is triggered from VSTS but runs on an on premises build server. I have a deployment working that deploys to an on premises development web server. All this works including unit tests running after the build.
When I try to add testing after the deployment is when I run into the problem. The tests are to be run on the build server and point to the dev server website. The deployment has two phases. A deploy and then an agent phase that runs a test assemblies task using the build agent on the build server. The problem seems to be that the test dll's are not being included in the build artifact and so are never found when the test process runs. Deploy setup us as follows.
I have a copy files before the publish artifact in the build definition that seems to copy the files in to the right place but they are not included in the zip file artefact. I've looked at several websites and posts on here but I still seem to be missing a vital bit of knowledge that will get this working.
Use of the log did help as recommended so thanks for that.
I have managed to get this working. I separated the selenium tests out into a separate solution and built that separately creating it's own build artefact. I then added this to the agent task in the deploy. This worked. The only thing I need to get sorted now is the correct search path to find the test DLL's. It's not quite as dynamic as I would like at the moment. I can play tunes on that until I get it right though.
I accept this is working around rather than solving the original problem but needs and timescales must. I think moving the end to end UI tests out of the main solution makes sense anyway but no doubt others may think differently.
Thanks for your help everyone
Our goal is to implement CI testing and deployment for our DEV web environments:
Goal
Run XUnit tests on check-in.
If tests fail, create individual, associated Bug work items. Stop.
If tests tests pass, deploy build to a UNC file path.
Current Setup
CI is on for the branch, and the build definition currently has enabled Create Work Item on Failure on the Options panel.
XUnit was integrated into the Visual Studio Test build step by providing the Path to Custom Test Adapters necessary.
Problem
Tests run and display results correctly in the build, but no bugs are created for the failed tests, only one for the overall build fail.
Question
How can I create individual Bugs (and include details about the bug in its description)?
You would have to write your own code to create Bugs for each test failure.
I would however recommend against it as this creates unessesery work items and they may not really be bugs. Maybe we have a single test that fails, and the other 200 tests fail as a result. We only have one bug. You will overwhelm people.
You can easily create bugs as you investigate failures using the failed test list that is part of the build results.
https://www.visualstudio.com/en-us/docs/test/continuous-testing/getting-started/getting-started-with-continuous-testing
Our department uses Visual Studio 2008 Team System, and we have a build server that integrates with our TFS source control server. It pulls the source code, builds the solution and runs the unit tests, just as we might do from within VS, and emails a report. The build server is setup using MSBuild and MSTest as the primary tools. All very sweet.
On our development machines we also run a set of selenium unit tests, and I want to include this in the test suite on the build server. I have been told that 'this is not possible using MSBuild/MSTest', but I am at a loss to understand why.
Does any one have experience of running selenium tests (they are just conventional test methods written in C#) who might be able to advise me on whether this is possible and what the gotchas are? Thinking about it, apart from giving the browser access to the desktop when the server is not logged in, once MSBuild has handed off a test list to MSTest it's exactly the same process as on our develop machines.
TIA
I know it's 3 years on, but someone might drop in on this post and not see an answer. This is possible todo.
In a similar fashion how you'd run unit test, in the build definition using default template you need to specify the name of the UnitTest DLL and ensure Run Unit Test is not disabled. Also ensure that the build is building your Automated UI Test solution.
Simples.