Azure Pipelines has a way of displaying test results after builds, described here. This seems to be very limited, however, and doesn't display much of the stuff we want.
We're using NUnit 3 with a .NET Core project, and NUnit outputs a bunch of stuff for test results. We specifically want to display FQN of the test at a minimum, but being able to easily view tests by category would suffice as well. This doesn't seem to be possible with the default Tests tab.
My question: Does DevOps support doing something like this or do I need to make my own site/API for viewing test results? I'm open to pretty much anything, we just want to avoid anything that requires manual offline processing.
As per the Microsoft comment to the question, this is not currently possible. I added a suggestion on their tracker here.
Related
Apologies, if this has already been asked, but please bare with me, i am new to restSharp and API testing. I have ready many posts,articles and responses. But due to my limitation, I was wondering if anyone can just guide me through, on how to achieve it.
What i would like to achieve, is to run the same test scenario in different test environments(dev, uat, pro etc..) for the same Gherkin scenarios, if possible by making use of the tags e.g #dev #uat etc..
Currently, we have 3 runsetting files for each of these environments, (dev.runsetting, uat.runsetting etc..in each we pass different parameters), to change to different environment, i do this through C# and select the necessary test setting files
If I understood you correctly, you want to control via tags which scenario is executed against which environment.
The SpecFlow+ Runner has a feature called Targets that could help you. With this, you can define multiple targets and their configuration, for your scenarios.
You can find an example for this feature here: https://github.com/SpecFlowOSS/SpecFlow.Plus.Examples/tree/master/SeleniumWebTest
In this example, we are using the targets to test against different browsers.
In your case, I would change the browser to your environments.
In your CI/CD pipeline, you could then use the filtering in the SpecFlow+ Runner to filter out for your different environments. In Azure DevOps the filter criteria is Name~\(target: <TARGETNAME>\).
Documentation for it is here: https://docs.specflow.org/projects/specflow-runner/en/latest/Profile/Filter.html
Full disclosure: I am one of the maintainers and developers on SpecFlow and SpecFlow+
We are three testers and going to prepare automation project with selenium and java code so what are the steps for environment setup , scripts integration and running the testcases and getting the results for the whole project suits
So there are a few things we have to use in order to allow multiple engineers to work on the same framework.
Step 1) Creating the framework, assuming you know how to do this already, you have working tests you can skip this stage, however if not please follow the tutorial i link below.
http://toolsqa.com/selenium-webdriver/
Step 2) Creating a REPO, my preference is GitHub, you can use any git repo however i will post the guide to set one up with GitHub, its a similar process for all. This will allow you to merge code properly without causing conflicts.
https://help.github.com/articles/create-a-repo/
Step 3) Source Control program - to push, pull and fetch from your GitHub Repo, you can do this from Command Prompt however i find cloning the repo into a program like 'SourceTree' is really easy, so i've posted that below.
https://confluence.atlassian.com/get-started-with-sourcetree
If you follow these 3 guides, you will be able to have your automation test scripts on GitHub by the end of the day.
If you have any more questions please do not hesitate to ask.
All the best, Jack
The easiest and most logical way to do this would be to create one branch in your CVS (git or SVN, etc) and have each person setup the dev environment in the same way. Work exactly like developers and pull code before you check-in/commit (this will ensure that one small error does not break your framework) and swear to resolve conflicts during merge (to ensure you don't step on each others' toes).
Also, before you kick off, agree on a standard of coding (including package naming, design pattern usage, filename and methodname usage) and if this is in sync with the dev coding standards in your company, even better.
There will be a few hiccups along the way, but experience is the best way to create a process for your development and check-in practices.
Good luck with your new project and happy coding!
You have asked two questions, in my opinion the answer of your questions is.
how multiple automation testers work in same selenium project - You can use any version control system, Git Hub is the best option which gives you a lot of facilities. You all three can work on same project at same time or you can go for any centralized version control system like tortoise svn which is not much likely used now a days. I will suggest Git Hub for that.
what are the steps for environment setup , scripts integration and running the test cases and getting the results for the whole project suits - It depends on various factors like application and the kind of framework you want to use, there are many frameworks which are widely used for automation testing like Modular Framework, Data Driven, Keyword Driven, BDD, Cucumber, TestNg etc or if you have bandwidth and time you can design your custom framework as per the needs.
I hope I put some glimpse on your queries.
Thanks
I've managed to get some project's task list thanks to the Gradle tooling API GradleProject.getTasks(). It's kinda cool, I can read task's name, description, group and whether it's public or not.
I was wondering if it was possible to get tasks outputs directory, especially for tests or code coverage stuff, the kind of tasks that produce HTML-like reports. It would be nice to display these reports in a web UI.
Does anyone know if this is possible, or at least planned to be added in a future release of the tooling API ?
Thanks alot :)
In order to get additional information about tasks like the TaskOutputs from tooling api you, would have to implement a tooling API plugin like this:
https://github.com/bmuschko/tooling-api-custom-model
See here: https://github.com/bmuschko/tooling-api-custom-model/blob/master/plugin/src/main/java/org/gradle/sample/plugins/toolingapi/custom/ToolingApiCustomModelPlugin.java#L31-L39. This method is where you can collect the information you are interested it and surface that in your "model" class.
I have successfully done this for one of the projects I work on: https://github.com/liferay/liferay-blade-cli/tree/master/gradle-tooling/src/main/java/com/liferay/blade/gradle/tooling
If my understanding is correct, currently Gradle tooling API doesn't support HTML-like reports. There reports should be implemented by the tasks you are using in your build.
For example, for Android testing tasks(task for unit and cAT for UI automation testing), you can find the HTML testing result in [your project path]/app/build/reports.
I am working at a company that uses Redmine as the issue tracking tool and for user stories.
I need to design the QA for them. However I didn't find a section where someone can tie a user story or an issue with a test case. I've used to have this functionality on Rally.
Is there a free tool that integrates with Redmine or something like a plugin?
I want the following (or the more the better!)
Description
test steps (input-output)
Status (pass-fail)
Suite name (the suite the test is part of)
Attached issues
Version of SW under test
Thanks in advance
There is a "test case management" section in the Redmine Third Party tools documentation.
thanks for the support.A former colleague brought this to my attention:
https://bitbucket.org/bugzinga/redcase/wiki/Home
One thing I often see is people trying to adapt tools to do things that they were not originally designed for. Redmine is a good tool for issue tracking and is designed mainly to be a project management tool, although it is possible to adapt redmine, I would suggest in this case looking for a dedicated test management tool that has been designed to do that specific job.
I would suggest looking at TestLodge test management tool which I have worked on and is a designed to help you manage and execute your tests but at the same time integrates with Redmine and does things like automatically create tickets whenever a test fails.
By doing this, you are going to benefit from a range of things such as an interface designed for testing that will make you and your team a lot more productive along with a series of reports that will allow you to spot trends within your testing.
The company I work for is building a managed force.com application as an integration with the service we provide.
We are having issues working concurrently on the same set of files due to the shoddy tooling that is provided with the force.com Eclipse plugin. If 2 developers are working on the same file, one is given a message that he can't save -- once he merges he has to manually force the plugin to push his changes to the server along with clicking 2 'Are you really sure' messages.
Basically, the tooling does a shoddy job of merging in changes and forces minutes of work every time the developer wants to save if another person has modified the file he's working on.
We're currently working around this by basically 'locking' individual files by letting co-workers know who is editing a file.
It feels like there has got to be a better way in this day and age. Does anyone know of a different toolset we could use, process we could change, or anything we can do to make this easier?
When working with the Force.com platform my current organisation has found a number of different approaches can work depending on the situation. We all use the Eclipse Force.com plugin without issues and have found the following set ups to work well.
We have a centralised version control system that we deploy from using a series of ant commands to a developer org instance. We then depending on the scope of the work either separate it off into chunks with each developer having their own development org and merging the changes and testing them regularly, or working in a single development org together (which if you have 2 developers should be no major problem) allowing you to have almost instant integration.
If you are both trying to work on the same file you should be pair programming anyway, but if working on two components of a similar system together, sharing the same org can allow you to develop in a fast and flexible manner by creating the skeleton of the system you wish to use and then individually fleshing out the detail.
I have used both methods extensively and a I say, work really well depending on the situation.
Each developer could work in separate development sandbox (if you have enterprise edition, I think 10 sandboxes with full config & limited amount of data are included in the fee?). From time to time you would merge your changes (diff tool from any version control system should be enough) and test them in integration environment. The chain development->integration->system test->Q&A-> production can be useful for other reasons too.
Separate trick to consider can be used if for example 2 guys work on the same trigger. I've learned it on the "DEV 401" course for Developers.
Move all your logic to classes. Seriously. They will be simpler to unit test too.
Add custom field (multi-select picklist) to User object. Values should be equal to each separate feature people are working on. It can hold up to 500 values so you should be safe.
For User account of developer 1 set "feature1" in the picklist. Set "feature2" for the other guy.
In the trigger write an if that tests presence of each picklist value and enters or leaves the call to relevant class. This wastes 1 query but you are sure that only the code you want will be called.
Each developer keeps on working in his own class file.
For integration test of both features simply set the multiselect to contain both features.
I found this trick especially useful when other guy's code turned out to be non-optimal and ate too many resources. I've just disabled his feature on my user account and kept on working.
This trick can be to some extend applied to Visualforce pages too (if you can divide them into components).
If you don't want to waste query - use some logic like "user's first name contains X" ;)
We had/have the exact same problem, we have a team of 10 Devs working on a force.com application that has loads of apex classes (>300) and VF pages (>300).
We started using Eclipse plugin but found it:
too slow working outside of the USA each time a save is called takes > 5 sections
to many merge issues with a team of 10 developers
Next we tried developing in our own individual sandboxes and then merging code. This is ok for a small project but when you have lots of files and need changes to be pushed between sandboxes it becomes impossible to manage as the only thing worse then force.com development tooling, is force.coms deployment/build tools. No automation its all manual. No easy way to move data between sandboxes either.
Our third approach was to just edit all our VF pages and Apex code in the browser. (not using their embedded editor that shows up in the bottom half of the page because that is buggy and slow) but just using the regular Editor under setup > develop > Apex classes. This worked ok. To supplement this we also had a scheduled job that would download all our code and save it into our SVN repository. We also built a tool that allow us to click a folder on our desktop and zip its contents and deploy it as static resources for us.
However this approach still has its short comings, i.e. it is slow and painful to develop in the cloud, their (salesforce) idea of Development As a Service is crazy. Also we have no real SCM we only have it acting as backups.
Bottom line is force.com is a CRM and not a Development platform, if you can? run, flee, get away from it as fast as you can. Using it for anything apart from a CRM is more trouble then it is worth. Even their Slogan "No Software" makes me laugh everytime
I'm not familiar with force.com, but couldn't you use source control and pull all the files down from force.com into your repository. Then you could all do your work, and merge your changes back into the mainline. Then whenever it's necessary push the mainline up to force.com?
Take a look at the "Development Lifecycle Guide: Enterprise Development on the Force.com Platform". You can find it on developer.force.com's documentation page.
You might want to consider working on separate static resources and pages and then just being careful when editing objects, classes etc.. If most of your development happens on client side code (page, staticresource, lightning component/app) you might be interested in this project: https://github.com/bvellacott/salesforce-build . In any case I strongly suggest using version control. If not on a server then at least locally on your machine, in case your peers overwrite your work.