I found a bug in an open source project on GitHub, and wrote a failing test for it, but haven't suggested a fix due to insufficient familiarity with the code.
How does one usually contribute such tests? Shall I create a pull request? Note that the continuous integration would fail for my commit as it adds a (currently) failing test..
(For reference here's the actual test)
You can try to use the "Issues" functionality of Git. Create an issue as a bug report, instead of creating a pull request.
Related
I am trying to formalize the development workflow and here's the first draft. Welcome suggestions on the process and any tweaks for optimization. I am pretty new when it comes to setting up the processes and would be great to have feedback on it. P.S: We are working on an AWS Serverless application.
Create an issue link in JIRA - is tested by. The link 'is tested by' has no relevance apart from correcting displaying the relation while viewing the story.
Create a new issue type in JIRA - Testcase. This issue type should have some custom fields to fully describe the test case.
For every user story, there will be a set of test cases that are linked to the user story using the Jira linking function. The test cases will be defined by the QA.
The integration/e2e test cases will be written in the same branch as the developer. E2E test cases will be written in a separate branch as it's a separate repository (Open for discussion).
The Test case issue type should also be associated with a workflow that moves from states New => Under Testing => Success/Failure
Additionally, we could consider adding capability in the CI system to automatically move the Test case to Success when the test case passes in the CI. (This should be possible by using JIRA API ). This is completely optional and we will most probably be doing it manually.
When all the Test cases related to a user story to success, The user story can then be moved to Done.
A few points to note:
We will also be using https://marketplace.atlassian.com/apps/1222843/aio-tests-test-management-for-jira for test management and linking.
The QA should be working on the feature branch from day 1 for adding the test cases. Working in the same branch will enable the QA and developer to be always in Sync. This should ensure that the developer is not blocked waiting for the test cases to be completed for the branch to be merged into development.
The feature branch will be reviewed when the pull request is created by the developer. This is to ensure that the review is not pending until the test cases have been developed/passed. This should help with quick feedback.
The focus here is on the "feature-oriented QA" process to ensure the develop branch is always release-ready and that only well-tested code is merged into the develop branch.
A couple of suggestions:
For your final status consider using Closed rather than Success/Failure. Success/Failures are outcomes rather than states. You may have other outcomes like cancelled or duplicate. You can use the Resolved field for the outcomes. You could also create a custom field for Success/Failure and decouple it from both the outcome and status. You ideally do not want your issue jumping back in forth in your workflow. If Failure is a status then you set yourself up for a lot of back and forth
You may also want to consider a status after New Test Creation for the writing of the test case and a status after that such as Ready for Testing. This would allow you to more specifically where the work is in the flow and also capture the amount of time that is spent writing tests, how long test cases wait, and how much time is spent actually executing tests and defect remediation
Consider adding a verification rule to your Story workflow that prevents a story from being closed until all the linked test cases are closed
AIO Tests for Jira, unlike other test management systems, does not clutter Jira, by creating tests as issues. So you need not create an issue type at all.
With it's zero setup time, you can simply start creating tests against stories. It has a workflow from Draft to Published (essentially equaling Ready for Testing).
The AIO Tests jira panel shows the cases associated with the stories and their last execution status to get a glimpse of the testing progress of the story as shown below. It allows everyone from the Product to the Developer to get a glimpse of the testing status.
You can also create testing tasks and get a glimpse of the entire execution cycle in the AIO Tests panel.
It also has a Jenkins plugin + REST APIs to make it part of your CI/CD process.
Just wondering if there is a way to integrate TFS with TestRail to replace(get rid of entirely) the Test Hub within TFS to use TestRail to record Test Plans?
My concern with removing the Test Hub, would be if Test Rail can still reference IDs in Bug and Stories within TFS and vice versa?
You currently cannot remove the standard Hubs in Visual Studio Team Services / TFS or replace them with something else. You can enable an extension that either adds its own functionality under said hub as a separate tab or adds another Test Rail hub to the top level (if there is any) or write your own. Extensions currently cannot leave their sandbox to overwrite standard functionality.
There is nothing preventing anything from keeping track of work item numbers anywhere, so the second part of your question, whether it would have broken any form of integrations, that's unlikely.
If you are on TFS, you could try creating a custom process template that doesn't have the Test Case, Shared Step, Test Suite and Test Plan work item types, this will likely at least completely cripple the existing functionality. In the on-prem version you can also customize the files on disk, I've never tried, but it's likely that you could probably hack the test hub away. That would be a totally unsupported scenario through.
Using TFS, is there any way to find out when a test first failed?
I am new to TFS and to the project, where we have a couple of unit tests failing since a while back, but we do not know what change that was committed when they first broke. When committing to TFS, the tests are run automatically. So is there any way TFS can save that info in a log, so we can take a close look at exactly what happened the the code when the tests started to fail?
Alternatively, suggestions for other tools that can do this are appreciated.
As you are aware, ultimately, it doesn't matter when a unit test first failed. What's important is (1) you fix the cause of the broken tests, (2) the team gets notified immediately when any test breaks. That said, I agree that knowing what changed can help find the cause.
I'm a bit rusty with TFS (and don't have one in front of me), but:
Test results are stored in tbl_TestResult in the collection database.
Test results' attachments are stored in tbl_Attachment also in the collection database.
Do you have the TFS cube setup?
Might be of use to you: Test steps and results in TFS 2010
In a flyway migration a FlyWayException can be throw according to differents cases : during a migration failure, if the given database url can not be found etc.
Everytime it's a FlyWayException with a JdbcSQLException as cause. But in my app I'd like to isolate these cases in order to provide differents behaviors.
Is there any way to do this ?
I can see that a JdbcSQLException contains a SQLState, maybe it can be a solution, but I don't know if it's the best one.
It is never good to parse out SQL exceptions as they may change from version to version of the database. I would submit a feature request to Flyway to have specific exceptions instead of one generic FlyWayException: https://github.com/flyway/flyway/issues.
We have a particular file, say X.zip that is only modified by 1 or 2 people. Hence we don't want the build to trigger on every check-in, as the other files are mostly untouched.
I need to check for a condition prior to building, whether the checked-in item is "X.zip" or not.. if yes, then trigger a build, else don't. We use only CI builds.
Any idea on how to trigger the build only when this particular file is checked-in? Any other approaches would be greatly appreciated as i am a newbie in TFS...
Tara.
I don't know of any OOTB feature which can do this, what you would need to do is write your own custom MSBuild task which is executed prior to the build running (pre-build action).
The task will then need to use the TFS API to check the current check in for the file you want and if it's not found you'll have to set the task to failed.
This isn't really ideal as it'll indicate to Team Build a build failure, which, depending on whether you're using check in policies, may be unhelpful. It'd also be harder to at-a-glance work out which builds failed because of the task and which failed because of a real problem.
You can change the build to occur less frequently rather than every check in, which will reduce load on your build server.
Otherwise you may want to dig into Cruise Control .NET, it may support better conditional builds.
If you could move X.zip into it's own folder, then you could set up a CI build with a workspace that only looked at the folder containing X.zip.
You would then need to add an explicit call to tf get to download the rest of the code as Team Build only downloads what the workspace is looking at.
But this might be simpler than the custom task approach?