I have been developing a project which contains a TestLauncher class that'll read a given directory and for each file it contains, run it against my tool and yield the results.
So, when coding in Eclipse, it would show up one result for each test (as expected). Today I've been toying with Intellij, and I've decided to try to run and code a bit of this project in Intellij.
When trying to run the tests, though, it seems to be only showing up 2 results instead of the 100+ it should. Although I am sure it is running the full suite, it seems to be folding all the results of a given category in a single result. That means that if I have at least one failing test in each category, it shows up as a "failed test".
I guess this must not be a bug, but rather some configuration that I am not aware about and that is on by default in Intellij but not in Eclipse. Could anyone explain what might be going on?
Edit: I am using the latest Intellij (downloaded one of these days).
Thanks
What you're seeing is simply a difference in the way the Eclipse and IDEA plug-ins are implemented. I implemented the Eclipse plug-in to be pretty clever in its display, so it will show different things depending on various factors such as the presence of a toString() method in your test class or whether your test class implements org.testng.ITest.
I suggest you ask this question on the IDEA forums and if you don't get any response, feel free to email the testng-users list and I can put you in touch with the JetBrains engineer in charge of the TestNG plug-in.
The IntelliJ-IDEA TestNG Plugin has a filter symbol called "Hide Passed" above the output Test Results. You can toggle that to display all tests, including the passed ones.
Related
I found a really nice set of functionality in IntelliJ but it is very manual...
use Analyze/Analyze backward dependencies on a object to find all the tests that eventually reference that class.
Create a run configuration using Test Kind "Pattern" and manually enter each of the test classes found into the "Pattern" field.
Run the test with code coverage
Navigate to the original class to view it's total test coverage.
This whole process is fairly slow and user intensive, but it could easily be automated with a single "Find test coverage for class" key-press (It would still be pretty slow, but I could go on and do something else). Does anyone know if this is in a key binding or plug-in I haven't found yet? It seems like a pretty obviously useful and easy to implement piece of functionality.
If not, can anyone suggest how I might do this with the IDE scripting console or a custom Intention (I've had no success finding really good usable documentation/examples of the IDE scripting console, haven't looked into intentions too much...)
How about the following 2 flows/options based on the Windows shortcuts (don't mind the failing stuff, it's just a quick copy-paste for the sake of brevity):
1) With the cursor placed on your class name:
CTRL+SHIFT+T (Chose test for launch)
SHIFT+END (Select all)
SHIFT+UP (Unselect Create new test...)
CTRL+SHFIT+F10 (Execute selected tests)
2) With the Group by test/production option selected in the find window and cursor placed on your class name:
ALT+F7 (Find usages)
chose the tests from the list
CTRL+SHFIT+F10 (Execute selected tests)
Is there a way to run a single test in isolation, or a group of tests?
I find that when I am trying to debug a particular test, it's really annoying because previous test cases may trigger that same break point-- So I typically comment out all my tests except the one I am working on, but that seems so ghetto... So I am wondering if there's a better way?
The only way I know of doing this is by editing the scheme. The "Test" action has a bunch of checkboxes where you can turn on or off groups of tests (i.e. classes) or single tests. If you're doing this with the same tests a lot, you could create a set of schemes for different groups and set it up just the way you want. (Or just edit it each time.)
Perhaps not the answer you're looking for, but in AppCode, the iOS and OSX IDE from Jetbrains you put the cursor anywhere over the test you want, and then trigger it with a keyboard shortcut. (I use IntelliJ mapping so its Ctrl+Shift+F10).
It also has a great test runner showing green/red bars for tests that pass or fail.
I've been trying to use the JFace ProjectionViewer to implement folding in a standalone Java app. I got the idea from this article:
http://www.eclipse.org/articles/Article-Folding-in-Eclipse-Text-Editors/folding.html
However the source code provided with the article is for an Eclipse plug-in, not for a standalone.
The particular problem I'm having is that I can't get the VerticalRuler to respond and cause folding/unfolding.
Since I get the expected results, i.e. line numbers in the ruler, when I change from using a VerticalRuler to a LineNumberRuler while leaving everything else the same, I think my problem is specific to the implementation of the relationship among the ProjectionViewer, the VerticalRuler, and the Annotations.
My exact question is whether anyone has gotten this to work in a stand-alone code and, if so, how?
For our webapp testing environment we're currently using watin with a bunch of unit tests, and we're looking to move to selenium and use more frameworks.
We're currently looking at Selenium2 + Gallio + Xunit.net,
However one of the things we're really looking to get around is compiled testcases. Ideally we want testcases that can be edited in VS with intellisense, but don't require re-compilling the assembly every single time we make a small change,
Are there any frameworks likely to help with this issue?
Are there any nice UI tools to help manage massive ammount of testcases?
Ideally we want the testcase writing process to be simple so that more testers can aid in writing them.
cheers
You can write them in a language like ruby (e.g., IronRuby) or python which doesnt have an explicit compile step of such a manner.
If you're using a compiled a compiled language, it needs to be compiled. Make the assemblies a reasonable size and a quick Shift F6 (I rewire it to shift Ins) will compile your current project. (Shift Ctrl-B will typically do lots of redundant stuff). Then get NUnit to auto-re-run the tests when it detects the assembly change (or go vote on http://xunit.codeplex.com/workitem/8832 and get it into the xunit GUI runner).
You may also find that CR, R# and/or TD.NET have stuff to offer you in speeding up your flow. e.g., I believe CR detects which tests have changed and does stuff around that (at the moment it doesnt support the more advanced xunit.net testing styles so I dont use it day to day)
You wont get around compiling test frameworks if you add new tests..
However there are a few possibilities.
First:
You could develop a native language like i did in xml or similar format. It would look something like this:
[code]
action name="OpenProfile"
parameter name="Username" value="TestUser"
[/code]
After you have this your could simply take an interpreter and serialize this xml into an object. Then with reflection you could call the appropriate function in the corresponding class. After you have a lot of actions implemented of course perfectly moduled and carefully designed structure ( like every page has its own object and a base object that every page inherits from ), you will be able to add xml based tests on your own without the need of rebuilding the framework it self.
You see you have actions like, login, go to profile, go to edit profile, change password, save, check email etcetc. Then you could have tests like: login change password, login edit profile username... and so on and so fort. And you only would be creating new xmls.
You could look for frameworks supporting similar behavior and there are a few out there. The best of them are cucumber and fitnesse. These all support high level test case writing and low level functionality building.
So basically once you have your framework ready all your have to do is writing tests.
Hope that helped.
Gergely.
Looking for a way to get a visual report about:
overall test success percentage over time (information about if and how quickly tests are going greener)
visualised single test results over time (to easily notice test gone red that has been green for long time or vice versa to pay attention to a test that has just gone green)
any other visual statistics that would benefit testers and the project as a whole
Basically a tool that would generate results from the whole test results directory not just off the single (daily) run.
Generally it seems it could be done using XSLT, but it doesn't seem to have much flexibility to work with multiple files at the same time.
Does such a tool exist already?
I feel fairly courageous to claim that most Continuous Integration Engines such as Hudson (for Java) provide such capability either natively or through plugins. In Hudson's case there's a few code coverage plugins available already and I think it does basic graphs from unit tests automatically by itself.
Oh and remember to configure the CI properly, for example our Hudson polls CVS every 10 minutes and if it sees any changes, it does all the associated tricks (get updated .java files, compile, run tests, verify dependencies etc.) to see if the build is still OK or not.
Hudson will do this and it will work with Nunit (here), Junit (natively), and MSTest.exe tests using the steps I outline here. It does all that you require and more. Even if you want it to ONLY run tests and give you feedback on those, it can.
There's such new report supporting NUnit \ JUnit called Allure. To retrieve information from NUnit you need to use NUnit adapter, for JUnit - read the following wiki page. You can use it with Jenkins via respective plugin.