I'm writing a very simple Maven plugin, and I am kind of lost with the testing part. Based on the plugin developers site, there are several different tools to achieve the same things, which documentation seem outdated... I found resources like this too, and took a look at the "core" plugins, but this is still very unclear to me.
As there does not seem to be one solution for this, I am asking for your personal experience here.
I don't have much to add to the Review of Plugin Testing Strategies page which is pretty exhaustive and it really depends on the kind of tests you want to write.
But since you're asking for practical feedback my recommendation would be to write both unit tests and integration tests, the later being extremely important when plugins (if you want to skip something, don't skip integration tests).
For Unit Tests, use the maven-plugin-testing-harness. check the Maven Plugin Harness Wiki and existing plugins like the maven-compiler-plugin (see the CompilerMojoTestCase).
For integration tests, I suggest to use the maven-invoker-plugin and/or the shitty-maven-plugin. They're similar but still have slight differences so you might prefer one over the other depending on the situation. Refer to the matrix for the details.
See also
unit-testing maven plugins
Take the Perl's advice - there's always more than one way to do it.
It really depends on the scope of your test. If you are testing against simply just the logic, a basic JUnit test case will do; If you are performing mock tests or integration tests with some contexts needed and you simply do not want to redo what people have done, the Maven's page has explained how these different techniques shall be applied.
Related
I have a project with login and other functionalities tests in Cucumber. There are different projects which use the same login function. I would like to reuse the cucumber login steps from one project to another project.
Eg:
Project1->TestLogin1
Project2->TestLogin1
In general don't try and do this. Cucumber scenarios should describe the behaviour of your system and their implementation should be specific to each particular system. People have been trying to do this in the cukes community for years, generally with little success.
Sure with something as simple as login you could share ... until one application starts allowing you to register by Facebook whilst the other requires you to confirm via email.
In practice the amount you save in sharing (which is very small) if offset by the amount you lose in being able to make you scenarios specific to the world of your application.
You could definitely benefit from sharing step definitions between projects, because there is likely to be a lot of overlap between certain parts of an app, such as admin tasks.
If you use an IDE for feature editing, you may then be able to benefit from leveraging those step defs through autocomplete.
It should be possible to package step defs into repos that are then included by module. You might be able to leverage tags or hooks to aid in setup so the context is correct.
Whether it’s worth the effort of coordinating across many projects will likely depend on your use case.
Along with a basic doubt I had here(which unfortunately did not get much attention) I have another basic doubt in context with preparing a Java Framework for Selenium-WebDriver.
I am not satisfied with the answers here and need some quality advice on best practices from real world case scenarios. (Counting on SO)
The main question is - Which one is better suited for Selenium? TestNG or JUnit? And Why?
I got some basic differences here,here(for Automation but unclear answers -all ask to pick one) and many more. Looking for some more details which will help me decide better.
P.S: I have experience with Selenium+JUnit. Will I regret if I go the same way?
simplicity with JUnit vs. adjustability with TestNG - that's all
Firstly, both work quite well.
If you want to scale down your execution time and have enough time to find out the best possible configuration I would recommend TestNG. In case of parallelization that's your tool
If you prefer the plainness of a top down execution JUnit is what you're looking for
--EDIT--
Regarding JUnit 3
I'm in the process of switching from JUnit to TestNG with a project right now. It started as an experiment until switching over completely became a 'done deal' in my mind. Some things that sold me on it are the ability to easily tag tests (groups), easily create dependencies (don't run test y unless test x succeeds first), more robust out of the box reporting (robust may not be the right word, but I find them much easier to read and manage) and just more whizz-bang options in general, easily available and extendable. I'm early in the process and may yet come to regret it and switch back, but unless there's some major unforeseen disaster lurking in the road ahead I don't think so. If you do make the switch, look up the Eclipse plugin. Apply it to an existing test and be sure and take out any references to JUnit. You'll have to do some polishing and tweaking here and there, but it's no big deal once you get going.
Also bear in mind that the stock asserts have the same names as JUnit assertions, but the order of expected and actual values is opposite between the two. Dig in a bit and you'll see what I mean. Again, no big deal, just need to be aware of it.
Best wishes with whichever you choose. You can make either do what you want, I'm just finding TestNG a bit more flexible and...pleasantly surprising, which is nice to experience now and then in Java land! ;)
First i have no experience on parasoft .test or jtest experience. I have read the datasheet that the product could automatically generate unit test.
but I am woundering how useful the auto generated unit test are. Does it really do not need any other effort by developer?
any experience sharing are welcome.
thanks a lot!
We used JTest for our product recently. We didn't use the standard product, we used the Eclipse Plugin. The standard product is built on the OSGI framework (read: it's like Eclipse), but you have to import and create your projects. We were already using Eclipse, so it made sense for us to simply use the plugin, which has all of the same capabilities.
While there are many things that JTest can do for you, there are also many irritating things about it. For example, Jtest's static analysis tool is what is really worthwhile, IMHO. It can look for lots of errors and has a pretty good reporting system. But, while unit test generation is okay, but I think I spent as much or more time fixing and enhancing the generated tests than I would have just making them myself. Administering Jtest is also somewhat complicated and involved.
The built-in mechanisms to make unit tests, stub objects, parameterized unit tests, etc. are not well documented. At least, my little brain couldn't make good use of them in the two years we used the product. However, a lot of their super awesome features (like GUI tracing, command-line interface, the Bug Detective, reporting system etc.) all require extra, very expensive licenses.
Really, Jtest just gives you an easy way to manage the execution of static and unit testing. But it's really expensive. I can't believe they charge thousands of dollars per license of that stuff. You'll also find that they will want to train you, which you almost need because the documentation is pretty bad. Which is odd, because the user's guide is like 900 pages long.
But here's a big hint: you can do it for free. If I had to do it over, I would have pushed hard for using these products (which, oddly enough, look and feel very similar to Jtest)
http://code.google.com/javadevtools/codepro/doc/index.html
I wouldn't get Jtest thinking that this will be a small something to add to your developer's routine. Jtest can become a huge time and process sink.
Jtest is very very useful.Yes it generates it own test cases which requires lot more efforts for fixing them.I use it in different form.I delete all the generated unnecessary test cases.I made one another file which create database connection and set various other parameters sets.Also after configuration the code will work without mocking if all of the code is ready and if it is not ready than you can stubs the required methods.
Static code analyzer is good(for checking null pointer exception)
Checking code conventions is very good.
Write your custom code guidlines as use cases and execute it on your code.
Code coverage.
Debug while testing.
The auto generated unit tests still needs a developer to decide what results are correct or not, so you have to sit down and do the job. A lot of the boiler plate code is of course auto generated, so a small time saver there. I haven't used it much, but did evaluate jtest for an earlier employer. Seemed like a great product, if I remember correctly. :)
Alas there will never be a silver bullet that addresses all unit testing requirements, but JTest & .Test (& C++Test for that matter) about as close as you will get. Uggwar is correct that the developer will still need to verify outcomes for the basic auto generated tests, however there is a whole lot more to it.
These tools can be used to create basic regression tests, these are there to tell you when something has changes, not whether what it is testing is right or wrong. You can also trace a running application and then generate JUnit/NUnit/CPPUnit tests that recreate what was going on in the application. These tend to be far more useful tests, which are used as regression tests for items of functionality.
Other functionality includes the ability to generate stubs, use spreadsheets as datasources and provide an object repository. There is a while lot more too ....
Give them a try.
http://www.parasoft.com
I've got analysis paralysis looking at all the different functional testing options for a new grails (v1.3.4) application. I've looked at
WebDriver/Selenium (which I've used before)
WebTest/Canoo
Geb
Tellurium
Grails Functional Test
and there must be others. I think some of the criteria that I would use to make a decision include (in no particular order):
Likely longevity, active development
Can do ajax/javascript
Support for PageObject or similar patterns
Maturity
Headless (eg htmlunit) is ok, especially if it makes things go faster
Good reporting
Support for NTLM credential provider or similar
Compact, robust test scripts
Takes advantage of groovy language
I would be particularly interested to hear from people who have tried more than one framework. Thanks!
I maintain the plugin for WebTest but I'd recommend giving Geb a try. I haven't used it personally on a project yet but I think it will tick off most of your criteria.
It is the most actively developed (IMO) but is quite new. It is built on WebDriver/Selenium so should also be a good fit with your past experience.
WebTest doesn't see a lot of development these days and does not have built in support for page objects. But it does give you great reports. It's downside is it's historical foundation of Ant. It makes it hard to test in a dynamic fashion as your test steps are built up when the test case is executed and then the actual test is run as a second pass.
I have used GFunc as well but the lack of reporting is a real pain and it does not get as much development as Geb. It is far more "groovy" than WebTest though and is a thin wrapper over HtmlUnit allowing you to "roll your own" testing functionality quite easily.
cheers
Lee
Grails Functional Test is HtmlUnit only. I wouldn't suggest coding directly to HtmlUnit these days with the other frameworks that are available that give you page object and better abstraction out of the box.
I've used the webdriver plugin which seems to work pretty well and has page object built in - but the page object support is coupled to the JUnit hierarchy, which makes it hard for me to use because I want to also use Cucumber.
I'm very excited about Geb and am hearing a lot of success stories with it but haven't gotten to finish setting it up myself.
In my limited experience with them executable requirements (i.e. specifying all requirements as broken automated tests) have proven to be amazingly successful. I've worked on one project in which we placed a heavy emphasis on creating high-level automated tests which exercised all the functionality of a given use case/user story. It was really amazing to me how much easier development became after we began this practice. Implementing features became so much easier after writing a test and we were able to make major architectural changes to the system with all the confidence in the world that everything still worked the same as it did yesterday.
The biggest problem we ran into was that the tools for managing these types of tests aren't very good. We used Fitnesse quite a bit and as a result I now hate the Fit framework.
I'd like to know 1) if anyone else has experience developing using this type of test-driven requirement definition and 2) what tools you all used to facilitate this.
The primary tool I've also used was FitNesse. I've used it at several companies, with very good results. We did have test cases numbering in the many thousands, and we had to be very disciplined in how we organized and used them.
I've tried some other tools, including writing my own DSL (domain-specific language) and using things like RSpec. I really like RSpec, but it is certainly more of a developer tool than a business one.
I know Rick Mugridge has been working on a tool called ZiBreve (http://www.zibreve.com/visit.php?page=index) which is supposed to have stronger refactoring support. I haven't used it myself, but I know Rick and have talked to him several times. I know there was discussion at Agile 2008 on some different ways to deal with the Fitnesse tests in general.
Other than that, I haven't seen a lot of good tools out there. Even tools like WinRunner are fine for QA type tests, but for exploratory testing of requirements by the business, FitNesse or a custom DSL seem to be the ways to go right now.
You might want to take look at Robot Framework (http://robotframework.org). It's FIT-like but hopefully easier to integrate to different testing tools, version control and continuous integration. Different abstraction levels in the test data also make it easier to maintain the data, and when the separate test data editor gets more mature maintenance gets even easier. The quick start guide introduces the most important features of the framework and acts also as an executable demo.
I've had to use, test and set up both fitnesse and one of it's competitor, GreenPepper for my work, and what I can say is :
GreenPepper is a confluence plugin (confluence is an enterprise wiki from atlassian) and have many of the things you need in an "enterprise" level tool with little to no additional work required :
Better user friendly -rich text- wiki
syntax (makes it easier to work with
for non technical people)
It integrates very well with many
development tools : Eclipse, VB,
maven2 and Nant plugin, I tested most
and was very pleased.
User and access rights are managed by
confluence, which is to say it's good
and make uses of database of your
likin (which might be mandatory
depending on where you work)
Many other functionalities that might
or might not be required : ssl support, remote execution (install the wiki on unix, execute on windows if you are working on a C# project, or reverse)
Looks way better :D
Big downs for GreenPepper are : Configuration is quite hard and documentation is poor (although they seem to be working on it and they answer quite fast on their forum) and also it is not free, you have to pay for both confluence and GreenPepper, which might add up to quite a lot.
Fitnesse is very basic in my opinion, very easy to set up, it works but that's it, you can use some of the fitnesse plugins developed by the open source community, and even some Fit plugins, such as the Eclipse plugin (build the skeletton of the fixture from a fitnesse test file, provided it's in a .fit extension, very usefull). Integration is not ideal, authentification and access rights management is poor, but it's FREE and if you need something, you can do it because it's open source.
I've found that using contracts is a great approach. Metaprogramming contracts are generally lower-level than the types of integration tests you describe, but the two are certainly not mutually exclusive. I find contracts help keep documentation, implementation, and testing all in sync -- this is a major problem of TDD (not that it isn't a problem in non-TDD).
I've tried Fitnesse and its really awful (particularly integration with SVN).
And our company develop similar open-source tool with fit engine: FitPro
Another brilliant tool I've used is Concordion. It has the only disadvantage - requrements in html format
My experience is limited to personal projects and found much the same advantages you mentioned. I recommend http://metacpan.org/pod/Test::Simple::Tutorial which was my inspiration for trying out testing-based development. The perl testing modules seem pretty useful and flexible, though I have nothing to compare them to.
I also believe tests are vital for the maintenance period of a project. If you have good tests to begin with, it saves a lot of time and mistakes later on. I wish I had put more work into tests on my current project.