I am new to JBehave and Maven and am struggling to properly run my JBehave tests.
From the command line, which is the proper command:
mvn integration-test
or
mvn jbehave:run-stories-as-embeddables
When I run the Etsy tutorial example with "mvn integration-test", it shows 'BUILD SUCCESS' even though there are 2 stories (Etsy Cart and Etsy Browse) that throw Exceptions. The JBehave HTML report shows the errors, but they are not shown in the console (ie., the BUILD SUCCESS I already mentioned). I am also going to integrate with Jenkins, so I need it to also report it on the console (or not?)
Anyhow, How can I properly run JBehave tests with Maven on the CLI so that it fails (but does not stop) when any of the tests fails and reports so on the CLI?
PS. I know there is one other similar question here on SO, but it just points to the tutorial, resulting in what I see above. Thanks!
Related
I'm trying to improve the testing process where I work, but without adjusting the structure.
What we have: VSTS, Selenium IDE, Testers who write test cases, but not code.
What I'd like to do is manage a way to marry our TFS continuous integration with the Selenium tests we write. These are NOT the code-driven selenium tests, but rather the IDE version where users click through, and set assertions using the IDE (All are just UI tests). I know we can export those tests plans as a .SIDE file, but what I can't figure out, is how to have our TFS server execute those as part of a deployment or build pipeline.
Ideally, developers/devops would setup projects in TFS from the onset with whatever solution makes sense to execute these Selenium .SIDE files, but afterwards, the testers would manage adding/modifying those tests cases elsewhere.
The real goal here is to not have testers writing code, or checking in code. Only writing these UI Selenium tests, but having TFS execute those as part of CI.
Researching this on the internet drives me basically always to something that requires testers to write code.
I don't think it can automate testing without code, at lease, you need a test project containing your automated tests.
Generally, in Azure DevOps, we use Visual Studio Test task to run tests. This task supports using the following tests:
Test assembly: Use this option to specify one or more test assemblies that contain your tests. You can optionally specify a
filter criteria to select only specific tests.
Test plan: Use this option to run tests from your test plan that have an automated test method associated with it. To learn more about
how to associate tests with a test case work item, see Associate
automated tests with test cases.
Test run: Use this option when you are setting up an environment to run tests from test plans. This option should not be used when
running tests in a continuous integration/continuous deployment
(CI/CD) pipeline.
This was a question that I had as well, and I think I found an imperfect but better solution.
I wasn't able to get my Selenium IDE tests running with Jenkins, but I was able to get them to run with TeamCity, another CI.
I created a build step like the following :
Runner type: Command Line
Working Directory: where the selenium IDE .side file is located
Run: Custom Script
With the build script content that I usually use to run my Selenium IDE Tests, such as selenium-side-runner sidefile.side
I also added the following so I could output the results in Junitor another form: --output-directory=results --output-format=junit
You can also add the following so the tests are run headlessly, this only works in Chrome : -c "goog:chromeOptions.args=[--headless,--nogpu] browserName=chrome"
Finally, I also use --filter to run one test suite at a time, but that is optional too.
I then used another build step to export the results to our test manger, xray, but I think that is beyond the scope of this question.
The problem with this solution is that it runs directly from a users individual machine still, but this can be work around.
I have a suite of Cucumber tests being run by Jenkins periodically. Most runs are not generating a JSON report. More specifically, a zero sized JSON file is created. I'm using version 4.3.1 of cucumber-java, cucumber-java8 and cucumber-junit and Java 1.8.
My test setup is a little convoluted. Jenkins runs a job every 2 hours to run the tests. This job runs in its own Docker container (running a Linux image) wherein a fresh clone of the test repo is created. Jenkins then executes Gradle to build and run the tests.
In the Jenkins console output I can see Gradle starts the tests and presumably executes some but never completes them all. But there is no error or exception from Gradle, it simply stops running. Nor is there any message about the JVM exiting with a non-zero status.
There is the occasional run of the tests that will produce a non-empty JSON report. This tends to coincide with all of the tests passing, but not always.
Unfortunately I am not able to post the Jenkinsfile, build.gradle or anything else. If you need further details I might be able to provide small snippets.
I would like to schedule task which is login to a partner site and prolongs a license. I found that it can do Selenium. After I faced issue with scheduler and found article here http://www.learnseleniumtesting.com/ about Jenkins configuration. Now I cannot understand how to run test from selenium in Jenkins. Is there need good knowledge in python or java ? Also I found that RC standalone selenium server is deprecated and avoided to be using now. Is there need deep diving into using these tools for executing 5-7 action on site ?
I feel that Selenium is great for doing any tasks that you may find yourself doing over and over again. For "prolonging a license" on a partner site I think this is the perfect tool for you. We setup Jenkins to run certain Selenium tests after each commit and other Selenium tests every night. If you need me to go into more detail about how the setup looks I would be glad to show you more details.
Adding Jenkins build information to help show how we did it.
1) The first step is to get a build in Jenkins that will compile any code you have and deploy a war if needed with something similar to the following: mvn -P release clean test package. We set ours to build #midnight shown in the picture below.
2) After this build finishes it triggers the next job to run this command shown in the image below: mvn -Dtest -P smokeTest clean verify
I have setup Bamboo to run JBehave tests on a remote agent (with JBehave-web plugin launching test using webdriver), and everything runs fine. Only problem is after the execution is finished Bamboo shows no test executed. I can see the option in Bamboo to select the output of the test results, but it has to be a JUnit xml, and Jbehave reports are only generated in plain text or html.
Any idea how to solve this?
Thanks
I ran in the same situation about a year ago. JBehave "doesn't" integrate with Bamboo out of the box. Although, they have a plugin for Hudson CI.
In my case, such as yours, I resorted in running the tests through the Surefire plugin; the outputs are considered as JUnit tests results and Bamboo can recognize them.
Hope it helps.
There is a really simple way to do this. And I'm currently doing this for our build system.
Write a simple parse script that transforms your plain of html report into JUnit compatible results. And add that script as a task in your Bamboo task, then use Junit parser to parse the results. Boo! You're done! Plus, you've got the capability to quarantine!
This is way much faster than writing a plugin for Bamboo, which involves considerable more time to learn/write.
Setup JBehave with Maven.In Bamboo build plan use Maven task to run it. For getting results in Bamboo use JBehave Task for Bamboo. It will convert JBehave scenarios in tests in Bamboo. If scenario names contains JIRA issue ids, it will link them to JIRA issues.
https://marketplace.atlassian.com/plugins/com.mdb.plugins.jebehaveforbamboo/server/overview
Sample JBehave as Maven Project
https://bitbucket.org/vikasborse/jbehavesampleproject/overview
Download or clone this repository on your local machine.
To run navigate to this project in command line and use command:
"mvn integration-test"
I've tried in each and every way to test a grails-app using hudson. I've tried testing with maven, I've tried testing with the grails plugin and I've tried testing with a shell builder it seems that building via shell is the only thing that works..
Every time I get the same error:
org.hibernate.HibernateException:
contains is not valid without active
transaction
But If i go to a shell and type
grails test-app
everything runs fine.
Does anyone have any idea on what's going on?
I'm using CentOS with Java 1.6, no slaves, just a simple hudson deploy over Tomcat6.
I've tried both with maven and grails builder, both fail.
Edit: it seems that if I run both unit and integration tests on the same command (either with grails or with mvn) the integration tests fail always.
Hudson/Jenkins usually just use the command line for executing grails plugins (You should be able to confirm that from the build output). You could probably add a pre build step to dump the environment, so you can see if anything there (or in your own shell) cause it to be fundamentally different.
Otherwise try to log in as the hudson user and find the hudson workspace and repeat the process manually. That has been the easiest way to debug hard problems like this..
regards