We are using Gherkin/Behave (in Python) to test an embedded application. The Gherkin code is executed on a server while the actual activity is performed by an application on the device, communicating over the network. The application on the device needs to be started manually.
I need a test to reboot the device. I can get the test application to perform a reboot, but then I need the code on the server to prompt the user to restart the test app so that the test can continue with subsequent steps. However I cannot get the Python code in the "steps" file to output any text.
I appreciate that Gherkin/Behave is meant to provide fully automated testing, but real world restrictions apply here.
for formatter in context._runner.formatters:
formatter.stream.write("Your message here\n")
formatter.stream.write("\n")
The extra newline is needed because Behave prints a description of the step first, then overwrites it in green if it passed. The extra newline ensures that this overwrite overwrites a blank line rather than your text.
Note that when I tested this I was using the default "pretty" formatter. I don't know how well it will work with other formatters.
Related
Is there a way to send a mail with result file (I set this file in console command with option --result) after running.
I have run my selenium test cases in following way
How to Schedule Selenium Web Drivers Tests in C#
The result file was created after OneTimeTearDown function.
If sending an e-mail into OneTimeTearDown function - the result file comes incomplete
Thanks in advance
Sangeetha P.
I'm not sure I'd actually recommend doing this - but I think it's possible. Personally, I'd instead handle the email sending outside of the NUnit console, in a separate script in your CI System.
Anyway. You could achieve this by writing your own ResultWriter extension. Take a look at the implementation of the standard NUnit3XmlResultWriter as an idea - you'd essentially want the same thing, except to send the file by email, rather than write a file. (You may even want to make your ResultWriter actually inherit the NUnit3XmlResultWriter class.)
I've hit a bit of a roadblock, and I'm hoping someone can help!
I've written a metro application that serves as a unit test runner, and I now need to be able to call this application headlessly so that it can be used for validation in the build process. The way the metro app works is it runs a bunch of unit tests, generates an XML file that contains the test results, and displays the results to the user.
Ideally, I would have a simple script that would run the metro app, execute the tests, exit the app, and then have the ability to read the results in the generated XML file. Is this possible, and if so, what's the best way to do it?
Here are some more specific questions:
How can one start a metro app headlessly, and in the metro app is there a way to detect this so that it does not wait for user input?
Is it possible to access files within the package of a metro app from an outside process?
EDIT - A workaround would be to create a custom Visual Studio test runner and then find a way to run the tests automatically with each build. I know this can be done within the IDE, but I'm not sure if there's a way to do this with a script.
I imagine you've long since moved past this problem, but for the sake of anyone else looking to do this, I got it to work without too much hassle. To execute a Metro app in an automated/headless fashion, I wrote a simple desktop command-line utility that takes the name of a metro app and makes use of the IApplicationActivationManager interface to launch it. I can then call that utility from a script.
The second argument to that inteface's ActivateApplication method is a string that gets passed in to the activated app, kind of like command-line arguments. It shows up as the Arguments property of the LaunchActivatedEventArgs that is received by the app's OnLaunched handler. The default implementation of OnLaunched in the Visual Studio template projects passes this value to the MainPage when it first navigates to it, where it comes through into the OnNavigatedTo handler as the Parameter property of the NavigationEventArgs. You could catch it in whichever place is more convenient.
My launcher utility passes a hard-coded flag through there, as well as forwarding its own command-line arguments. That allows the top-level script to pass arbitrary data down into the Metro app. The app can use that data to realize that it's running headless and run its tests. It can spit out whatever kind of result data you like into one of its folders (like its LocalFolder), which a desktop app can then read from %LOCALAPPDATA%\Packages\APPNAME\LocalState. I setup my launcher utility to wait for the result files to appear after launching the app, and then use them to determine its own exit code. The launcher utility can't kill the app afterward, but the app can kill itself when it's done via CoreApplication.Exit.
That setup worked great for a while, but a problem that I'm running into now is that the app isn't always launched to the foreground, and the runtime will suspend/terminate the app after it hasn't been the foreground app for some amount of time (currently ~10-15 seconds). So any tests that take too long won't work with this approach, barring some workaround that I haven't discovered yet (which I was searching for when I came across this question).
I doubt you'll be able to do it.
It's the same sort of problem as trying to run a WPF app headlessly, but harder since you'd also have to deal with the Metro sandbox security model.
P.S. Happy to be proven wrong!
No, sorry. You hit a wall with your first requirement of a script that runs the Metro application in "headless" mode in the first place. Your second requirement would be your next wall. One application cannot see, let alone monitor, another application/thread/process. Then your third requirement is also impossible. Files inside an application are isolated. It sounds to me like you found a good candidate for a desktop app. Having said that, don't mistakenly think that you can't have a companion Metro application that is your dashboard. It's just the execution core can't be hosted inside the WinRT sandbox.
I have Jenkins project that perform some sort of sanity check on couple of independent documents. Check result is written in JUnit XML format.
When one document test fails, entire build fails. Jenkins can be simply configured to send email to commiter in this situation. But I want to notify commiters only when new test failed or any failed test was fixed with the commit. They are not interested in failed tests for documents they have not edited. Email should contain only information of changes in tests, not full test report. Is it possible to send this kind of notification with any currently available Jenkins plugins? What could be the simplest way to achieve this?
I had the same question today. I wanted to configure Jenkins sending notifications only when new tests fail.
What I did was to install email-ext plugin.
You can find there a special trigger that is called Regression (An email will be sent any time there is a regression. A build is considered to regress whenever it hasmore failures than the previous build.)
Regarding fixed tests, there is Improvement trigger (An email will be sent any time there is an improvement. A build is considered to have improved wheneverit has fewer failures than the previous build.)
I guess that this is what you are looking for.
Hope it helps
There's the email-ext plugin. I don't think it does exactly what you want (e.g. sending only emails to committers who have changed a file that is responsible for a failure). You might be able to work around that/extend the plugin though.
Also have a look at the new Emailer, which talks about new email functionality in core hudson that is based on aforementioned plugin.
I am retrofitting unit testing into a fairly complex system designed and written by other developers in VB.net. I am trying to develop unit tests for the GUI forms using NUnit and the NUnit Forms extension. (I've been looking at c# examples that are fairly easy to port over if you have a solution but don't know VB syntax as long as it uses NUnit classes)
I will try and explain what I am doing but first a brief description of the program. It basically monitors server activity. You need to connect to a server via a modal form with IP and Port fields(amongst others). Once you have connected to a server other parts of the program unlock and become usable (such as configuration of the server).
Desired process: Load program > click connect button > modal connect form loads > enter details > click OK to connect > main form updates to logged-in state > other functionality
The problem is that I cannot test the functionality of the connect form and then the logged-in functionality of the program. I can test that it loads the modal connect form correctly; enters the details and clicks OK (all fine so-far) but it does not appear to logically progress the program. The modal form just closes again seemingly without running the connect code from the program back-end and I’m back at the main menu not logged in to anything.
I have a feeling that I’ve either missed something really obvious or that it’s simply not doable in NUnit. I have trawled the internet in search of anything similar but the closest was another SO thread that was really generic. Without being able to actually test the logged-in version of the program, I'm at a major hurdle.
Another issue is handling message boxes that don’t have unique identifiers (e.g. “are you sure you want to exit?”); these also seem to be a major pain in the arse with NUnit
(If it makes any difference, I’m running the tests as a stand-alone project using a reference to the executable file of the built project, not the actual source)
Can post some of my testing code if required.
IMHO the best approach to make GUI classes feasible for unit tests is to apply the Model-View-Presenter pattern and factor almost every program logic out of the form (=View) class to a separate Presenter class. Then you can unit test the Presenter class without the need for tools like "NUnit Forms".
Read Michael Feathers' article "The Humble Dialog Box" for an example in C++, you can easily apply that to Winforms, I guess.
I'm not sure about NUnit forms, but using the White library (which also works with NUnit), you're able to test the application by running the exe and mimicking user actions. The application runs normally so all application logic is performed.
Here's some example code for launching an app with White:
Dim app = White.Core.Application.Launch("MyApp.exe")
Accessing a form from your app:
Dim mainForm = app.GetWindow(SearchCriteria.ByAutomationId("MainForm"),
InitializeOption.NoCache)
Performing an action such as clicking a menu item:
mainForm.MenuBar.MenuItem("Edit", "Jobs...").Click()
Getting a control and validating its state:
Dim someTextBox = mainForm.Get(Of TextBox)(SearchCriteria.ByAutomationId("txtValue"))
Assert.IsTrue(someTextBox.Text = "12345")
I'm not sure if NUnit Forms has similar capabilities, but if not, maybe you should look into White. I ran into some issues setting it up so make sure to read the documentation carefully (not very exhaustive unfortunately) before setting it up.
I'm writing some integration test with plone.app.testing.
Sometimes I want to print something to the console, but it seems to me that stuff gets printed on the console only when a particular test has finished.
Does plone.app.testing or some of the packages behind have some logging facility which I can use?
To support testing logging output, IOW tests that check that your code is logging things as it should, I think zope.testing intercepts all logging. Additionally, depending on how you're testing, stdout may be replaced or intercepted, as with doctests for example, so printing to stdout may not work either.
Personally, I used pdb.set_trace() or I force a failure temporarily, IOW without committing it, at the point I want to inspect and run the tests with "-D" for pdb.post_mortem() debugging.
You may, however, be able to make some use of zope.testing.loggingsupport to collect the information you want and then you can use a pdb.set_trace() or "-D" to get a pdb prompt at which you can inspect whatever handler you created to capture logging output.
Take a look on this also: Redirecting log output to sys.stdout on tests.