Using Pytest to output only failures? - testing

I'm trying to use Pytest.
How do I set flags such that all tests run, logs only to be output on failure on a per test basis while the test is running, and silent otherwise?
I don't want it to hold my test failure logs hostage while waiting for the entire test suite to finish.

You should use pytest-sugar pytest package. It gives test failure information and python traceback as soon as any test fails.
Install using pip install pytest-sugar and run the tests normally.
More info here: https://github.com/Frozenball/pytest-sugar
Edit: If you don't want to see the passing test logs, you can pass -rf argument. More info here: https://stackoverflow.com/a/55367504/2312300

Related

How to run CI selenium side runner tests on Jenkins

I have a .side file generated by the Selenium IDE, which I need to run on CI using Jenkins.
I am running it as a build step with the following shell command:
selenium-side-runner /path/to/file.ide
The problem arises due to the fact that no matter if the selenium test fails, Jenkins always shows is as success.
In this thread it's suggested to upload the file as generic, but still, the commands to execute it are missing
How to upload a generic file into a Jenkins job?
I've found a possible solution to it on this posts, but I would appreciate having a cleaner way to solve this instead of parsing the results checking for errors.
How to mark a build unstable in Jenkins when running shell scripts
Is there a plugin able to run selenium .side files on Jenkins and this one showing the success/failures of the test?
You can generate a Junit test report file and then use the Jenkins Junit plugin after your tests execution.
selenium-side-runner --output-directory=results --output-format=junit
# Outputs results in `junit` frormat in `./results/projectName.xml'
Check the official documentation for more details.

Running python unit tests as part of bamboo build

I need to run python units test cases as part of bamboo build step and the build needs to fail if unit tests fail.
For this, I have a Script step in bamboo build and i am trying to run the following in it:
python -m unittest discover /test
Here, /test folder has all the unit tests.
The output of the above script it
Ran (0) tests
So the problem is that bamboo isn't able to discover these tests. Bamboo agent is linux.
Wondering if anyone has done such a thing before and has any suggestions.
The following worked. Used -p (pattern) attribute to discover/run the unit tests in bamboo (unix build agent)
python -m unittest discover -s test -p "T*.py"
Note: 1. all my test cases start with "T" e.g. Test_check.py
2. "test" is the package where all my test cases are.
If you haven't figured it out, likely because in windows filenames aren't case sensitive but in Linux they are...
And you're test file named Test_xxxx.py isn't the same as test_xxxx.py which is the pattern that discovery is trying to use...

Is there a way to run benchmarks with failing tests?

Is it expected that benchmarks don't run unless all tests in the package have passed.
I've looked at the testing package doc and the testing flags and I can't find it documented that benchmarks run only after all tests pass.
Is there a way to force benchmark functions to run even when some tests in the package have failed ?
You can skip the failing tests using the -run flag, or choose to run none at all
go test -bench . -run NONE

Explicitly specifying the main package to run tests for in golang with goconvey

How do I explicitly say with my go test command to run only tests for the main package and not others in my source directory.
At the moment it's working with $go test -v. But... I am using goconvey as well and it seems to be running recursively. According to this page https://github.com/smartystreets/goconvey/wiki/Profiles I have a file where I can pass arguments into the go test command. I know you can go test -v ./... for recursive or go test -c packagename/... but how do I just do it for the main?
Profiles is one to accomplish this, but you can also specify a 'depth' for the runner:
$ goconvey -depth=0
A value of 0 limits the runner to the working directory.
Run goconvey -help for details.

Jenkins succeed when unit test fails (Rails)

I'm barely started to use Jenkins and this is the first problem I've had so far. Basically my jenkins job always succeed even when an error happened in some of the tests. This is what I'm running in the shell config:
bundle install
rake db:migrate:reset
rake test:units
rake spec:models
Thing is that Jenkins only reports a failure when the task which fails is the last one. For instance, if I put "rake test:units" the last task it will notify an error if something go wrong. Using this configuration I only get error reports for the rspec tests but not for the unit tests.
Anyone wondering why I don't only use rspec or unit test, we are currently migrating to rspec but this problem is still painful.
This is part of the log from Jenkinsm as you can see one of the unit test fails but jenkins still finish with success.
314 tests, 1781 assertions, 1 failures, 0 errors, 0 skips
rake aborted!
Command failed with status (1): [/var/lib/jenkins/.rvm/rubies/ruby-1.9.3-p1...]
Tasks: TOP => test:units
(See full trace by running task with --trace)
Lot of rspec tests here....
Finished in 3.84 seconds
88 examples, 0 failures, 42 pending
Pushing HEAD to branch master of origin repository
Pushing HEAD to branch master at repo origin
Finished: SUCCESS
Jenkins executes the commands you type into a Build Step box by writing them to a temporary file and then running the script using /bin/sh -xe.
Usually this produces the desired effect: Commands are executed in sequence (and printed) and the script aborts immediately when a command fails i.e. exits with non-zero exit code.
If this is not happening to you, the only reason can be that you have overridden this behavior. You can override it by starting the first line of your Build Step with these two characters: #!.
For example, if your Build Step looks like this:
#!/bin/bash
bundle install
rake db:migrate:reset
rake test:units
rake spec:models
Then it means Jenkins will write the script to a temporary file and it will be executed with /bin/bash. When invoked like that, bash will execute commands one-by-one and not care if they succeed. The exit code of the bash process will be the exit code of the last command in the script and that will be seen by Jenkins when the script ends.
So, take care in what you put on the first line of the Build Step. If you do not know how shell works, do not put a hash-bang at all and let Jenkins decide how the script should be run.
If you need more control over how the Build Step is executed, you should study the man page of the shell you use to find out how to make it behave the way you want. Jenkins doesn't have much of a role in here. It just executes the shell you wanted the way you wanted.
Jenkins can only see the result code of the last command run so it has no way of knowing what the result of rake test:units is.
The easiest thing is probably to have each command of those commands as a separate jenkins build step.
An alternative solution is change your first line to the following:
#!/bin/bash -e
This tells your script to fail if any of the commands in the script return an error.
See: Automatic exit from bash shell script on error