Currently, I define the following function in the REPL at the start of a coding session:
(defn rt []
(let [tns 'my.namespace-test]
(use tns :reload-all)
(cojure.test/test-ns tns)))
And everytime I make a change I rerun the tests:
user=>(rt)
That been working moderately well for me. When I remove a test, I have to restart the REPL and redefine the method which is a little annoying. Also I've heard bad rumblings about using the use function like this. So my questions are:
Is using use this way going to cause me a problem down the line?
Is there a more idiomatic workflow than what I'm currently doing?
most people run
lein test
form a different terminal. Which guarantees that what is in the files is what is tested not what is in your memory. Using reload-all can lead to false passes if you have changed a function name and are still calling the old name somewhere.
calling use like that is not a problem in it's self, it just constrains you to not have any name conflicts if you use more namespaces in your tests. So long as you have one, it's ok.
using lein lets you specify unit and integration tests and easily run them in groups using the test-selectors feature.
I also run tests in my REPL. I like doing this because I have more control over the tests and it's faster due to the JVM already running. However, like you said, it's easy to get in trouble. In order to clean things up, I suggest taking a look at tools.namespace.
In particular, you can use clojure.tools.namespace.repl/refresh to reload files that have changed in your live REPL. There's alsorefresh-all to reload all the files on the classpath.
I add tools.namespace to my :dev profile in my ~/.lein/profiles.clj so that I have it there for every project. Then when you run lein repl, it will be included on the classpath, but it wont leak into your project's proper dependencies.
Another thing I'll do when I'm working on a test is to require it into my REPL and run it manually. A test is just a no-argument function, so you can invoke them as such.
I am so far impressed with lein-midje
$ lein midje :autotest
Starts a clojure process watching src and test files, reloads the associated namespaces and runs the tests relevant to the changed file (tracking dependencies). I use it with VimShell to open a split buffer in vim and have both the source and the test file open as well. I write a change to either one and the (relevant) tests are executed in the split pane.
Related
I'm new to Cursive and Clojure in general and am having some difficulty getting a decent TDD workflow.
My problem is that subsequent test runs depend on state in the REPL. For example suppose that you have the code below.
(def sayHello "hello")
(deftest test-repl-state
(testing "testing state in the repl"
(is (= "hello" sayHello))))
If you run this with "Tools->REPL->Run tests in current ns in REPL" it will pass.
If you then refactor the code like this
(def getGreeting "hello")
(deftest test-repl-state
(testing "testing state in the repl"
(is (= "hello" sayHello))))
If you run this with "Tools->REPL->Run tests in current ns in REPL" it will still pass (because the def of sayHello still exists in the repl). However, the tests should fail because the code is currently in a failing state (sayHello is not defined anywhere in the code).
I've tried toggling the "locals will be cleared" button in the REPL window but this does not seem to fix the issue.
If there is a way to run the tests outside of the REPL (or in a new REPL for each test run) I'd be fine with that as a solution.
All I want is that there is a 1 to 1 correspondence between the source code under test and the result of the test.
Thanks in advance for your help.
Yes, it's annoying to have old defs available. I don't even create tests usually (whoops), but this bites me during normal development. If I create a function, then rename it, then change it, then accidentally refer to the first function name, I get odd results since it's referring to the old function. I'm still looking for a good way around this that doesn't involve killing and restarting the REPL.
For your particular case though, there's a couple easy, poor workarounds:
Open IntelliJ's terminal (button at bottom left of the window) and run lein test. This will execute all the project's tests and report the results.
Similarly to the above, you can, outside of IntelliJ, open a command window in the project directory and run lein test, and it will run all found tests.
You can also specify which namespace to test using lein test <ns here> (such as lein test beings-retry.core-test), or a specific test in a namespace using :only (such as lein test :only beings-retry.core-test/a-test; where a-test is a deftest). Unfortunately, this doesn't happen in the REPL, so it kind of breaks workflow.
The only REPL-based workaround I know of, as mentioned above, is to just kill the REPL:
"Stop REPL" (Ctrl+F2)
"Reconnect" (Ctrl+F5).
Of course though, this is slow, and an awful solution if you're doing this constantly. I'm interested to see if anyone else has any better solutions.
You could use Built-in test narrowing (test selector) feature of test-refresh lein plugin. It allows to test only those tests that have been marked with ^:test-refresh/focus meta every time you save a file.
The usual solution for this kind of problem is either stuartsierra/component or tolitius/mount.
A complete description would be out of place here, but the general idea is to have some system to manage state in a way that allows to cleanly reload the application state. This helps keeping close to the code that is saved in your source files while interactively working on the running system.
Thanks to everyone for their suggestions. I'm posting my own answer to this problem because I've found a way forward that works for me and I'm not sure that any of the above were quite what I was looking for.
I have come to the conclusion that the clojure REPL, although useful, is not where I will run tests. This basically came down to a choice between either running a command to clean the repl between each test run (like the very useful refresh function in tools.namespace https://github.com/clojure/tools.namespace) or not running tests in the REPL.
I chose the latter option because.
It is one less step to do (and reloading is not always perfect)
CI tests do not run in a REPL so running them directly in dev is one step closer to the CI environment.
The code in production does not run in a REPL either so running tests outside the repl is closer to the way that production code runs.
It's actually a pretty simple thing to configure a run configuration in IntelliJ to run either a single test or all tests in your application as a normal clojure application. You can even have a REPL running at the same time if you like and use it however you want. The fact that the tooling leans so heavily towards running things in the REPL blinded me to this option to some extent.
I'm pretty inexperienced with Clojure and also a stubborn old goat that is set in his TDD ways but at least some others agree with me about this https://github.com/cursive-ide/cursive/issues/247.
Also if anyone is interested, there is a great talk on how the REPL holds on to state and how this causes all sorts of weird behaviour here https://youtu.be/-RaFcpNiYCo. It turns out that the problem I was seeing with re-defining functions was just the tip of the iceberg.
One option that may help, especially if you're bundling several assertions, or have repeating tests is let. The name-value binding has a known scope, and can save you from re-typing a lot.
Here's an example:
(deftest my-bundled-and-scoped-test
(let [TDD "My expected result"
helper (some-function :data)]
(testing "TDD-1: Testing state in the repl"
(is (= TDD "MY expected result")))
(testing "TDD-2: Reusing state in the repl"
(is (= TDD helper)))))
Once my-bundled-and-scoped test finishes executing, you'll no longer be in the let binding. An added benefit is that the result of some-function will be reusable too, which is handy for testing multiple assertions or properties of the same function/input pair.
While on the subject, I'd also recommend using Leiningen to run your tests, as there are plenty of plugins that can help you test more efficiently. I'd checkout test-refresh, speclj, and cloverage.
I'd like to make some changes to my copy of the rust standard library, then run the tests in the source files I changed. I do not need to test the compiler itself. How can I do this without testing a lot of things I am not changing and do not care about?
Here are some things I've already tried. A note - the specific file I want to play around with is libstd/io/net/pipes.rs in rust 0.12.0.
I tried rustc --test pipes.rs - the imports and options are not set up properly, it seems, and a multitude of errors is the result.
Following the rust test suite documentation, I tried make check-stage1-std NO_REBUILD=1, but this failed with "can't find crate for `green`". A person on the #rust-internals irc channel informed me that "make check-stage1 breaks semi-often as it's not the 'official way' to run tests."
Another person on that channel suggested make check-stage0-std, which seems to check libstd, but doesn't restrict testing to the file I changed, even if I use the TESTNAME flag as specified in the rust test suite documentation.
As of 2022 the way to run the test suite for the rust standard library is documented at https://rustc-dev-guide.rust-lang.org/tests/running.html .
While the documentation mentions the ability to test a single file it appears to malfunction
$ ./x.py test library/core/tests/str.rs
Updating only changed submodules
Submodules updated in 0.01 seconds
Building rustbuild
Finished dev [unoptimized] target(s) in 0.09s
thread 'main' panicked at 'error: no rules matched library/core/tests/str.rs', src/bootstrap/builder.rs:286:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Build completed unsuccessfully in 0:00:00
Later on it mentions a different syntax like ./x.py test library/alloc/ --test-args str which appears to succesfully run the unit tests in library/alloc/tests/str.rs
make check-stage1-std NO_REBUILD=1 or ... check-stage2-std ... should work, if you've done a full build previously. They just build the test runner directly without doing the rest of the bootstrap.
In any case, the full std test runner is built always, since, as you noticed, the imports etc are set up for the full crate. TESTNAME is the correct way to restrict which tests are run, but there's no way to restrict tests are built.
Another option is to pull the test/relevant code into an external file, and yet another is to build the test runner by running rustc manually on libstd/lib.rs: rustc --test lib.rs. One could edit the rest of the crate to remove the tests/code you're not interested in.
I have implemented several packages for a web API, each with their own test cases. When each package is tested using go test ./api/pkgname the tests pass. If I want to run all tests at once with go test ./api/... test cases always fail.
In each test case, I recreate the entire schema using DROP SCHEMA public CASCADE followed by CREATE SCHEMA public and apply all migrations. The test suite reports errors back at random, saying a relation/table does not exist, so I guess each test suite (per package) is run in parallel somehow, thus messing up the DB state.
I tried to pass along some test flags like go test -cpu 1 -parallel 0 ./src/api/... with no success.
Could the problem here be tests running in parallel, and if yes, how can I force serial execution?
Update:
Currently I use this workaround to run the tests, but I still wonder if there's a better solution
find <dir> -type d -exec go test {} \;
As others have pointed out, -parallel doesn't do the job (it only works within packages). However, you can use the flag -p=1 to run through the package tests in series. This is documented here:
http://golang.org/src/cmd/go/testflag.go
but (afaict) not on the command line, go help, etc. I'm not sure it is meant to stick around (although I'd argue that if it is removed, -parallel should be fixed.)
The go tool is provided to make running unit tests easier using the convention that *_test.go files contain unittests in them. Because it assumes they are unittests it also assumes they are hermetic. It sounds like your tests either aren't unittests or they are but violate the assumptions that a unittest should fulfill.
In the case that you mean for these tests to be unittests then you probably need a mock database for your unittests. A mock, preferrably in memory, of your database will ensure that the unittest is hermetic and can't be interfered with by other unittests.
In the case that you mean for these tests to be integration tests you are probably better off not using the go tool for these tests. What you probably want is to create a seperate test binary whose running you can control and write you integration test scripts in there.
The good news is that creating a mock in Go is insanely easy. Change your code to take an interface with the methods you care about for the databases and then write an in memory implementation of that interface for testing purposes and pass it into your application code that you want to test.
Just to clarify, #Jeremy's answer is still the accepted one:
Since my integration tests were only run on one package (api), I removed the separate test binary in the end and created a pattern to separate test types by:
Unit tests use the normal TestX name
Integration tests use Test_X
I created shell scripts (utest.sh/itest.sh) to run either of those.
For unit tests go test -run="^(Test|Benchmark)[^_](.*)"
For integration tests go test -run"^(Test|Benchmark)_(.*)"
Run both using the normal go test
In my Mac Cocoa unit tests, I would like to output some files as part of the testing process, and delete them when the test is done, but only when there are no failures. How can this be done (and/or what's the cleanest way to do so)?
Your question made me curious so I looked into it!
I guess I would override the failWithException: method in the class SenTestCase (the class your tests run in inherits from this), and set a "keep output files" flag or something before calling the super's method.
Here's what SenTestCase.h says about that method:
/*"Failing a test, used by all macros"*/
- (void) failWithException:(NSException *) anException;
So, provided you only use the SenTest macros to test and/or fail (and chances are this is true in your case), that should cover any test failure.
I've never dug into the scripts for this, but it seems like you could customize how you call the script that actually runs your tests to do this. In Xcode 4, look at the last step in the Build Phases tab of your test target. Mine contains this:
# Run the unit tests in this test bundle.
"${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests"
I haven't pored through the contents of this script or the many subscripts it pulls in on my machine, but presumably they call otest or some other test rig executable and the test results would be returned to that script. After a little time familiarizing yourself with those scripts you would likely be able to find a straightforward way to conditionally remove the output files based on the test results.
I'm wondering if there is any solution to let Scala tests run automatically upon change of test class itself or class under the test (just to test automatically pairs Class <---> ClassTest) would be a good start.
sbt can help you with this. After you setup project, just run
~test
~ means continuous execution. So that sbt will watch file system changes and when changes are detected it recompiles changed classes and tests your code. ~testQuick can be even more suitable for you, because it runs only tests, that were changed (including test class and all it's transitive dependencies). You can read more about this here:
http://code.google.com/p/simple-build-tool/wiki/TriggeredExecution
http://php.jglobal.com/blog/?p=363
By the way, ~ also works with other tasks like ~run.