I've been studying the discipline of Test Driven Development and for me it has worked well for implementing algorithms and input-output systems.
So, as far as I understand, the "essence" of TDD is to actually write tests for each requirement of the application. Normally this requirement defines a behavior with inputs and outputs.
So, now. Going to real time applications. Let´s say your application runs an infinite loop. A common example is a graphics application or an audio application where each iteration of the loop means output to the screen/speakers.
Having a system like that, let´s say the requirement is something like:
"When pressing Enter button, the screen should show a circle with the text Hello World inside the circle"
So how would you test drive this kind of requirement.
Another example, just to illustrate my question better.
Let´s say I´m emulating a CPU. In each iteration, I fetch an opcode from the file, translate it and execute it. Basically there is no actual output. What happens is there is input which changes some state related to the emulation of the CPU. So no public interface for the CPU internals.
My requirement would say something like "Implement the mov operation on the cpu emulator"
Which may be part of the bigger requirement "Implement opcodes emulation"
So. What would be a good approach for tackling this behaviors/requirements using TDD?
What would be a good approach for tackling this behaviors/requirements using TDD?
What I normally see happen is that the design partitions into two pieces
A piece that is complicated, but really "easy" to test
A piece that is hard to test, but is really "simple"
Basically, you are arranging that the "risk" lies predominantly in the code that is easy to test.
One of the properties of code that is really simple: it also tends to be very stable. The combination of low risk, stable, and difficult to test means that investing in test automation here is less attractive.
Basically there is no actual output.
The write a no-op; there's no advantage to doing any work that doesn't have an observable side effect of some sort.
It's a common pattern that we look at an intermediate stage of the output. For example, if we are supposed to produce a tone from the speakers, what we might do in code is create a seam between the work of choosing the tone, and the actual mechanism of delivering the representation of the tone to the speaker. At that seam, we also capture information so that we can check it.
So no public interface for the CPU internals.
Having a test interface is normally a satisfactory outcome. Often, it will turn out that you want to publish the test interface, for use in satisfying monitoring or observability requirements.
The public interface is way too broad to test that single behavior.
Yes, that's common. The usual response is to refactor your broad testable module into several, perhaps even many, narrow testable modules. Review Parnas 1971.
It may help also to think about the distinction between public methods (accessible outside a module) and published methods (accessible by code that you don't control).
Related
If you have nothing, you cannot write a test because there is nothing to test. This seems pretty obvious to me but never seems to be addressed by proponents of TDD.
In order to write a test, you have to first decide what the method or function looks like that you're going to test. You have to know what parameters to pass to it and what you expect to get back. That is what comes first, not the test.
Tests can never come first. The thing that comes first is the design which specifies what classes and methods are going to exist.
It's true that in order to write a test, the test writer must form some conception on how the test code can interact with the System Under Test. In that sense, conceptual design 'comes first'.
Test-driven development (TDD), however, is valuable because it's not (only) a quality assurance methodology. It's first and foremost a fast feedback loop.
While you may have an initial design in mind, once you start to write a test, you may discover that this design doesn't work (or is awkward to use). This often happens, and should cause you to immediately adjust course.
The red-green-refactor cycle suggests a model to think of TDD. Each such cycle may be a minute or two.
Thus, you may start with an initial design in mind, but then adjust it (or completely rethink it) every other minute.
never seems to be addressed by proponents of TDD
I disagree. Plenty of introductions to TDD discuss this. Two good books that discuss this (and much more) are Kent Beck's Test-Driven Development by Example and Nat Pryce and Steve Freeman's Growing Object-Oriented Code Guided by Tests.
It's the other way round.
If you write a test that calls a function which does not exist, your test suite fails and you get an error forcing you to define that function, just like writing any other test forces you to write the implementation.
Your tests don't need to run to be good tests. But this kind of test is not meant to stay in your test suite. They are sometimes referred to as "staircase tests": you need to write them to get going but they are only instrumental.
What happens generally is that as soon as this test passes, you make it fail by being more specific. Technically the test you end up with is the same you would have written after the fact and it didn't take more time to write it, but during this process you were able to run the test suite one or more times, so you're spending less time in an invalid state, so to speak.
I would like to add that there is nothing untrue in your question, but your conclusion doesn't follow the premise: it is true that what come first is the specification, but there is nothing inconsistent with formalising this specification in a test before the code is written. the spec, and the tests, force you to write the code. TDD is an incremental way of formalising the spec that ensures the spec always comes first.
To write a test, you have to first decide what the method or function looks like that you're going to test. You have to know what parameters to pass to it and what you expect to get back. THAT is what comes first, NOT the test. Tests can NEVER come first. The thing that comes first is the design which specifies what classes and methods are going to exist.
Not quite right (not entirely wrong either - It's Complicated[tm])
If you look at the first example in Test Driven Development by Example, you'll see that Beck doesn't begin with classes and methods. He doesn't even begin with a test.
The very first thing that he creates is a "to-do" list, where each of the entries in the todo list is a representation of a behavior (my terminology, not his). So we see things like
$5 + 10 CHF = $10 if rate is 2:1
These days, you'd be more likely to see this idea expressed as Hoare triple (Given/When/Then, Arrange/Act/Assert, etc). But what we have here is a reminder to the programmer that we want an automated check that measures the result of adding two different currencies together, and confirms that the result matches some specification.
In his exercise, his to do list includes a "simpler" test, which is the one he attempts first
$5 * 2 = $10
That same todo list also includes some other concerns the has about the design, NOT expressed in test form. Also, the list grows as he works through the problem.
In this sense, the test absolutely comes first. We write the test in a language to be consumed by humans. Translating the test into a language understood by the machine comes later.
In the second step, where we describe the test to the machine, things get messier. It is absolutely the case that, as we are designing the test, we are also designing the communication protocol that allows the test to measure what the production code does. So there's a certain amount of communication design that is happening in parallel with the "test" design.
But even here, the test is not specifying all of the classes that are going to exist, it's only specifying what it needs to perform its measurement. We describe a facade, but we aren't specifying what lies beyond that facade.
It can happen, as we design more of the system, that the facade we specify is used only by tests, as a way of communicating with a different underlying design of production code.
(Note: I say classes here for consistency with the question and with early literature, taken primarily from examples in Smalltalk or Java. Feel free to substitute "functions" for "classes" if that makes you more comforatble.)
Now, the most common case is that the facade is the production code; we don't typically add elements to the design until we have a non-speculative motivation for them.
"Unit testing" puts some strain on these ideas - how can you possibly write a unit test without first designing your unit boundaries?
The real answer is an unfortunate one -- Kent Beck didn't write unit tests. He wrote "programmer tests" (a term that got retconned in later) and called them unit tests.
Using the testing language of the 1990s (which is when all this mess started), a more appropriate term is probably "composite tests".
You've also got "the London School", that was trying to figure out how to TDD a particular design style; writing a test for that style requires a more complicated testing facade "up front" (roles and interfaces and stable substitute implementations and so on).
It can also be worth keeping in mind the setting.
(Disclaimer: this isn't something I witnessed first hand - think "based on a true story" rather than "facts")
TDD (and its parent idea "test first" programming in XP) are pushing back against "up front design" of the sort where you decide what the class hierarchy and relationships should be, and document them, before you actually sit down to write the code.
The core argument being that the design process needs shorter feedback loops; that we don't get deeply committed to a particular design until we've acquired a lot of evidence that it is going to work out OK.
All that said, yes it is absolutely the case that TDD, as a technique, works so much better in the hands of someone who is already good at software design. See Michael Feathers on the Look, Ma, no hands! era.
There is no magic.
I have a task to write tests for future Django Channels+DRF project, don't ask why (we only have swagger documentation for now). So the tests have to test the user use cases (like scenario that may be complex). I have researched about that and found BDD. Here is the question, considering that our project later may have simple unit tests too what should I use, i.e. BDD seems decent but I think it may be excessive for use and may be there is a way of just writing unittests for user use case scenarious and I can get by with that. Does anyone have experience with that? It would be great if you provide articles and code examples.
Scenarios are a bit different to use-cases. A use-case often covers several capabilities. For instance, in the simple laundry use-case shown here, a housekeeper does several things when performing a wash:
washes each load
dries each load.
folds certain items
irons some items
All of these go into the "weekly laundry" use-case.
A scenario in BDD is much more fine-grained. It describes one capability taking place in a particular context or set of contexts. So for instance you might have:
Given the weekly laundry has been washed and dried
And it contains several sheets
And some underpants
When the housekeeper does the folding
Then the sheets should be folded
But the underpants should not.
You can see that we've skipped a couple of the capabilities. This scenario is focused on the capability of folding, and shows how a well-behaved housekeeper would do it. Washing and drying would have to be covered in separate scenarios.
So that's the difference between a use-case and a scenario. Now let's look at a unit test.
When we write code, we don't write it all in one big class or function. We split it up into small pieces. In the same way that a scenario describes an example of the behaviour of the system from the perspective of the users, a unit test describes the behaviour of a class or other small piece of code from the perspective of its users - usually other classes!
So let's imagine that we're on a car purchasing site. We have several capabilities:
Authentication
Searching for cars
Purchasing a car
Listing a car
Removing a car from the list
Each of these will have lots of different classes making it up. Even searching for a car could involve a front-end, a search component, a database of cars, a persistence layer, a webserver, etc.. For each piece of code, we describe the behaviour of that code.
(BDD actually started out at this level; with examples of how classes behave - JBehave was intended to replace JUnit. But JUnit got better and we didn't need this bit any more. I still find it helpful to think of these as examples rather than tests.)
Typically I'll have both scenarios and unit tests in my codebase; one set of them looking from a user / stakeholder perspective at the whole system, and the other set describing my classes in finer detail.
The scenarios help me show how the system behaves and why it's valuable. The unit tests help me drive out good design and separate responsibilities. Both of them provide living documentation which helps to keep the system maintainable and make it easier for newcomers to come on board.
Generally this is how I program:
I have a rough idea of what I want to achieve
I talk to someone about it and write down some scenarios
If we don't quite know what we're looking for, I'll get something working (a spike)
Once we understand better what we're looking for, I automate the scenario first
I take the simplest case and start writing the UI
When the UI needs another class to work, I write some examples of how that code should work (unit tests) first
Then I write the code (or refactor it, because spikes are messy)
When that code needs another class to work, I write some examples of it
If I don't have code that's needed at any point in my unit tests, I use mocks.
Generally we keep the scenarios and the unit tests in different places.
You can see some examples of how I've done this here. It's a tetris game with scenarios which automate the whole game through the UI, and lower-level unit tests which describe the behaviour of particular pieces like the heartbeat which drops the shapes.
Having said that - if your codebase is very simple, you can probably get away with just the scenarios or just the unit tests; you might not need both. But if it starts getting more complex, consider refactoring and adding whatever you need. It's OK to be pragmatic about it, as long as it's easy to change.
i want to learn test automation using meta programming.i googled it could not find any thing.can anybody suggest me some resources where can i get info about "how to use Meta Programming for making test automation easy"?
That's a broad topic and not a lot has been written about it, because of the "dark corners" of metaprogramming.
What do you mean by "metaprogramming"?
As background, I consider metaprogramming to be any activity in which a tool (which we call a "metaprogramming tool") is used to inspect or modify the application software to achieve some effect.
Many people consider "reflection" to be a kind of metaprogramming; other consider (C++-style) templates to be metaprogramming; some suggest aspect-oriented programming.
I sort of agree but think these are weak versions of what you want, because each has severe limits on what it can see or do to source code. What you really want is a metaprogramming tool that has access to everything in your source program (yes, comments too!) Such tools are called Program Transformation Systems (PTS); they work by parsing the source code and operating on the parsed representation of the program. (I happen to build one of these, see my bio). PTSes can then analyze the code accurate, and/or make reliable changes to the code and regenerate valid source with the changes. PS: a PTS can implement all those other metaprogramming techniques as special cases, so it is strictly more general.
Where can you use metaprogramming for testing?
There are at least 2 areas in which metaprogramming might play a role:
1) Collection of information from tests
2) Generation of tests
3) Avoidance of tests
Collection.
Collection of test results depends on the nature of tests. Many tests are focused on "is this white/black box functioning correctly"? Assuming the tests are written somehow, they have to have access to the box under test,
be able to invoke that box in a realistic ways, determine if the result is correct, and often tabulate the results to that post-testing quality assessments can be made.
Access is the first problem. The black box to be tested may not be easily accessible to a testing framework: driven by a UI event, in a non-public routine, buried deep inside another function where it hard to get at.
You may need metaprogramming to "temporarily" modify the program to provide access to the box that needs testing (e.g., change a Private method to Public so it can be called from outside). Such changes exist only for the duration of the test project; you throw the modified program away because nobody wants it for anything but the test results. Yes, you have to ensure that the code transformations applied to make things visible don't change the program functionality.
The second problem is exercising the targeted black box in a realistic environment. Each code module runs in a world in which it assumes data and the environment are "properly" configured. The test program can set up that world explicitly by making calls on lots of the program elements or using its own custom code; this is usually the bulk of a test routine, and this code is hard to write and fragile (the application under test keeps changing; so do its assumptions about the world). One might use metaprogramming to instrument the application to collect the environment under which a test might need to run, thus avoiding the problem of writing all the setup code.
Finally, one might want to record more than just "test failed/passed". Often it is useful to know exactly what code got tested ("test coverage"). One can instrument the application to collect what-got-executed data; here's how to do it for code blocks: http://www.semdesigns.com/Company/Publications/TestCoverage.pdf using a PTS. More sophisticated instrumentation might be used to capture information about which paths through the code have been executed. Uncovered code, and/or uncovered paths, show where tests have not been applied and you arguably know nothing about what the program does, let alone whether it is buggy in a straightforward way.
Generation of tests
Someone/thing has to produce tests; we've already discussed how to produce the set-up-the-environment part. What about the functional part?
Under the assumption that the program has been debugged (e.g, already tested by hand and fixed), one could use metaprogramming to instrument the code to capture the results of execution of a black box (e.g., instance execution post-conditions). By exercising the program, one can then produce (by definition) "correctly produces" results which can be transformed into a test. In this way, one might construct a huge variety of regression tests for an existing program; these will be valuable in verifying the further enhancements to the program don't break most of its functionality.
Often a function has qualitatively different behaviors on different ranges of input (e.g., for x<10, produced x+1, else produces x*x). Ideally one would like to provide a test for each qualitively different results (e.g, x<10, x>=10) which means one would like to partition the input ranges. Metaprogrammning can help here, too, by enumerating all (partial) paths through module, and providing the predicate that controls each path.
The separate predicates each represent the input space partition of interest.
Avoidance of Tests
One only tests code one does not trust (surely you aren't testing the JDK?) Any code consructed by a reliable method doesn't need tests (the JDK was constructed this way, or at least Oracle is happy to have you beleive it).
Metaprogramming can be used to automatically generate code from specifications or DSLs, in relaible ways. Such generated code is correct-by-construction (we can argue about what degree of rigour), and doesn't need tests. You might need to test that DSL expression achieves the functionaly you desired, but you don't have to worry about whether the generated code is right.
Greetings!
I inherited a C#.NET application I have been extending and improving for a while now. Overall it was obviously a rush-job (or whoever wrote it was seemingly less competent than myself). The app pulls some data from an embedded device & displays and manipulates it. At the core is a communications thread in the main application form which executes a 600+ lines of code method which calls functions all over the place, implementing a state machine - lots of if-state-then-do type code. Interaction with the device is done by setting the state/mode globally and letting the thread do it's thing. (This is just one example of the badness of the code - overall it is not very OO-like, it reminds of the style of embedded C code the device firmware is written in).
My problem is that this piece of code is central to the application. The software, communications protocol or device firmware are not documented at all. Obviously to carry on with my work I have to interact with this code.
What I would like some guidance on, is whether it is worth scrapping this code & trying to piece together something more reasonable from the information I can reverse engineer? I can't decide! The reason I don't want to refactor is because the code already works, and changing it will surely be a long, laborious and unpleasant task. On the flip side, not refactoring means I have to sometimes compromise the design of other modules so that I may call my code from this state machine!
I've heard of "If it ain't broke don't fix it!", so I am wondering if it should apply when "it" is influencing the design of future code! Any advice would be appreciated!
Thanks!
Also, the longer you wait, the worse the codebase will smell. My suggestion would be first create a testsuite that you can evaluate your refactoring against. This makes it a lot easier to see if you are refactoring or just plain breaking things :).
I would definitely recommend you to refactor the code if you feel its junky. Yes, during the process of refactoring you may have some inconsistencies/problems at the start. But that is why we have iterations and testing. Since you are going to build up on this core engine in future, why not make the basement as stable as possible.
However, be very sure on what you are going to do. Because at times long lines of code does not necessarily mean evil. On the other hand they may be very efficient in running time. If/else blocks are not bad if you ask me, as they are very intelligent in branching from a microprocessor's perspective. So, you will have to be judgmental and very clear before you touch this.
But once you refactor the code, you will definitely have fine control over it. And don't forget to document it!! Tomorrow, someone might very well come and say about you on whatever you've told about this guy who have written that core code.
This depends on the constraints you are facing, it's a decision to be based on practical basis, not on theoretical ones. You need three things to consider.
Time: you need to have enough time to learn it, implement it, and test it, without too many other tasks interrupting you
Boss #1: if you are working for someone, he needs to know and approve the time and effort you will spend immediately, required to rebuild your solution
Boss #2: your boss also needs to know that the advantage of having new and clean software will come at the price of possible regressions, and therefore at the beginning of the deployment there may be unexpected bugs
If you have those three, then go ahead and refactor it. It will be surely be worth it!
First and foremost, get all the business logic out of the Form. Second, locate all the parts where the code interacts with the global state (e.g. accessing the embedded system). Delegate all this access to methods. Then, move these methods into a new class and create an instance in the class's constructor. Finally, inject an instance for the class to use.
Following these steps, you can move your embedded system logic ("existing module") to a wrapper class you write, so the interface can be nice and clean and more manageable. Then you can better tackle refactoring the monster method because there is less global state to worry about (only local state).
If the code works and you can integrate your part with minimal changes to it then let the code as it is and do your integration.
If the code is simply a big barrier in your way to add new functionality then it is best for you to refactor it.
Talk with other people that are responsible for the project, explain the situation, give an estimation explaining the benefits gained after refactoring the code and I'm sure (I hope) that the best choice will be made. It is best to speak about what you think, don't keep anything inside, especially if this affects your productivity, motivation etc.
NOTE: Usually rewriting code is out of the question but depending on situation and amount of code needed to be rewritten the decision may vary.
You say that this is having an impact on the future design of the system. In this case I would say it is broken and does need fixing.
But you do have to take into account the business requirements. Often reality gets in the way!
Would it be possible to wrap this code up in another class whose interface better suits how you want to take the system forward? (See adapter pattern)
This would allow you to move forward with your requirements without the poor design having an impact.
It gives you an interface that you understand which you could write some unit tests for. These tests can be based on what your design requires from this code. It ensures that your assumptions about what it is doing is correct. If you say that this code works, then any failing tests may be that your assumptions are incorrect.
Once you have these tests you can safely refactor - one step at a time, and when you have some spare time or when it is needed - as per business requirements.
Quite often I find the best way to truly understand a piece of code is to refactor it.
EDIT
On reflection, as this is one big method with multiple calls to the outside world, you are going to need some kind of inverse Adapter class to wrap this method. If you can inject dependencies into the method (see Dependency Inversion such that the method calls methods in your classes then you can route these to the original calls.
We all know the various ways of testing OO systems. However, it looks like I'll be going to do a project where I'll be dealing with PLC ladder logic (don't ask :/), and I was wondering if there's a good way of testing the validity of the system.
The only way I see so far is simply constructing a huge table with all known states of the system and which output states that generates. This would do for simple 'if input A is on, turn output B on' cases. I don't think this will work for more complicated constructions though.
The verification of "logical" systems in the IC design arena is known as "Design Verification", which is the process of ensuring that the system you design in hardware (RTL) implements the desired functionality.
Ladder logic can be transformed to one of the modern HDL's like Verilog..
transform each ladder
|---|R15|---+---|/R16|---------(R18)--------|
| |
|---|R12|---+
to an expression like
always #(*) R18 = !R16 && ( R15 | R12);
or you could use an assign statement
assign R18 = R16 && (R15 | R12);
a latching relay
assign R18 = (set condition) || R18 && !(break condition);
Then use a free verilog simulator like Icarus to develop a testbench and test your system.
Make sure you're testcases give good CODE coverage of your logic! And If your ladder editing software gives you decent naming capabilities, use them, rather than Rnn.
(Note: in Ladder Logic for PLC convention, Rnn is for internal relays, while, Xnn is an input and Ynn is an output, as can be quickly gleaned from one of the online tutorials.
Verilog will be an easier language to develop your tests and testbenches in!
It may be helpful to program in some unit delays.
Sorry, I have never looked for ladder logic to/from verilog translators..
but ladder logic in my day was only just being put into a computer for programming PLC's - most of the relay systems I used were REAL Relays, wired into the cabinets!!
Good luck.
jbd
There are a couple of ladder logic editors (with simultors) available for free..
here is one that runs on windows supposedly:
http://cq.cx/ladder.pl
We've experimented with test coverage tools for Rockwell Control Logix controllers. Most procedural language test coverage tools do branch coverage or some such; because Relay Ladder Logic typically doesn't branch, this doesn't work very well.
What we have prototyped is MC/DC (modified/condition/decision coverage) for RLL code for Rockwell controllers.. This tells, for each condition in rung, whether that condition has been tested as TRUE, tested as FALSE, and more importantly, if there the condition controlled the output of the decision in the rung (well at least the action controlled by the decision) in both true and false directions under some test.
This work is done using a general purpose program analysis and transformation tool called
DMS used to instrument the RLL code with additional logic to collect the necessary data.
You still have to code unit tests. The easiest way to do that is to get another PLC to act as a replacement for the mechanical hardware you intend to control, and simply write another RLL program to exercise the first one.
There is a program called LogixPro which has an IO simulator for ladder logic, you can try that.
Sometimes on small PLC programs a test program (or subroutine, or ladder file) is written in the project, which is only run when the project is being emulated. The file has some simple logic that says when an output is energised, turn on the input associated with the feedback. You can then control your PLC through whatever HMI is wired up to it and see that the code behaves as expected. Its very important to disable or delete the test program when the software is downloaded to a real site as it can do very strange things in the real world.
On larger projects each device has a simulation mode that does something slightly similar. http://www.batchcontrol.com/s88/01_tutorial/06-modules.shtml
This is nothing like using test frameworks for OO languages, but I haven't really seen any test driven development for PLCs, or even much automated testing.
My boss on a constant basis tells me that the testing is built in the logic itself . PLC’s are in fact deterministic so you should practically be able to follow logic and not need to simulate testing. However we’re not perfect. Having framework would really only allow us to step through what we already know, ladder logic really just takes practice to understand how PLCS work.
That being said I did have some good success with a program I made that essentially flipped on and off IO , it could even simulate the counts of an encoder to test what happens when an object gets to a position. Their were assert statements that could get tripped and inform me where my logic faulted. It did catch a few bugs, and that implementation went very well for a system I’ve never touched. It itself was very beneficial and I do think that it could be useful but I’ve gotten a lot better so I find myself not needing it because of my experience.