Much has been written about testing of code. But how do we ensure that our design is functionally correct in the first place? Just as we have JUnit for testing Java code, are there some tools that can be used, say, to test a UML based design, where the tests are expressed in form of functional requirements?
These are some vague thoughts, but wanted to know if there's a methodical, automatable approach to testing the design first. In other words, can we also have 'Test Driven Design'?
are there some tools that can be used to test a UML based design, where the tests are expressed in form of functional requirements?
You cannot test a design based on functional requirements because the satisfaction of functional requirements depends on the implementation, not on the design.
In other words, you can follow a bad design (any one of the infinite possible designs) and still meet that set of functional requirements if you implement the required behavior.
Interesting topic!
Firstly, no software architects in my personal network use UML as the only way to design their systems, and I also know of no software architects who create UML at the level of detail required to execute a mechanical test.
Secondly, I personally have a deep dislike of the UML modeling tools. If such a formal verification method is implemented, it's likely to be in something like Rational Rose - I swore long ago I'd never go anywhere near that again.
However, having said all that - in formal software shops, it's common to have requirements tracability, typically implemented as a matrix which shows business requirements on one axis, and design artifacts on the other. This way, you can see whether any requirements are not matched with a corresponding solution, or if there are elements in the solution which do not meet a specific business requirement.
Keeping this matrix updated is a pain, so it's not often used in agile teams, but if you're building softwware for a bank or the space shuttle, it is a valuable technique.
This tells you whether your design is complete - though not whether it's "right".
There's no way I know of to tell whether a design is "right" without either building it and testing, or relying on human experience and knowledge.
Related
Most object oriented analysis and design books and resources describe the process where the analysis phase is followed by identifying classes. I understand that experience will often give you an idea of which architecture (if any) you should apply but is there a specific point in the object oriented design phase where this should occur? I'm about to start a large personal project and I want to make sure my choice of architecture doesn't disregard something from the analysis phase.
This question implies that architectural patterns are chosen all at once. In an ideal world (where requirements don't change, and where developers can read a client/stakeholder's mind), it might be possible to come up with a huge design up front, and stick to it. That never happens. The only way to come up with software that is both functional and well designed is to constantly refactor as requirements become more clear. And at each stage of refactoring, it's possible that a subsystem requires a different architectural pattern.
Of course, it's important to enter a project with some kind of "plan of attack". But don't expect the design phase to be over once that is completed. No one understands all the requirements up front (even if you're your own client). Things will always change.
In short, if you're not choosing architectural patterns throughout the development process, you're either a mind reader, or you're racking up technical debt.
I am interested in proving that some robot controller does not reach any faulty state, which I would define by a set of predicates. I know that there are open-source software tools to achieve that. For instance, I heard of BLAST (Berkeley Lazy Abstraction Software Verification Tool), but are you aware of any other that may be simpler to use and/or more targeted to my particular application?
Have you ever used BLAST or another such tool in one of your project, and do you think that the benefits outweigh the effort needed to deploy such tools?
You might find Frama-C useful.
For evaluations by people who are not Frama-C developers, see these two articles. Some engineers developing safety-critical code (e.g. DO-178B level A) have found formal annotations and analysis based on weakest precondition techniques worth the investment, but traditional tests are very expensive for them. This last link is about Caveat, a closed-source analyzer that Frama-C intends to replace in due time.
Your question makes it sound as if you might perhaps appreciate Frama-C's Aoraï plug-in.
Whether this is all time well spent in your case is probably more a matter of whether you consider learning about these techniques a joy or a chore.
Let's say I've made a list of concepts I'll use to draw my Domain Model. Furthermore, I have a couple of Use Cases from which I did a couple of System Sequence Diagrams.
When drawing the Domain Model, I never know where to start from:
Designing the model as I believe the system to be. This is, if I am modelling a the human body, I start by adding the class concepts of Heart, Brain, Bowels, Stomach, Eyes, Head, etc.
Start by designing what the Use Cases need to get done. This is, if I have a Use Case which is about making the human body swallow something, I'd first draw the class concepts for Mouth, Throat, Stomatch, Bowels, etc.
The order in which I do things is irrelevant? I'd say probably it'd be best to try to design from the Use Case concepts, as they are generally what you want to work with, not other kind of concepts that although help describe the whole system well, much of the time might not even be needed for the current project. Is there any other approach that I am not taking in consideration here? How do you usually approach this?
Thanks
Whether DDD, or not, I would recommend with determining the ubiquitous language (UL) by interviewing the product owner(s). Establishing communication in a way that will have you and the product owners speaking the same language not only aides in communication, but being able to discuss the project in common terms tends to help the domain model define itself.
So, my answer is basically to discuss, listen, and learn. Software serves a need. Understanding the model from the viewpoint of the experts will lay the solid groundwork for the application.
I'd start by a drawing a class diagram with all the relationships and implement only the classes that are necessary according to the requirements of your application.
You can use an anemic approach (attributes plus getters and setters) to keep things simple and avoid the step of writing business logic in the same step. With an anemic model, the logic would go into a corresponding Service class. That way you can consider Use Cases later on.
I know some people don't appreciate this way of doing things but it does help with maintenance and avoids some dependency issues.
Answer to devoured elysium's question below:
In terms of analysis, starting with Use cases (What) and then proceeding to the class diagram (How) sounds like a good rule of thumb. Personally, I'd do the Sequence diagram (When and Who?) afterwards, as you'd need to know between which processes/objects messages need to be sent.
Beyond that my take on things is that UML is simply a way to model a system/project and not a methodology by itself (unlike Merise, RAD, RUP, Scrum, etc.). There is nothing stopping someone starting off with any diagram as long as they have the sufficient information to complete it. In fact, they should be done simultaneously since each of the diagrams is a different perspective of the same system/project.
So, all in all it depends on how you go about the analysis. During my studies I was taught the rigid waterfall approach, where you do a complete analysis from start to finish before producing some code. However, things can be different in practice, as the imperative might be to produce a working application in the least time possible.
For example, I was introduced to the Scrum methodology recently for an exercise involving the creation of a web site where people can post their fictions. As there was a time constraint and a clear vision of what should be achieved, we started right away with a bare bones class diagram to represent the domain model. The Use cases were then deduced from a series of mock screens we'd produced.
From memory, the classes were Story, Chapter, User and Category. This last class was phased out in favour of a more flexible Tag class. As you'd imagine, the complete class diagram of the existing project would be much more complex due to applying domain driven design and the specificities of the Java programming language.
This approach could be viewed as sloppy. However, a web site like this could easily be made in a couple of weeks using an iterative process and still be well designed. The advantage an iterative process has over the waterfall approach is that you can continually adjust requirements as you go. Frequent requirements changing is a reality, as people will often change their minds and the possibility of producing a working application after each iteration allows one to stay on course so to speak.
Of course, when you're presenting a project to a client, a complete analysis with UML diagrams and some mock screens would be preferable so they have an idea of what you're offering. This is where the UML comes in. Once you've explained some of the visual conventions, an individual should be able to understand the diagrams.
To finish off, if you're in the situation where you're trying to determine what a client wants, it's probably a good idea to gradually build up a questionnaire you can bring with you. Interviewing a person is the only way you can determine what concepts/features are really needed for an application, and you should expect to go back in order to clarify certain aspects. Another tip would be to do some quick research on the web when you're confronted with a subject matter you're unfamiliar with.
In your example, this would be to go through the basics of anatomy. Among other things, this will help you decide what the model should contain and what granularity it should have (What group of organs should be considered? How precise does it need to be? Do only the organs need to modeled or should they be decomposed into their constituents like tissues, cells, chemical composition, etc. ?).
I think the place to start would be whatever feels logical and comfortable. It's probably best to start with the use cases, as they give you clear direction and goals, and help you avoid YAGNI situations. Given that you should be trying to develop a strong domain model, it shouldn't really matter, as the whole picture of the domain is important.
I would like to share my experience for such type of situations. I usually start with writing tests and code. And try to cover one end to end use case. This gives me fair enough idea about problem and at the end I also have something working with me which I can show case to my client. Most of the time subsequent stories build on top of previous one, but it also happens to me that subsequent stories require changes in the previous model I came up with. But this does not impact me as I already have good test coverage. In this way I came up with the model which fits for the current problem, not the model which maps the real world.
You start with Business Requirements which can be formalized or not. If formalized you would use Use Case Diagrams.
For example here are use case diagrams for an e-commerce app:
http://askuml.com/blog/e-commerce/
http://askuml.com/files/2010/07/e-commerce-use-case.jpg
http://askuml.com/files/2010/07/e-commerce-use-case2b.jpg
From these use cases, you can naturally deduce the business entities: product, category of product, shopping cart, ... that is start to prepare class diagrams.
This is best practice in many methodologies but this is also just common sense and natural.
Short answer
Pick a use case, draw some collaboration diagram (and a class diagram) to realize the domain objects involved. Concentrate only on those objects participated in order to accomplish use case goal. Write TDD test case to set the expectation and gradually model your domain classes to meet the expectations. TDD is very helpful to understand the expected behaviors and it helps to get the cleaner domain model. You will see your domain evolve gradually along with the TDD expectations.
Long answer
My personal experience with DDD was not easy. That was because we didn't have necessary foundations in the first place. Our team had many weak points in different areas; requirements were not captured properly and we only had a customer representative who was not really helpful (not involved). We didn't have a proper release plan and developers had a lack of Object Oriented concepts, best principles and so on. The major problem we had was spending so much time on trying to understand the domain logic. We sketched many class diagrams and we never got the domain model right, so we stopped doing that and found out what went wrong. The problem was that we tried too hard to understand the domain logic and instead of communicating we made assumptions on the requirements. We decided to change our approach, we applied TDD, we started writing the expected behavior and coded the domain model to meet the TDD's expectations. Sometimes we got stuck writing TDD test cases because we didn't understand the domain. We straight away talked to the customer representative and tried to get more input. We changed our release strategy; applied agile methodology and release frequently so that we got real feedback from the end user. However, needed to ensure the end user expectation was set at the right level. We refactored based on the feedback, and in that way the domain model evolved gradually. Subsequently, we applied design patterns to improve reusability and maintainability. My point here is that DDD alone cannot survive, we have to build the ecosystem that embraces the domain, developers must have strong OOP concepts and must appreciate TDD and unit test. I would say DDD sits on top of all the OOP techniques and practices.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
So...
I teach formal methods in software engineering. I also teach "agile methodologies". Most people seem to think this is contradictory. I think it makes a lot of sense... I also work for a company, where we need to actually get things done :) While I can apply my earned skill points on "specification" in a day-to-day basis, my colleagues typically flee away from the word "formal".
I used to think that this was due to the intrinsic way we learn how to program: we are usually driven to find a working solution, not to understand the problem. Then I thought this was due to the fact that most people in the formal community are not engineers, but mathematicians or computer scientists. Nowadays, I wonder if it just because the formal-methods community hide behind some kind of "obfuscation" law to use all the available UNICODE symbols, actively develop rude, unesthetic tools, and laugh in the face of standards.
Yes, I've been moving from a "blame them" to a "blame us" perspective ;-)
So, my question is: do you use any kind of formal methods in your company? Have you introduced them, or were they pre-requisites? What techniques do you use to clear the fog of mathematics from people's fears and incite them to use formal methods? What do you think current tools are lacking for a more general usage?
The key to getting people to buy into any methods or methodologies is to show them how it solves problems they are having. If they can see it will make their lives better you have a much improved chance of getting them to adopt the techniques.
And if you can't show them that, perhaps you wanted to adopt the methods based on philosophy rather than practicality. Unless the others share your philosophy then you're not going to get anywhere. And perhaps you shouldn't.
Over the decades there have been a great many methodologies. Newer ones always address the shortcomings of the old ones, yet projects still get in trouble and fail. Why? Because the rock stars that come up with new methodologies are rock stars, and have made a new methodology precisely because they understand the underlying issues and how to apply them. Those who come after tend to blindly follow the recipe, and it doesn't work so well.
So I think the best thing is to teach about the underlying problems and then show how various methods attempt to deal with those problems. The differences in companies, projects, and teams is so great that no one methodology can be applied successfully to all combinations. Learning to choose an appropriate tool and apply it well is crucial.
Thank you for all contributions. They are very insightful. Allow me to flame a bit (don't take it personal, though :-)
Most people seem to think that formal methods are just about program verification. Or critical systems. This may be true if we pursue the ultimate cliche: to prove we are doing the program right (v.s. validation, which asks, as a contributor said, if we are doing the right program).
But consider model finding/checking tools, such as Alloy. Learning to use a tool like this takes a negligable ammount of time for anyone used to UML and OO. Still, it can give you immediate insight over your model. It usually takes no more than 10 minutes to find a counter-example over a small enough subset of the model one's trying to use (and that includes describing the model in Alloy in the first place).
Take requirements engineering as an example. One usually draw a lot of UML. Few people use OCL, though, and many business rules are informally annoted in natural language. Why? Time constraints?
Now consider the fact that the majority just uses her/his gut-feeling to prove that a model is satisfiable. Again, why? I can take the same amount of time (probably even less, since I don't need to care about drawing aesthetics) to write that model in Alloy, and just check for satisfiability? And what kind of mathematics do I need to now? "Predicates"? Fancy name for IFs and booleans ;-) Quantifiers? Fancy names for ForEachs()...
What about big information systems? They don't need to be critical... Just try to analyze in your head a conceptual (not implementation!) diagram with over 600 classes. I see many people banging their head in the wall with easy-to-make model mistakes because they missed some constraint, or the model allows stupid things to happen.
The fact is, one does not need to use formal approaches from head to tail. Granted, I could prove a whole application in Coq, and certify that it is 100% compliant with some specification. This may be the Computer Scientist/Mathematician approach.
Still, with a GTD philisophy, why can't I delegate some tasks for the computer and allow it to help improving my development? Is it really a matter of "time", or plain, simple lack of technical abilities and will to learn/inovate?
Working with line of business IT development in an enterprise means having to transfer knowledge about the business from actual business people into the heads of developers. While I myself find abstract maths to be one of the greatest pastimes there is, it's a terrible communications tool. And communications is what it's all about. While I might conceivably have some success convincing IT people to embrace more abstract notations, I basically have no chance with the business people.
While there are some areas where I can see a role for formal methods in an enterprise (math- and logic-heavy specialist software, significant need for provable properties as in safety critical software) they provide little help with getting correct requirements on e.g. how to fulfil a customer order by issuing one or more supply orders to a set of possible external or internal providers.
I think the jury is still out on model based approaches and domain specific languages. I think they will succeed or fail depending on whether they provide quicker feedback from IT to the wishes and needs of the business side, and whether they presume business people will have to do any significant studying.
Technology is easy. Communication is hard. Formal methods may help us do things right, but those I've seen do nothing to help us do the right things. (Yes, these are cliches, but that's because they're inescapably and painfully true.)
I'm taking a course on 'Specification and Verification'. As part of the course structure we are doing the following-
1. Learning tools like PVS(Prototype Verification System) http://pvs.csl.sri.com/ and SMV(Software Modeling and Verification) http://www.cs.cmu.edu/~modelcheck/smv.html
2. Apart from that we do dissect accidents which happened because of software failures. For e.g. - Failure of Ariane V
I feel formal methods are more applicable to scenarios where the failure cost is more than the design cost. And it seems apt to use them for softwares being used in critical systems. I guess it is used in avionics, chip design etc. and the current automobile industry is also drafting it into practice.
I have tried to get people to embrace formal specification methods a few times (Z and Alloy) and have made the same expirience that you have: Most people, while feeling that they serve a useful purpose, are very uncomfortable using them for actual work.
Funny enough, the same people are more than happy to produce utterly useless UML diagrams in ginormous quantities.
I think there are two main reasons for this:
a.) Many developers are uncomfortable with the level of abstraction required by a formal approach. The fact that most entry-level mathematics education is all calculus and non discrete-mathematics might have to do something with this.
b.) Formal methods require a very bottom up design aproach where you design your core model from the ground up and make it airtight and then connect it up to the actual user requirements by providing an interface on top of it. Since we tend to have requirements drive development efforts, a top-down approach feels more natural although it often leads to inconsistent models. It's like retrofitting a basement underneath your house after it has already been built.
Formal methods make no sense in systems where the cost of failure is low.
In a production web application, you've got multiple front-end boxes, multiple back-end boxes, multiple database boxes - if a program on any one of them fails, it's a non-event. Hardware is so cheap that you can build these systems for far less than the cost of formally specifying all your software.
What strategies have you used with Model Based Testing?
Do you use it exclusively for
integration testing, or branch it
out to other areas
(unit/functional/system/spec verification)?
Do you build focused "sealed" models or do you evolve complex onibus models over time?
When in the product cycle do you invest in creating MBTs?
What sort of base test libraries do you exclusively create for MBTs?
What difference do you make in your functional base test libraries to better support MBTs?
[There are several essays worth reading on this. Stack Overflow won't let me post more than one, so I've aggregated them in a blog post, linked at the end of this answer.]
First, a quick note on terms. I tend to use James Bach’s definition of Testing as “Questioning a product in order to evaluate it”. All test rely on /mental/ models of the application under test. The term Model-Based Testing though is typically used to describe programming a model which can be explored via automation. For example, one might specify a number of states that an application can be in, various paths between those states, and certain assertions about what should occur in on the transition between those states. Then one can have scripts execute semi-random permutations of transitions within the state model, logging potentially interesting results.
There are real costs here: building a useful model, creating algorithms for exploring it, logging systems that allow one to weed through for interesting failures, etc. Whether or not the costs are reasonable has a lot to do with what are the questions you want to answer? In general, start with “What do I want to know? And how can I best learn about it?” rather than looking for a use for an interesting technique.
All that said, some excellent testers have gotten a lot of mileage out of automated model-based tests. Sometimes we have important questions about the application under test that are best explored by automated, high-volume semi-randomized tests. Harry Robinson (one of the leading theorists and proponents of model-based testing) describes one very colorful example where he discovered many interesting bugs in Google driving directions using a model-based test (written with ruby’s Watir library). 1
Robinson has used MBT successfully at companies including Bell Labs, Microsoft, and Google, and has a number of helpful essays.[2]
Ben Simo (another great testing thinker and writer) has also written quite a bit worth reading on model-based testing.[3]
Finally, a few cautions: To make good use of a strategy, one needs to explore both its strengths and its weaknesses. Toward that end, James Bach has an excellent talk on the limits and challenges of Model-Based Testing. This blog post of Bach’s links to his hour long talk (and associated slides).[4]
I’ll end with a note about what Boris Beizer calls the Pesticide Paradox: “Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffective.” Scripted tests (whether executed by a computer or a person) are particularly vulnerable to the pesticide paradox, tending to find less and less useful information each time the same script is executed. Folks sometimes turn to model-based testing thinking that it gets around the pesticide problem. In some contexts model-based testing may well find a much larger set of bugs than a given set of scripted tests…but one should remember that it is still fundamentally limited by the Pesticide Paradox. Remembering its limits — and starting with questions MBT addresses well — it has the potential to be a very powerful testing strategy.
Links to all essays mentioned above can be found here: http://testingjeff.wordpress.com/2009/06/03/question-about-model-based-testing/
We haven't done any/much I&T and use unit testing almost exclusively, seasoned with a bit of system testing. But our focus is clearly on unit testing. I'm pretty strict on the APIs we build/provide, so the assumption is, if it works by itself, it will work in conjunction and there hasn't been much wrong in it yet.
Our models are focused on a single purpose/module with as little dependencies as possible.
The focus is always to start as early as possible (TDD-kinda), but unfortunately we don't always get to it. The problem is, you always have to sell it to management and then it's hard because while testing improves stability (overall QA), the people from the outside (outside of tech) can't really relate to what that means until something bad happened.
Since we use PHP, we employ PHPUnit for the unit tests. All in all, we do CI with various different tools. :)
Harry Robinson, an author of MBT-books and worked a lot with it for example at Google and Microsoft have this site with some great info and whitepapers.
http://www.geocities.com/model_based_testing/
The best way is to try by yourself a Model based testing tool. It's the best way for know if the model based testing is adapted in your context. And what sort of strategies is the good one.
I advise you the "MaTeLo" tool of All4Tec (www.all4tec.net)
"MaTeLo is a test cases generator for black box functional and system testing. Conformed to the Model Based Testing approach, MaTeLo uses Markov chains for modeling the test. This statistic addin allows products validation in a Systematic way. The efficiency is achieved by a reduction of the human resources needed, an increase of the model reuse and by the enhancement of the test strategy relevance (due to the reliability target). MaTeLo is independent and user-friendly, offers to the validation activities to pass from test scripting to real test engineering and to focus on the real added value of testing: the test plans"
You can ask an evaluation licence and try by yourself.
You can find some exemples here : http://www.all4tec.net/wiki/index.php?title=Tutorials