I wonder is there any tool or standard methods to automatically testing programs that produce graphic outputs.
For example, a simple painting application is built allowing users to draw circles and rectangles in specific locations. The tests probably need to check whether the shapes are located in the exact place as specified.
My problem is: is there a standard way to automate the test procedure instead of letting tester manually check the outputs again and again?
There are several approaches, but the most important is, that the GUI part comes last.
The GUI is only responsible (or should be) for visualizing data. This implies that you have some underlying models and functionality, which is told to create a circle or rectangle at a certain position. You would usually test this first in order to make sure, your functionality does the right things and the underlying data is correct. Functionality and models can be fully covered by regular API tests.
Your particular question is to check, whether the visualization part is correct. You have IMHO two options for automation:
Use screenshots and diff the drawing canvas between a static expectation screenshot and the actual test result
Use tracing: You would take a screenshot from the canvas area and convert it to some vector image that allows you to check certain vectors for being at the right place
In general, GUI specifics such as the right color, exact placing are still human tasks. You can only try testing as much as you can using API tests and reduce the human part to a minimum.
Related
i want to know what is the difference between box and module in programming
i have been asked this question
and somehow i am confuse now
reading on web and document that what is box in programming
and i found below link
https://www.nbs-system.com/en/blog/black-box-grey-box-white-box-testing-what-differences/
if the box is the top link and similar link then the box is the testing of the
program and module is the proram
The "box" is just a common word from an object you know from real world. It servers as an analogy with code put together to form some kind of software component. This is because it's internal parts are so related to each other, on responsabilities and communication, on internal data use, on overall goal, that it makes sense to group them together. We name it module and many times this maps to a source file (but doesn't have to), but the same grouping concept applies to classes, packages (usually modules/classes grouped together), libraries or even complete applications or systems.
What the box term mostly refers to is the fact that there is a frontier between what is inside the box and actors interacting with it from the outside (users or other systems).
The box has to present a public interface to external world, in order to become usable, or useful. This has to be documented, otherwise you don't know how to use it. When you use it based on this info only to achieve your goals, not knowing anything about the internals, we say you are using a "black box".
This has lots of advantages because it can make the box a powerful abstraction that is simple to use, wrapping a possible complex implementation inside. It encapsulates things and hides them from the user.
If by any means, you as an external actor use the box in some way, because you know how something is done inside, you are violating this encapsulation principle. You are probably entering in "grey box" usage mode. This is dangerous because if the box is changed your assumptions (and code) may fail.
When using it as a "white box" you really know entirely how it is made inside so you can make very well informed decisions in your code, but now the box code can't be touched really. So there goes the abstraction.
When coding, you mostly want to code against black boxes, and want also to build your own black boxes, for abstraction, cohesion, and modularity reasons.
Grey and white boxes make most sense when it comes the time (hopefully from the start) to test the code you have written. Here you still want to test your system as a black box, but want also to use white box testing, because you want both observable behaviour and internal detailed behaviour to be correct.
Grey testing in particular applies probably when you are testing code you have written that uses other modules or libraries you have not, and generally do not want to test (code of others was tested already), but still you have some knowledge about its internals and you make additional tests to cover your code, that explore this knowledge.
Edit:
So unless they want you to distinguish the module as the code that's inside, and the box as the wrapping public interface for the module, there's no difference actually.
i want to learn test automation using meta programming.i googled it could not find any thing.can anybody suggest me some resources where can i get info about "how to use Meta Programming for making test automation easy"?
That's a broad topic and not a lot has been written about it, because of the "dark corners" of metaprogramming.
What do you mean by "metaprogramming"?
As background, I consider metaprogramming to be any activity in which a tool (which we call a "metaprogramming tool") is used to inspect or modify the application software to achieve some effect.
Many people consider "reflection" to be a kind of metaprogramming; other consider (C++-style) templates to be metaprogramming; some suggest aspect-oriented programming.
I sort of agree but think these are weak versions of what you want, because each has severe limits on what it can see or do to source code. What you really want is a metaprogramming tool that has access to everything in your source program (yes, comments too!) Such tools are called Program Transformation Systems (PTS); they work by parsing the source code and operating on the parsed representation of the program. (I happen to build one of these, see my bio). PTSes can then analyze the code accurate, and/or make reliable changes to the code and regenerate valid source with the changes. PS: a PTS can implement all those other metaprogramming techniques as special cases, so it is strictly more general.
Where can you use metaprogramming for testing?
There are at least 2 areas in which metaprogramming might play a role:
1) Collection of information from tests
2) Generation of tests
3) Avoidance of tests
Collection.
Collection of test results depends on the nature of tests. Many tests are focused on "is this white/black box functioning correctly"? Assuming the tests are written somehow, they have to have access to the box under test,
be able to invoke that box in a realistic ways, determine if the result is correct, and often tabulate the results to that post-testing quality assessments can be made.
Access is the first problem. The black box to be tested may not be easily accessible to a testing framework: driven by a UI event, in a non-public routine, buried deep inside another function where it hard to get at.
You may need metaprogramming to "temporarily" modify the program to provide access to the box that needs testing (e.g., change a Private method to Public so it can be called from outside). Such changes exist only for the duration of the test project; you throw the modified program away because nobody wants it for anything but the test results. Yes, you have to ensure that the code transformations applied to make things visible don't change the program functionality.
The second problem is exercising the targeted black box in a realistic environment. Each code module runs in a world in which it assumes data and the environment are "properly" configured. The test program can set up that world explicitly by making calls on lots of the program elements or using its own custom code; this is usually the bulk of a test routine, and this code is hard to write and fragile (the application under test keeps changing; so do its assumptions about the world). One might use metaprogramming to instrument the application to collect the environment under which a test might need to run, thus avoiding the problem of writing all the setup code.
Finally, one might want to record more than just "test failed/passed". Often it is useful to know exactly what code got tested ("test coverage"). One can instrument the application to collect what-got-executed data; here's how to do it for code blocks: http://www.semdesigns.com/Company/Publications/TestCoverage.pdf using a PTS. More sophisticated instrumentation might be used to capture information about which paths through the code have been executed. Uncovered code, and/or uncovered paths, show where tests have not been applied and you arguably know nothing about what the program does, let alone whether it is buggy in a straightforward way.
Generation of tests
Someone/thing has to produce tests; we've already discussed how to produce the set-up-the-environment part. What about the functional part?
Under the assumption that the program has been debugged (e.g, already tested by hand and fixed), one could use metaprogramming to instrument the code to capture the results of execution of a black box (e.g., instance execution post-conditions). By exercising the program, one can then produce (by definition) "correctly produces" results which can be transformed into a test. In this way, one might construct a huge variety of regression tests for an existing program; these will be valuable in verifying the further enhancements to the program don't break most of its functionality.
Often a function has qualitatively different behaviors on different ranges of input (e.g., for x<10, produced x+1, else produces x*x). Ideally one would like to provide a test for each qualitively different results (e.g, x<10, x>=10) which means one would like to partition the input ranges. Metaprogrammning can help here, too, by enumerating all (partial) paths through module, and providing the predicate that controls each path.
The separate predicates each represent the input space partition of interest.
Avoidance of Tests
One only tests code one does not trust (surely you aren't testing the JDK?) Any code consructed by a reliable method doesn't need tests (the JDK was constructed this way, or at least Oracle is happy to have you beleive it).
Metaprogramming can be used to automatically generate code from specifications or DSLs, in relaible ways. Such generated code is correct-by-construction (we can argue about what degree of rigour), and doesn't need tests. You might need to test that DSL expression achieves the functionaly you desired, but you don't have to worry about whether the generated code is right.
I am working on the Black Box case as part of a Software Test Document and I am not quite sure how to do it. My professor states that we dont need to provide actual results. I am just confused as to what and how I am suppose to do this. Is there any good examples out there that I can reference. I looked at the IEEE 829 but thats not really helpful.
Perhaps your professor is asking you to apply Black Box Design techniques to design test cases to test certain functionality or requirements.
Some examples:
equivalence partitioning
state transition
boundary value analysis
pairwise testing
Definition:
Black box testing is a Testing, either functional or non-functional, without reference to the internal structure of the component or system. So in this method internal structure of program is not considered, tester should provide input set to the program and test whether the program is giving expected output or not.
This method is called as black box because, tester is not aware of the software program. Software program is like a black box; inside which tester cannot see.
BLACK BOX TESTING TECHNIQUES
Following are some techniques that can be used for designing black box tests:
Equivalence partitioning
Equivalence Partitioning is a software test design technique that involves dividing input values into valid and invalid partitions and selecting representative values from each partition as test data.
Boundary Value Analysis
Boundary Value Analysis is a software test design technique that involves determination of boundaries for input values and selecting values that are at the boundaries and just inside/outside of the boundaries as test data.
Graph Based Testing Methods
Each and every application is build up of some objects. All such objects are identified and graph is prepared. From this object graph each object relationship is identified and test cases written accordingly to discover the errors.
Error Guessing
This is purely based on previous experience and judgment of tester. Error Guessing is the art of guessing where errors can be hidden. For this technique there are no specific tools, writing the test cases that cover all the application paths.
Example of Black Box Testing
A tester, without knowledge of the internal structures of a website, tests the web pages by using a browser and providing inputs (i.e. clicks, keystrokes) and verifying whether the output produced is the expected output.
Black box testing is a software testing method where in testers are not required to know coding or internal structure of the software. Black box testing method relies on testing software with various inputs and validating results against expected output. You can write the Software Test Document using various Black box techniques like Equivalence partitioning, State transition, Boundary value analysis etc. based upon your Application scope.
I've searched through the web, but each source says differently.
So I've made two kinds of test. The first one is the 'data cycle test' from TMap and the second a input-output black box test.
Now I know that the black box test, is testing the input-output values without looking at the code.
Below is a template of a Black box test:
Nr. Definition Expected value actual value
But Tmap says that blackbox test is a collection of different kinds of test techniques. Like the 'data cycle test'.
So what is blackbox test exactly? Is it ONE test technique or a collection of tests techniques? And if it is a collection of test technique, what is this expected-actual test technique called?
Black Box Testing:
Approach to testing where the program is considered as a black-box.
Testing based on solely on analysis of requirements [specification, user documentation etc.]
Also called as
Functional testing (Testing all the features)
Data-Driven Testing (Same action for different set of data)
I/O-Driven Testing
Black-Box testing applies to all levels of testing (e.g unit, component and system) - conducted during integration, system and acceptance testing.
Test case design methods:
Commonly used methods:
Equivalence partitioning: It is a process of dividing the input domain into valid/invalid classes, and for a valid input class, make the equal partition so that it will reduce the test cases.
Boundary value analysis: It is a process of checking the inputs on boundaries, one less than boundary and one greater than boundary.
Error guessing: is a ad hoc approach, based on intuition and experience, to identify the tests that are likely to expose errors.
Reference: http://en.wikipedia.org/wiki/Exploratory_testing
Definition:
Black box testing is a Testing, either functional or non-functional, without reference to the internal structure of the component or system. So in this method internal structure of program is not considered, tester should provide input set to the program and test whether the program is giving expected output or not.
This method is called as black box because, tester is not aware of the software program. Software program is like a black box; inside which tester cannot see.
BLACK BOX TESTING TECHNIQUES
Following are some techniques that can be used for designing black box tests:
Equivalence partitioning
Equivalence Partitioning is a software test design technique that involves dividing input values into valid and invalid partitions and selecting representative values from each partition as test data.
Boundary Value Analysis
Boundary Value Analysis is a software test design technique that involves determination of boundaries for input values and selecting values that are at the boundaries and just inside/outside of the boundaries as test data.
Graph Based Testing Methods
Each and every application is build up of some objects. All such objects are identified and graph is prepared. From this object graph each object relationship is identified and test cases written accordingly to discover the errors.
Error Guessing
This is purely based on previous experience and judgment of tester. Error Guessing is the art of guessing where errors can be hidden. For this technique there are no specific tools, writing the test cases that cover all the application paths.
I always thought of it using an analogy. Imagine you’re a mechanic testing whether a car engine works.
Black box testing is like having the hood/bonnet closed, getting in the car and pressing all the buttons and pedals and driving it around to see if it all works correctly. You might not know what type of engine is in the car or exactly how that specific engine works, but you can test whether the engine is working as you’d expect it to by messing around with all the external parts which interact with the engine.
Black box testing is a specification based testing. There variout black box testing techniques like:
1. Equivalence Partitioning
2. Boundary Value Analysis
3. Decision Table
4. State Transition
5. Use Case Testing
Black box testing technique is a dynamic testing technique. In this type od testing technique tester does not know about code. He or She test on the bases of input & output. In this type of testing functional and non functional testing included.
Can you give me certain examples in which black box testing gives the impression that "everything is ok" but white box testing might uncover an error. And examples where white box testing gives an impression that "everything is ok" but black box testing might uncover an error??
Thanx in advance
Blackbox testing can miss pretty much anything that isn't clearly documented or intuitive. For example, in this SO answer entry section, I have a toolbar that I can "test", but w/o taking a look at the code, I may not discover that I need to test the hotkeys, or understand how highlighted text responds to bold and italic attributes in random combinations. I can experiment and figure this out, but it's not as efficient.
In larger applications, control flow issues are often missed - think of obscure logic flows, or even rareley executed case statements.
However, if you do white box testing only, usability is typically the first to suffer. A perfectly functional piece of software can also be difficult to use, have unaligned UI elements, etc.
Why do you ask?
I recently came across it while studying for exam wish me luck.
Lets suppose you being a programmer keeping a track of users logging into your website or whatever, and the counter you have kept is of type int, the range of int is as you know 65,535 and your number of users exceeds the range of the type. in that case black box test might be unable to detect what's going on in between, but white box test will do.
For specific input, an error occurs internally resulting in:
Improper data placed in a global data area;
Improper flags that will be tested in a subsequent series of tests;
Improper hardware control that can only be uncovered during system test; yet "correct" output is produced.
Error detection by white-box testing contradicting black-box testing:
Testing to ensure that all independent paths within a module will be executed at least once.
Testing to exercise all logical decisions on their true and false branches.
Testing to ensure that all loops execute at their boundaries and within their operational bounds.
Error detection by black-box testing contradicting white-box testing:
Testing for interface functionality.
Testing system behavior and performance.
Test for classes of input.