I would like to ask a question regarding Blackbox testing
I cite this site
Black Box Testing is a software testing method in which the
functionalities of software applications are tested without having
knowledge of internal code structure, implementation details and
internal paths. Black Box Testing mainly focuses on input and output
of software applications and it is entirely based on software
requirements and specifications. It is also known as Behavioral
Testing.
Alright, I understand the concept. But my situation has the following characteristics:
The application to test has harcoded in the code the input file it uses.
The output of the application are some plots and graphs
I have to test this application so my questions are:
It is ok for me to modify that one line of the code so as the input points to my test datafiles? (Because there is no other way to modify the input -other than creating a data file and naming it with the fake filename it uses)
Is blackbox testing manual or automatized? I don't see how can checking the output plots can be automatized.
Blackbox testing can be considere system testing? Does it use testing frameworks?
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
Guys I am looking for Web Testing Generic Automation Framework which can be used to do automation testing of various web based applications .Looking for C# based framework as that is the language I am more familiar with. But any other language framework will also do and it should not use any proprietary/licensed language.
Framework should have some open source and free of cost license model.
I searched for selenium based framework on Google and SO. But could not come with any which have source code available. It will be good if the framework encapsulates all the functionality provided by Selenium WebDriver and/or Selenium RC and empower the functional tester to create and maintain test in human readable scripts.
Requirements of the framework:
The framework code should avoid hard coding of test steps. My idea is to maintain the test scripts outside the automation framework code , so that they can be easily be modified if needed. The framework should read through the step tables and the data tables and run the test accordingly.
If there is no such framework available now right then we can collectively build such a framework in a open source community model.
P.S.
I have read a little about Hermes Framework and Robot Framework, but not yet tried them, any help is welcome.
The good side of this problem: there are a lot of flexible tools and approaches, you can get together and build a flexible, reliable and robust test automation framework.
The hard part is: yes, there is no “out of box” solution, and you’ll need to find and put together lots of tools in order to solve this test automation puzzle.
What I would recommend:
First you need to choose a unit-test test framework. This is a tool which helps to identify separate methods in code as tests, so you can run them together or separately and get the run results, such as pass or fail.
My personal opinion, is that the testing tool – MS-Test – which ships with Visual Studio 2013 (and also Express Edition) is good enough. Another alternatives are: NUnit or Gallio Icarus
All unit-testing frameworks includes a mechanism for doing assertions inside the test. The capability of assertions class depends on given unit-testing framework. Here, I would like to recommend a popular library which works great for the entire unit testing framework.
This is Fluent Assertions (also available from NuGet repository).
That’s a hard moment. You need to decide: are you going to use the PageObject approach in order to build your test automation framework, or you are going to choose simpler approach, without heavy utilization of the Object Oriented Programming.
Properly designed Page Objects makes your test automation code much maintainable. Utilizing the OOP – you can do a magic in your code: write less to do more. Although, such approach requires more skill.
Here are a good articles on this topic:
Maintainable Automated UI Tests
And this one:
Tips to Avoid Brittle UI Tests
The alternative to the PageObject is a scripted approach. This approach can be also successful and requires less time to start.
Coypu is a good and usable example of such framework for Selenium Web Driver.
All the popular unit-testing frameworks support data-driven tests. The best support is in NUnit – you can run/re-run and see the tests generated for individual data row in the tests tree.
MS-Test supports reading data from different data-sources: text files, excel, mssql etc., but it is not possible to re-run the test for individual data row. Although, there is a hack for this – Ms-Test Rows.
For my data-driven tests, I am using a great library – Linq to Excel
I have a lot more to say. There are so many approaches to build test automation framework – and there is no ready solution yet.
I am trying to build one according to my testing methodology – SWD.Starter .
This project is still on its early development stages. But, at least, probably you’ll find a few tips how to build and organize the test automation code.
I've implemented https://github.com/leblancmeneses/RobustHaven.IntegrationTests based on my prior experience on large projects "trying" to implement full end to end testing.
I've been using this and and have a lot of useful extensions for general selenium, angularjs, and kendo ui work. Since this framework is not obtrusive you could just use these extensions without using anything else.
I'm using this on my latest project and everyone is loving it.
There are a lot of bdd/spec frameworks (specflow, mspec, nspec, storyq) to help wire the behavior of your system to tests.
What I've learned:
make it frictionless for any .net developer/tester to begin writing/running tests.
Most fail here because it requires installing additional pluggins into visual studio.
mine uses the standard nunit
Logically you would think that a feature is a class file and scenarios are [Test] methods - to support some of these frameworks they make each scenario a class file.
use the original spec to create stubs of your tests - hopefully readable code
I used spec flow back in 2010 - so things might have changed. I generated my tests from my bdd document. A year later when I went to add more tests and update existing tests, I felt I wasted a lot of time with ceremony than writing code I really wanted - I stopped using it.
My approach uses t4 to generate stubs - developer has a choice to generate from feature file, for a specific scenario or don't use generated code at all.
how is state shared across steps / nested steps
most use dictionary<string,object> to help you separate data from being hardcoded in your tests accessed from a context object.
mine uses viewmodels and pointers to those viewmodels - if your using something like angularjs you are using viewmodels in your server side display/editor templates and in angularjs controller so why not reuse these in your tests!
start early with CI - make development transparent
My project has ResultDiff that given the nunit testresult.xml file, folder location to your gherkin feature files, and output json file; Read description on why this is important on the screenshot: https://github.com/leblancmeneses/RobustHaven.IntegrationTests#step-5-ci-setup-resultdiff
Example:
Modified means business and developers have a mismatch of Gherkin statements - did something change that we need to talk about?
What is missing? a dashboard to render the .json file created by ResultDiff. It's on my backlog.....
With a centralized dashboard that supports multiple environments(branches of your code) this dashboard will serve all stakeholders (business, developers) what is the status of features being developed.
There is a framework named "omelet" which is built in java on top of testng for selenium,
For cross browser multi-parallel testing , it easily blends with your CI tools and have some cool reporting features with step level reports
Running your test cases on BrowserStack and Grid was never so easy as with omelet with few config changes.
if you want to give it a try then do follow the 5 min tutorial available on the website, there is archetype available on maven central + there are many more features available
Stable version is 1.0.4 and we are currently looking for people to contribute to project.
Documentation over here
Github link
Can you suggest on this points related to Autosar, taking into consideration I am a software developer who can write some software in C?
Now I Develop a functionality in C, that has to read some ECU specific data, process it & update some ECU specific data (which can be some variable or i/o signal).
Now how I will be using Autosar RTE & virtual functional bus?
What will be there use to a software developer?
Also, as Autosar says "standardization of interfaces" what does it mean? Does it mean that if some else anywhere around the world is also developing same functionality (in C language, like me) we both will be using same name of the API's for those I/O signals?
How RTE will be helpful for me in Unit testing? Or what really RTE is doing from software developer point of view?
http://www.autosar.org/gfx/AUTOSAR_TechnicalOverview_b.jpg
I read a lot technical terms... but being a software developer these points are important for me to know. Can you explain it a bit to me.
Your reply will be appreciated.
I don't think it is going to be that easy...
I believe that you are developing Autosar SWC (software component).
I would recommend for you to develop a portable C module. That has very clear inputs, outputs and req. on execution (check Autosar runnables).
Remember Autosar ECU includes RTOS, therefore your module will be part of a OS task.
When and if you come to the point of building an Autosar ECU, you will be able to wrap the module and connect ins/outs with Autosar virtual functional bus signals. For that you will need Autosar framework and probably configuration tools. These are complex and expensive.
Unit test the module the usual way you test C module.
Good luck.
P.S. RTE is just the "glue" code generated automatically by configuration tools according the configuration of ECU BSW and System Extract for that ECU. You will worry about it during wrapping.
The Idea behind dividing the functionality in AUTOSAR SWC and Basic software is to make the application SW development independent of any platform. To answer your questions.
RTE is giving the application a signal based interface, hence you expect the other SW components (inter-ECU /intra-ECU) to provide the required data in the form of signals, you dont care about the platform or type of communication medium
Yes by standardizing the interfaces (all kind of interactions), a software component or any Basic software module can be Fixed into the SW architecture. Read more about the different type of AUTOSAR interfaces.
Refer to answer 1
RTE is there as a layer to 'abstract' the inner components of the system. For example, if you need to get access to the system's installed flash memory, you have to use the RTE-related memory functions.
You are correct. You only need to read the specifications and use the corresponding functions to get your desired result in an AUTOSAR system.
RTE makes sure that the developers of the software components and the middle-layer systems would work properly with minimal interaction between them. SWC developers just need to read the AUTOSAR standard and follow it to ensure compatibility with the middle-layer systems, since it is expected that the middle-layer system developers would follow that same standard in providing functionalities on their side. It also helps developers with the portability of their software.
I think all your questions can be answered by reading the AUTOSAR standard documents at the AUTOSAR website. Most of my limited knowledge in development of AUTOSAR systems (started reading about it for close to a month already), I got there.
I am a Software developer who Developed a Console Application Tool for Autosar RTE, Test Case Generation for RTE, and wrote Unit Testing Scripts for the tool I created.
I Developed these using C# and NUnit Framework. Same can be Developed using C or a java or any other language. Ultimate goal is to generate AUTOSAR modules (.c and .h files) based on the requirement.
1. Software Developer Scope
As a Software Developer, i had a task to implement complete RTE and Test Applications for the Implemented RTE code.
Inputs and Outputs:
Basically our inputs were Software Component files and ECU Extract which were in ARXML format and Outputs were Rte and test application source and header files (.c and .h) which were created based on the requirements.
Tasks as a developer:
Here, as a developer, we need to perform Input parsing from AXXML to our own data structure, Schema Validation, Modal Validation, File generation etc.
2. Standardization
Yes, AUTOSAR Architecture provides standardized interface. Irrespective of the implementation strategy, API structure remains same which eases the usage. This acts as a generalised library where you can use already developed Module or you can implement the module in your own way by considering API specification. All you need is to follow the specifications provided for every module you use.
Requirement varies from Company to Company but the way of using APIs remains same.
3. Unit Testing
Unit Testing has nothing to do with RTE or AUTOSAR modules. You will be testing the Uints of Your Code. When i say your code, it is the one which you used to develop any particular module (eg. Rte.c) and not testing the generated module itself. You will be testing the Source code you developed to generate the specfic module. Your source code is not part of RTE or any other module implementation but is tool which generates the module implementation.
Overview:
Software developer have various scope in generating AUTOSAR modules depends on the Requirement.
You can develop a tool which will generate AUTOSAR modules.
You can develop an Editor which will is used to edit/create AUTOSAR XML files. (Eg: Artop)
Developing might sound complex as we do not get direct resources other than specifications. Once you are in, you will learn a lot.
To answer your question
If you will go through the Layered Architecture of AUTOSAR, you will come to know this architecture is followed to minimize the dependency of the each module
(layer) with lower layer.
Again, RTE is a like wrapper to separate the lower layered dependency, this enables to work on each layer independently. Most of the virtual buses are mapped with RTE, in my experience I have worked on IOC which is allowed to map with RTE and which communicates with other SWC's with memory and core boudary. To OS Developer its via to the application layer and Mapped software partitions.
The standard is used to maintain uniformation in all software layers, however to meet the requirements the developers may have different way of implementation and design, but the API's and requirements will be universal.
This is useful for standardised intefacing too.
For Unit testing of the developers OS design and implementation RTE works as abstract module.
Reading Specs for different module will resolve most of doubts.
How do you experienced developers comment these lines by Michael Feathers:
A test is not a unit test if:
It talks to the database
It communicates across the network
It touches the file system
It can't run at the same time as any of your other unit tests
You have to do special things to your environment (such as editing
config. files) to run it.
Now I was wondering if I should unit test my DAO classes...
Will I get more advantages or disadvantages by unit testing the DAO layer? Share your thoughts please.
Feathers isn't saying don't write such tests. He's saying they're not unit tests, because he defines a unit test as "small, they test a method or the interaction of a couple of methods. ... a "binary chop" that allows you to discover whether the problem is in your logic or in the things are you interfacing with." And he's right - the Agile/XP/Scrum intention of unit testing is to provide a fast red light/green light determination if a small piece of code is functioning correctly.
I need to automate testing of my windows mobile application. My application does not have any UI. So, normal testing tools which works with random key strokes and mouse clicks will not work here. Are there any tools available for windows mobile to test only background processing?
You have a couple of options depending on what level of testing you want.
Integrated Test
An integrated test aims to test the application in the real world. Therefore you would create all of the "real" things and write code to specifically test to see if your conditions are met. I would believe however, if you're trying to test GPS then this would not be practical. As someone would actually have to move the device around.
Unit Test by mocking
I've done this before for GPS testing. The idea is that you SANDBOX the object being tested. You ensure that all external references (e.g. anything that isn't the object) are interfaces. You then MOCK these interfaces with "test-only" implementations.
For example I worked on a GPS test where we used an interface called: INmeaInterpreter to fire certain events which would be picked up by a class named PositioningService.
The default implementation was a 3rd party component.
However as INmeaInterpreter was an interface we could create an implementation that instead of using the REAL data uses (for example) an NMEA file to read from. This enabled us to test how the PositioningService worked in certain (and sometimes strange) scenarios.
I would then suggest mocking the other external references. The call to the database can just be a dummy object with a counter for the database call that is incremented if it is called. You could then write a test with an NMEA file that should result in a database call and then at the end of the unit test check that dummy object to see if that call occurred.
We did all the above with horrible MSTest but you could use any testing framework (I recommend NUnit). I'm not sure if there are options to specifically test on the device. We ran all of our tests on desktop as we'd split the code nicely so that device specific code was isolated and could easily be replaced with Desktop equivalents.
Obviously the only problem with unit tests is that they dont test the hardware.
I would recommend (depending on the size of the project and the team) to do BOTH types of testing but to place the larger emphasis on the unit tests (as they are easier to run/manage).
Am looking for a regression test framework where I can add tests to.. Tests could be any sort of binaries that poke an application..
This really depends on what you're trying to do, but one of the features of the new Test::Harness (disclaimer: I'm the original author and still a core developer) is that if your tests output TAP (the Test Anything Protocol), you can use Test::Harness to run test suites written in multiple languages. As a result, you don't have to worry about getting "locked in" to a particular language because that's all your testing software supports. In one of my talks on the subject, I even give an example of a test suite written in Perl, C, Ruby, and HTML (yes, HTML -- you'd have to see it).
Just thought I would tell you guys what I ended up using..
QMtest ::=> http://mentorembedded.github.io/qmtest/
I found QMTest to full fill my needs. Its extensible framework, allows you to write very flexible test classes. Then, these test classes could be instantiated to large test suites to do regression testing.
QMTest is also very forward thinking, it allows for weak test dependencies and the creation of test resources. After a while of using QMTest, I started writing better quality tests. However, like any other piece of complex software, it requires some time to learn and understand the concepts, the API is documented and the User Manual give a good introduction. With sometime in your hand, I think QMTest is well worth it.
You did not indicate what language you are working in, but the xUnit family is available for a lot of different languages.
/Allan
It also depends heavily what kind of application you're working on. For a commandline app, for example, its probably easy enough to just create a shell script that calls it with a whole bunch of different options and compares its result to a previously known stable version, warning you if any of the output differs so that you can check whether the change is intentional or not.
If you want something more fancy, of course, you'll probably want some sort of dedicated testing framework.
I assume you are regression-testing a web application?
There are some tools in this kb article from Microsoft
And if I remember correctly, certain editions of Visual Studio also offer its own flavor of regression testing tools as well.
But if you just want a unit testing framework, the xUnit family does it pretty well.
Here's JUnit and NUnit.