I wrote all possible tests according to the requirements, made an analysis of the structural coverage (to make sure that I reached full coverage), added tests based on the results of the conducted analysis. However, there are still areas of code that cannot be covered by tests (hardware errors that are difficult to simulate or protected code designs). How do I verify these uncovered sections of code in this case?
The second question is, are there any authoritative sources that tell about such cases and how to check such code sections correctly?
The first thing that comes to mind is an inspection, with an analysis of the source code and the assertion that this code has a built-in logic.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 10 months ago.
Improve this question
I'm new to Cucumber, and I'm trying to understand the tool. While reading the documentation, I found that it is defined shortly as "a tool that supports BDD":
Cucumber is a tool that supports Behaviour-Driven Development(BDD).
Also it is described as a "validation tool":
Cucumber reads executable specifications written in plain text and validates that the software does what those specifications say.
In the other side, I noticed the excessive use of the word "test" on the 10-minute tutorial.
AFAIK, what does this tool is agile testing, since it is used massively in e2e testing (test basis = Gherkin feature specs + step definitions). However, the blog says something different:
Finally, remember that Cucumber is not a testing tool. It is a tool for capturing common understanding on how a system should work. A tool that allows you, but doesn't require you, to automate the verification of the behaviour of your system if you find it useful.
Now if this tool is not really about testing, then what use is it intended for?
TL;DR
Cucumber is a BDD framework. Its main components are:
Gherkin: a ubiquitous language used as communication tool than
anything, and can be used as a springboard for collaboration. It
helps manage the expectations of the business, and if everyone can
see the changes that you are making in a digestible format, they'll
hopefully get less frustrated with the development teams, but also
you can use it to speed up the reaction time of your team if there
are bugs, by writing the tests that cucumber wraps with the mindset
that someone is going to have to come back and debug it at some
point.
CLI implementation (or the CLI): a test runner based on
Gherkin. It is developed by volunteers who are all donating part of
their spare time. Every implementation is specific to a programming
language supporting a production code that is ready for test. It is
considered as the concrete tool / utility.
The Long Version
The intended use of Gherkin as a communication tool, describing interactions with a system from multiple perspectives (or actors), and it just so happens that you can integrate it with test frameworks, which aids in ensuring that the system in place correctly handles those interactions.
Most commonly, this is from a users perspective:
Given John has logged in
When he receives a message from Brenda
Then he should be able to click through to his message thread with Brenda via the notification
But it can also be from a component/pages perspective too:
Given the customer history table is displayed
And there have been changes to the customer table for that user since the page was first loaded
When it receives a click to refresh the data
Then the new changes should be displayed
It's all about describing the behaviours, and allowing the business and developers to collaborate freely, while breaking down the language barriers that usually end up plaguing communication, and generally making both sides frustrated at each other because of a lack of mutual understanding of an issue
This is where the "fun" begins - Anakin, Ep III
You could use these files to create an environment of "living documentation" throughout your development team (and if successful, the wider business), and in theory - worded and displayed correctly, it would be an incredible boon for customer service workers, who would more easily be able to keep up with changes, and would have extremely well described help documentation - without any real additional effort, but this isn't something that I've seen much in the wild. I've written a script at work that does this by converting the features into markdown, and alongside various other markdown tools (mermaid for graphs, tsdoc-plugin-markdown to generate the API docs, and various extensions for my chosen HTML converter, docsify) I've managed to generate something that isn't hard to navigate and open up communication between teams that previously found it harder to communicate their issues to the dev team (most people know a little markdown these days, even if it has to be described as "the characters you type in reddit threads and youtube comments to make text bold and italics, etc" for people to get what it is, but it means everyone can contribute to it)
It is an extremely useful tool when it comes to debugging tests, especially when used with the screenplay pattern (less so with the standard page object model, because of the lack of additional context that the pom provides, but it's still useful), as everything is described in a way that breeds replication of the issue from a users or components perspective if it fails.
I've paired it with flow charts, where I draw out the user interactions, pinning the features to it and being able to see in a more visual way where users will be able to do something that we might not have planned for, or even figure out some garish scenario that we somehow missed.
The Long Version Longer
My examples here will mostly in javascript, as we've been developing in a node environment, but if you wanted to create your own versions, it shouldn't be too different.
The Docs
Essentially, this is bit is just for displaying the feature files in a way that is easily digestible by the business (I have plans to integrate test reports into this too, and give the ability to switch branches and such)
First, you want to get a simple array of all of the files in your features folder, and pick out the ones with ".feature" on the end.
Essentially, you just need to flatten an ls here (this can be improved, but we have a requirement to use the LTS version of node, rather than the latest version in general)
const fs = require('fs');
const path = require('path');
const walkSync = (d) => fs.statSync(d).isDirectory() ? fs.readdirSync(d).map(f => walkSync(path.join(d, f))) : d;
const flatten = (arr, result = []) => {
if (!Array.isArray(arr)){
return [...result, arr];
}
arr.forEach((a) => {
result = flatten(a, result)
})
return result
}
function features (folder) {
const allFiles = flatten(walkSync(path.relative(process.cwd(), folder)))
let convertible = []
for (let file of allFiles) {
if (file.match(/.feature$/)) {
convertible.push(file)
}
}
return convertible
}
...
Going through all of those files with a Gherkin parser to pull out your scenarios requires some set up, although it's pretty simple to do, as Gherkin has an extremely well defined structure and known keywords.
There can be a lot of self referencing, as when you boil it down to the basics, a lot of cucumber is built on well defined components. For example, you could describe a scenario as a background that can have a description, tags and a name:
class Convert {
...
static background (background) {
return {
cuke: `${background.keyword.trim()}:`,
steps: this.steps(background.steps),
location: this.location(background.location),
keyword: background.keyword
}
}
static scenario (scenario) {
return {
...this.background(scenario),
tags: this.tags(scenario.tags),
cuke: `${scenario.keyword.trim()}: ${scenario.name}\n`,
description: `${scenario.description.replace(/(?!^\s+[>].*$)(^.*$)/gm, "$1<br>").trim()}`,
examples: this.examples(scenario.examples)
}
}
...
}
You can flesh it out fully to write to either a single file, or output a few markdown files (making sure to reference them in a menu file)
Flowcharts
Flow charts make it easier to help visualise an issue, and there are a few tools that use markdown to help generate them like this:
In the back, it'll end up looking like this:
### Login
Customers should be able to log into their account, as long as they have registered.
...
```mermaid
graph TD
navigateToLogin["Navigate to Login"] -->logIn{"Login"}
logIn -->validCredentials["Valid<br>Credentials"]
logIn -->invalidCredentials{"Invalid<br>Credentials"}
invalidCredentials -->blankPass["Blank Password"]
invalidCredentials -->wrongPass["Wrong Password"]
invalidCredentials -->blankEmail["Blank Email"]
invalidCredentials -->wrongEmail["Wrong Email"]
...
click blankPass "/#/areas/login/scenario-blank-password" "View Scenario"
...
```
It's essentially just a really quick way to visualise issues, and links us to the correct places in the documentation to find an answer. The tool draws out the flowchart, you just have to make the connections between key concepts or ideas on the page (i.e. a new customer gets a different start screen)
Screenplay Pattern, Serenity and Debugging
I think all that really needs to be said here is that when you run a test, this is your output:
✓ Correct information on the billing page
✓ Given Valerie has logged into her account
✓ Valerie attempts to log in
✓ Valerie visits the login page
✓ Valerie navigates to '/login'
✓ Valerie waits up to 5s until the email field does become visible
✓ Valerie enters 'thisisclearlyafakeemail#somemailprovider.com' into the email field
✓ Valerie enters 'P#ssword!231' into the password field
✓ Valerie clicks on the login button
✓ Valerie waits for 1s
It will break down any part of the test into descriptions, which means if the CSS changes, we won't be searching for something that no longer exists, and even someone new to debugging that area of the site will be able to pick up from a test failure.
Communication
I think all of that should show how communication can be improved in a more general sense. It's all about making sure that the projects are accessible to as many people who could input something valuable (which should be everyone in your business)
Can any one please tell me is there any kind of tools or eclipse base plugins available for generate relevant test cases for SalesForce platform related Apex classes. It seems with code coverage they are not expecting out come like we expect with JUnit, they want to cover whether, test cases are going through the flows of the source classes (like code go through).
Please don't get this post in wrong, I don't want anyone is going to write test cases for my codes :). I have post this question due to nature of SalesForce expecting that code coverage should be. Thanks.
Although Salesforce requires a certain percentage of code coverage for your test cases, you really need to be writing cases that check the results to ensure that the code behaves as designed.
So, even if there was a tool that could generate code to get 100% coverage of your test class, it wouldn't be able to test the results of those method calls, leaving you with a false sense of having "tested code".
I've found that breaking up long methods into separate, sometimes static, methods makes it easier to do unit testing. You can test each individual method, and not worry so much about tweaking parameters to a single method so that it covers all execution paths.
it's now possible to generate test classes automatically for your class/trigger/batch. You can install "Test Class Generator" app from AppExchange and see it working.
This would really help you generating test class and saves lot of your development time.
A command line utility/software could potentially consist of many different switches and arguments.
Lets say your software is called CLI and lets say CLI has the following features:
The general syntax of CLI is:
CLI <data structures> <operation> <required arguments> [optional arguments]
<data structures> could be 'matrix', 'complex numbers', 'int', 'floating point', 'log'
<operation> could be 'add', 'subtract', 'multiply', 'divide'
I cant think of any required and optional arguments, but lets say your software does support it
Now you want to test this software. And you wish to test interface itself, not the logic. Essentially the interface must return the correct success codes and error codes.
Essentially a lot of real word software still present a Command Line interface with several options. I am curious if there is any formal testing methodology established for this. One idea i had was to construct a grammar (like EBNF) and describing the 'language' of the interface. But I fail to push this idea ahead. What good is a grammar for in this case? How does it enable the generation of many many combinations .
I am curious to learn more about any theoretical models which could be applied to such a problem or if anyone in here has actually done such testing with satisfying coverage
There is a command-line tool as part of a product i maintain, and i have a situation thats very similar to what you describe. What i did was employ a unit testing framework, and encode each combination of arguments as a test method.
The program is implemented in c#/.NET, so i use microsoft's testing framework that's builtin to Visual Studio, but the approach would work with any unit testing framework.
Each test invokes a utility function that starts the process and sends in the input and cole ts the output. Then, each test is responsible for verifying that the output from the CLI matches what was expected. In some cases, there's a family of test cases that can be performed by a single test method, wih a for loop in it. The logic needs to run the CLI and check the output for each iteration.
The set of tests i have does not cover every permutation of arguments, but it covers the 80% cases and i can add new tests if there are ever any defects.
Using a recursive grammar to generate switches is an interesting idea. If you where to try this then you would need to first write the grammar in such a way that all switches could be used, and then do a random walk of the grammar.
This provides an easy method of randomly walking a grammar and outputting the result.
My team is working on educating some of our developers about testing. They understand why to write tests and are on board that they should write tests, but are falling a little short on writing good tests.
I just saw a commit like this
public void SomeTest{
#Test
public void testSomething{
System.out.println(new mySomething.getData());
}
So they were at least making sure their code gave them the expected output by looking.
It will be a bit before we can really sell the idea of code reviews. In the mean time I was considering having JUnit fail any tests that do not have actual assertXXX or fail statements in them. I would then like to have that failure message say something like "Your tests should use assertions and actually examine the output!".
I fully expect this to lead to calls like assertTrue(1 == 1);. We're working on the team buy in for proper testing and code reviews, are there any technical mechanisms we can use to make life easier for the developers that already get it?? What about technical mechanisms to help the new guys understand?
I think you should consider organizational changes: mentoring, training, code reviews.
The tools can only help you if you're using them in good faith with a base understanding of the goals. If one of these is missing they won't help you.
Humans are just to intelligent to do dump things or work around metrics. I think your assessment is not correct that "they" are on board if they can't write a single useful test. Automatic tools are simply not the correct tools at this stage. You can't learn by being told by a program what to do next.
You can use some static code analyzer.
I use PMD which includes a JUnit rule set. There are a lot of IDE plugins which will mark rule violations in the IDE. You can configure the rule sets to your needs.
You will also profit from the other rule sets - which will warn you on code style / best practice violations (although you have to decide sometimes if the tool or you are the fool :-)).
to answer the stated question for future viewers.
JUnit uses reflection to run tested function if any Exception, Error throws -> test fails, otherwise succeed. Assert class is just a utils class.