I am trying to get bytecode coverage analysis using a code coverage tool (like Emma or Jacoco) after testing with a GUI based functional testing tool (like HP QuickTest Pro or Selenium).
Anyone who has done this could please give me an idea to start this project?
I am doing this now. My approach is to use JaCoCo ant tasks to instrument the binary byte-code files, and use a specific CLASSPATH to execute the instrumented binaries from an ant build.xml from Jenkins.
The reason for doing the code coverage from byte-code is that there is an existing set-up that runs test scripts for a large application using HP QuickTest Pro . I would imagine that the test coverage is in the single digits, but we need an empirical baseline to demonstrate the possible improvements in code coverage from doing unit tests during a build.
Related
I have a scenario like this - Application written in hybrid languages (Python, C++ and Java majorly). There are around 100 test cases written in Robot framework to test the application. Now I want to see code coverage of my application. Is there any tool that can work in such scenario?
Thanks in advance.
The same way you would do outside robotframework: Using external code analysis tools. Let me elaborate.
In Python, you can use tools like Coverage (https://pypi.org/project/coverage) to run your testing suite while gathering coverage data. For example, if you usually run your robot test suites using:
robot suites
(supposing you have a "suites" directory with your .robot files) then you would run robot as a Python module over Coverage like this:
coverage run -m robot suites
And you could get your report with:
coverage report
You'll probably need to filter the report (--include option) if you're only interested in code inside a directory. For example, for a directory "myproy" you would do:
coverage report --include *myproy/*
You could use a similar strategy with other Python test coverage tools as long as you figure out how to tweak their execution command to run robot as a module. The same holds for C++ and Java code analysis tools; For example, check the following link for a guide that uses Java with JaCoCo and Maven: https://www.cnblogs.com/z1500592/p/6676646.html
Is there an open source tool available to control the running of BDD cucumber tests?
We are developing BDD cucumber tests and would like the option to control the tests when running them (start/stop/pause/restart) using an open source (or proprietary) test tool.
The short answer it, yes.
The somewhat longer answer is that it depends on your echo system.
If you are using Java, then any build tool will be sufficient. That is Maven, Gradle or similar. These are easy to integrate in your Continuous Integration, CI, environment. With a tool chain like that, you are able to execute Cucumber on every build and will always know if your system works or not.
Yes , but in small scope (Automation tests) and less process control related to run and control tests ,In high scope with multiple branches and projects i think you have to move to Jenkins with full control.
Following link describe the coparsion : https://www.saashub.com/compare-jenkins-vs-cucumber
I have just started learning about test automation in Selenium and found out that most online tutorials would tell you to run the test suite inside an IDE together with a test framework such as TestNG (with testng.xml) and a build tool such as Maven.
When you are working in a software company and told to build a test framework and run automated tests, I don't believe you actually need to fire up your IDE every time you want to execute your test suite. So, my question is, what is the typical setup a software company follows to 'automate' running your test automation scripts?
Software companies are following agile practices and wanna keep up with industry practices. In real projects, CI & CD are used to continuously integrate, deploy and test the software.
Tests are written by SDET using test automation frameworks. While developing test scripts test developers use IDEs like eclipse. However, tests are executed over Jenkins as a job, after required frequency/event.
For example, after every code deployment, Jenkins can automatically trigger your sanity suite, and run regression bi-weekly.
The process' are automated now-a-days with stakeholders demanding agility.
One can invoke selenium java project from command line via .bat file in Jenkins, or using ant/maven as build tools.
IDEs are seldom used to run tests in real world.
We have seperate integration test project which fires the integration test cases on different modules . At present we do not have unit test cases within each module. We would like to ensure that the integration test covers most of the domain functionality.
Since we have the integration tests in a different project , Sonar always reports the test coverage as zero for the modules under test.
Is there any way to have the test coverage reported on a project , when the actual test is run from a different project
Thanks
You should be able to achieve what you want by reading the Code Coverage documentation page on the wiki. Most notably, you'll be able to use the following sample project to see how it works:
IT JaCoCo Sonar Runner sample project
Basically, you have to run your integration tests first using JaCoCo to generate the coverage report (jacoco.exec) and then you reuse this report during the SonarQube analysis.
Is it possible to run unit tests when a non-maven project is analyzed with Sonar, in Sonar light mode?
Sonar doesn't run unit tests. But it should be able to analyze existing unit tests reports. From Reuse in Sonar unit test reports generated by other systems:
2. Using Sonar in its full capability in an ANT environment
If you are using ANT to build your
applications, the main weakness so far
in Sonar was that it did not allow to
display Unit tests results nor Code
coverage. I am sure that now you have
read the first use case, you know that
by using the
“-Dsonar.dynamicAnalysis=reuseReports”
parameter, this limitation does not
exist anymore. You simply need to
specify where those reports to reuse
are going to be found, by using the
following properties :
sonar.cobertura.reportPath,
sonar.clover.reportPath,
sonar.surefire.reportsPath...