I'm working on a Flutter app using Android Studio as my IDE. I'm attempting to write tests and check the code coverage but I can't work out how to view the data in the IDE or any other application.
By running flutter test --coverage, a coverage report seems to be generated into a file /coverage/lcov.info. That file looks something like this:
SF:lib\data\Customer.g.dart
DA:9,2
DA:10,2
DA:11,2
DA:12,2
DA:13,2
DA:20,0
DA:21,0
DA:22,0
DA:23,0
DA:24,0
...
Looking at the file it seems to have a list of my project files with line by line coverage data. Is there a way to view this information in Android Studio?
You can also install lcov and convert the lcov.info file to HTML pages and then see the result in the browser with sorting option.
1. Installation
1.1. Installing in Ubuntu
sudo apt-get update -qq -y
sudo apt-get install lcov -y
1.2. Installing in Mac
brew install lcov
2. Run tests, generate coverage files and convert to HTML
flutter test --coverage
genhtml coverage/lcov.info -o coverage/html
3. Open coverage report in browser
open coverage/html/index.html
Note This way you can add it to circleci artifacts and coveralls as well.
This is what you want to run to see tests coverage in your browser on macOS
flutter test --coverage
genhtml coverage/lcov.info -o coverage/html
open coverage/html/index.html
You can view the code coverage generated by flutter with the Atom editor.
You just need to install the Dart and lcov-info packages.
Then you load your project folder and press Ctrl+Alt+c, coverage will be displayed with a summary of the whole projects coverage and also with specific line highlighting.
There doesn't appear to be any plugin for Android studio which does this as of yet.
Update 9/18/2021:
See new answer for how it's done within the editor
Update 5/9/2020:
Turns out you can just run flutter test --coverage, then in the same terminal session run bash <(curl -s https://codecov.io/bash) -t token token should be the repository token you get from CodeCov. That command should automatically find and upload the coverage data and will be visible on your CodeCov dashboard. So you don't need Bitrise.
Original:
I've been using Bitrise for continuous integration on my flutter project and there is an easy way to send your reports to CodeCov then visualize it there. This requires you to gain some knowledge on how to set up and use Bitrise but a lot of its automatic so don't worry, also if you're a small team you should be fine with the free tier. Here are the key points for getting CodeCov to work.
Make sure you add the --coverage variable to the Flutter Test workflow.
Add the token from CodeCov as a secret key, you will need to sign up for CodeCov and link your repository to receive a token.
Add the CodeCov workflow and select the CODECOV_TOKEN key.
After that, you should be able to fire off a build and if successful you should see your dashboard update at CodeCov.
The Flutter Enhancement Suite does exactly that. It is an Android Studio/IntelliJ plugin which generates coverage reports.
It shows the coverage per file and also highlights covered lines (red/green bars next to the line numbers):
install the plugin from the plugin options (Preferences > Plugins > Marketplace tab > Search for Flutter Enhancement Suite).
Create a new Run Configuration for testing with coverage
(Run > Edit Configurations > click the plus button to add a new configuration > Choose Flutter Test in the dropdown)
Name your configuration (e.g. "All tests"), set the scope and the file or directory containing your tests.
Run your tests with coverage from the top menu.
I just developed a simple dart package (test_cov_console), so you can run it directly from Android Studio terminal. The tool would read the lcov.info that was generated by flutter test --coverage. Find this link for source code.
You can install the lib globally, so it would not change your current project:
flutter pub global activate test_cov_console
And run it:
flutter pub global run test_cov_console
Here is the sample of output:
flutter pub run test_cov_console
---------------------------------------------|---------|---------|---------|-------------------|
File |% Branch | % Funcs | % Lines | Uncovered Line #s |
---------------------------------------------|---------|---------|---------|-------------------|
lib/src/ | | | | |
print_cov.dart | 100.00 | 100.00 | 88.37 |...,149,205,206,207|
print_cov_constants.dart | 0.00 | 0.00 | 0.00 | no unit testing|
lib/ | | | | |
test_cov_console.dart | 0.00 | 0.00 | 0.00 | no unit testing|
---------------------------------------------|---------|---------|---------|-------------------|
All files with unit testing | 100.00 | 100.00 | 88.37 | |
---------------------------------------------|---------|---------|---------|-------------------|
The output can be saved to CSV file:
flutter pub run test_cov_console -c --output=coverage/test_coverage.csv
Sample CSV output file:
File,% Branch,% Funcs,% Lines,Uncovered Line #s
lib/,,,,
test_cov_console.dart,0.00,0.00,0.00,no unit testing
lib/src/,,,,
parser.dart,100.00,100.00,97.22,"97"
parser_constants.dart,100.00,100.00,100.00,""
print_cov.dart,100.00,100.00,82.91,"29,49,51,52,171,174,177,180,183,184,185,186,187,188,279,324,325,387,388,389,390,391,392,393,394,395,398"
print_cov_constants.dart,0.00,0.00,0.00,no unit testing
All files with unit testing,100.00,100.00,86.07,""
With the release of Flutter 2.5, you can now view test coverage within IntelliJ and Android Studio.
See this post
In addition, the latest IJ/AS plugin for Flutter allows you to see the
coverage information for both unit test and integration test runs. You
can access this from the toolbar button next to the “Debug” button:
Android Studio and IntelliJ:
Coverage reporting is now available on Android Studio
You can use SonarQube with an additional plugin for Flutter as per this link SonarQube plugin for Flutter / Dart.
I have tried it with the free version of SonarQube on docker, and if you have configured it correctly, you just need to run the following commands on Android Studio terminal:
# Download dependencies
flutter pub get
# Run tests with User feedback (in case some test are failing)
flutter test
# Run tests without user feedback regeneration tests.output and coverage/lcov.info
flutter test --machine --coverage > tests.output
# Run the analysis and publish to the SonarQube server
sonar-scanner
Here is the sample of the report, and you can drill deep into line codes.
So, the actual answer is no, you cannot view a coverage report within Android Studio (or in IntelliJ IDEA) at this time.
Unlike JavaScript/TypeScript and Java and probably Python, the IntelliJ IDE (and by extension, Android Studio) do not have integrated IDE support for showing test coverage of Flutter code in the editor. This is a shame because the ability to see your untested code branches and lines highlighted in the source code of your editor is a beautiful thing. Not sure why a plugin for this does not exist yet, since it is well-supported for other languages, and a standard lcov.info file is generated.
There is a bundled code coverage tool window in IntelliJ that is supposed to allow you to browse the lcov.info file in a tree/table drill-down format, but it doesn't seem to work with the coverage report generated by flutter (flutter test --coverage). I thought it might be the relative paths in the lcov.info and my multi-module app structure, but I tried to manually edit the file paths in lcov.info, but I had no luck getting the stats to show.
FOR WINDOWS
Required:
Chocolatey
Perl
LCOV
1. INSTALL CHOCOLATEY
Open PowerShell (with Admin)
Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
2. INSTALL PERL
choco install strawberryperl
Add path to the environmental variable
3. INSTALL LCOV
choco install lcov
LCOV OPERATION
go to the android studio terminal & run this flutter test --coverage
now next, open your project root dir in the power shell in my case (eg :C:\Users\CIPL\Documents\Project\Flutter\myProject)
& run this cmd perl C:\ProgramData\chocolatey\lib\lcov\tools\bin\genhtml coverage/lcov.info -o coverage/html
that's it open the html folder and click the HTML file to view the visual in the browser.
NOTE: when I tried to do 3'rd point, I faced this error "ERROR: cannot create directory 'coverage/html'"
so manually created the html folder and tried 3rd point again.
Found this Windows solution in this https://blog.tech-andgar.me/flutter-test-coverage
I am currently working on configuring Jenkins to run selenium tests.
I am using Seleniumhq plugin and the Selenium HTML runner in the Selenium RC slot in the Jenkins configuration*.
In the particular job, I have a shell script running export PATH=$PATH:/path/to/geckodriver.exe to add it to the path followed by the SeleniumHQ htmlSuite Run step.
When it runs with this configuration, the job fails with the following errors message:
Started by timer
Building in workspace /var/lib/jenkins/jobs/MYJOB-selenium-ide-test/workspace
[workspace] $ /bin/sh -xe /tmp/hudson2672803749243149546.sh
+ export PATH=/var/lib/jenkins/tools/hudson.model.JDK/Oracle_8u45/bin:/var/lib/jenkins/tools/hudson.model.JDK/Oracle_8u45/bin:/var/lib/jenkins/.rvm/gems/ruby-2.2.2/bin:/var/lib/jenkins/.rvm/gems/ruby-2.2.2#global/bin:/var/lib/jenkins/.rvm/rubies/ruby-2.2.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/var/lib/jenkins/.rvm/bin:/path/to/geckodriver.exe
java -jar /var/lib/jenkins/users/me/selenium-html-runner-3.0.1.jar -htmlSuite *firefox example.com /var/lib/jenkins/jobs/MYJOB-selenium-ide-test/workspace/suite1/JenkinsTestSuite /var/lib/jenkins/jobs/MYJOB-selenium-ide-test/workspace/result.html
[workspace] $ java -jar /var/lib/jenkins/users/ME/selenium-html-runner-3.0.1.jar -htmlSuite *firefox example.com /var/lib/jenkins/jobs/myjob-selenium-ide-test/workspace/suite1/JenkinsTestSuite /var/lib/jenkins/jobs/MYJOB-selenium-ide-test/workspace/result.html
Multi-window mode is longer used as an option and will be ignored.
Dec 14, 2016 6:31:00 PM org.openqa.selenium.server.htmlrunner.HTMLLauncher mainInt
WARNING: Test of browser failed: *firefox
java.lang.IllegalStateException: The path to the driver executable must be set by the webdriver.gecko.driver system property; for more information, see https://github.com/mozilla/geckodriver. The latest version can be downloaded from https://github.com/mozilla/geckodriver/releases
at com.google.common.base.Preconditions.checkState(Preconditions.java:199)
at org.openqa.selenium.remote.service.DriverService.findExecutable(DriverService.java:109)
at org.openqa.selenium.firefox.GeckoDriverService.access$000(GeckoDriverService.java:37)
at org.openqa.selenium.firefox.GeckoDriverService$Builder.findDefaultExecutable(GeckoDriverService.java:95)
at org.openqa.selenium.remote.service.DriverService$Builder.build(DriverService.java:296)
at org.openqa.selenium.firefox.FirefoxDriver.createCommandExecutor(FirefoxDriver.java:277)
at org.openqa.selenium.firefox.FirefoxDriver.<init>(FirefoxDriver.java:247)
at org.openqa.selenium.firefox.FirefoxDriver.<init>(FirefoxDriver.java:242)
at org.openqa.selenium.firefox.FirefoxDriver.<init>(FirefoxDriver.java:135)
at org.openqa.selenium.server.htmlrunner.HTMLLauncher.createDriver(HTMLLauncher.java:285)
at org.openqa.selenium.server.htmlrunner.HTMLLauncher.runHTMLSuite(HTMLLauncher.java:107)
at org.openqa.selenium.server.htmlrunner.HTMLLauncher.mainInt(HTMLLauncher.java:245)
at org.openqa.selenium.server.htmlrunner.HTMLLauncher.main(HTMLLauncher.java:273)
Publishing Selenium report...
ERROR: Test reports were found but none of them are new. Did tests run?
For example, /var/lib/jenkins/jobs/myjob-selenium-ide-test/workspace/suite1/results.html is 7 days 1 hr old
Build step 'Publish Selenium Report' changed build result to FAILURE
Finished: FAILURE
I can see that the issue is that the line System.setProperty("webdriver.gecko.driver","path/to/geckodriver.exe"); needs to be run. How do I properly configure this to happen? (I have tried to have it run as a Groovy build step, but this failed).
*(I have also attempted replacing it with an older version of the standalone server)
For Test Automation of web project we use Hudson, PHPUnit, and Selenium. The results of the build are stored in the JUnit XML format.
When I try to include generation of reports using the Hudson Publish JUnit test result report option, the build finishes with Failed status.
Below is my Hudson configuration to run the tests.
sudo -u apache phpunit - log-junit /var/lib/hudson/jobs/Work-stars-Tests/builds/${BUILD_ID}/ seleniumReports/seleniumTests.xml + path to test php files
Generation of reports is enabled through the Hudson configuration option «Publish JUnit test result report», where I specify the path to the folder with PHP tests.
The user we use to run Hudson has permission to write/read files in the folder with the reports. As for the path we've tried to specify both full and relative.
The error No test report files were found. Configuration error? is displayed in the console after the build.
How do we resolve this issue?
Found my own solution to this problem :)
In Hudson project configuration settings modified execute shell command to
> #!/bin/sh -x phpunit --log-junit ${WORKSPACE}/zf/tests/_tmp/reports/seleniumTests.xml
> ${WORKSPACE}/zf/tests/selenium/; sed
> -i '/<testsuite name=".*\/"/D;/^ <\/testsuite>$/D'
> ${WORKSPACE}/zf/tests/_tmp/reports/seleniumTests.xml
and set following path to Junit reports
**/zf/tests/_tmp/reports/*.xml
Problem is solved. Yahoo!
I made then saved a test case with the Firefox extension "Selenium IDE".
Now I want to use command line to run this exported html file.
I try to follow this how-do-i-launch-the-selenium-ide-from-the-command-line-with-a-specific-test-case but it doesn't work.
Please help me.
You will need the Selenium RC which you can get from:
http://seleniumhq.org/download/
And Java 1.5 or higher (Download Java here)
1) Install Java
2) Unpack Selenium RC.
3) Open a cmd.exe window and go to the directory containing the Selenium Server (selenium-remote-control-1.0.1\selenium-server-1.0.1)
4) Run the command below:
java -jar selenium-server.jar -htmlSuite "*firefox" "http://10.8.100.106" "C:\mytestsuite\mytestsuite.html" "C:\mytestsuite\results.html"
This should run your test suite in Firefox and write the results to the html file. Obviously you will need to change the "http://10.8.100.106" argument to your own server (this might just be localhost / 127.0.0.1)
It is possible to run individual test cases using Selenese Runner. You can specify a single test case file or a test suite as the unit to run.
We should execute the SeleniumRC in using following command;
java -jar filename.jar
ex:
java -jar program1.jar
the program1 consist of the followings are:
program1.class file
Resource library file such as SeleniumRC Server.jar and Selenium Java client.jar file
This method is applicable for SeleniumRC execution. We can directly create the program1.jar file from eclipse using
File->Export.
Here is an article that explains you step-by-step process of how to run Selenium RC application in Java.
Create a Java Selenium RC test script and executing the script
I have needed to do this before, and used the following:
An Ant Build (complex)
Creating a test runner class(a part of junit framework)class.
Most commonly we would run into build path errors while trying to run from cmd.
If you want to run it from command prompt you may consider writing your selenium test in python.
Make sure you have python installed if you are on windows. Mac will have python by default.
Running test from CMD is quite easy.
Follow below steps
1- Go to home directory and Set class path
Home Directory > set classpath=Home Directory\bin; and press enter
Home Directory > set classpath=Home Directory\lib*; and press enter
2-Home-directory > java org,testng.TestNG testng.xml testng2.xml testng2.xml and hit enter
I have documented all steps here. Hope it will help. Cheers
1) Running from CMD
java -cp "C:\ProjectX\Mortgage\bin;C:\Selenium_latest\selenium2.49.1\*;C:\Selenium_latest\selenium-2.49.1\libs\*" org.testng.TestNG C:\ProjectX\Mortgage\testng.xml
Run above command in C:\ProjectX\Mortgage
2) Create batch file name runner.bat
SET projectLocation=C:\ProjectX\Mortgage
CD %projectLocation%
SET classpath=%projectLocation%\bin;C:\Selenium_latest\selenium-2.49.1\*;C:\Selenium_latest\selenium-2.49.1\libs\*
java org.testng.TestNG %projectLocation%\testng.xml
PAUSE
3) Run the batch file by double clicking on it.
To be able to run in Chrome browser, you can use *chrome option instead of *firefox like below
java -jar selenium-server.jar -htmlSuite "*chrome" "http://localhost" "C:\testsuite\testsuite.html" "C:\testsuite\results.html"
Other browsers list include:
*firefox
*mock
*firefoxproxy
*pifirefox
*chrome
*iexploreproxy
*iexplore
*firefox3
*safariproxy
*googlechrome
*konqueror
*firefox2
*safari
*piiexplore
*firefoxchrome
*opera
*iehta
*custom
on session null