I have a program with a COM API. I have a suite of functional tests using Python, but I want to provide a suite of VBscript code samples, and I'd like to test this suite of code automatically.
Is there a tool I can use to run a suite of VBscript tests without interaction, and collect pass/fail and timing statistics?
I found a test runner for vbscript, ScriptUnit.
It can be run without the GUI using the /Q option, so it looks like this one will fit the bill.
Related
I'm trying to improve the testing process where I work, but without adjusting the structure.
What we have: VSTS, Selenium IDE, Testers who write test cases, but not code.
What I'd like to do is manage a way to marry our TFS continuous integration with the Selenium tests we write. These are NOT the code-driven selenium tests, but rather the IDE version where users click through, and set assertions using the IDE (All are just UI tests). I know we can export those tests plans as a .SIDE file, but what I can't figure out, is how to have our TFS server execute those as part of a deployment or build pipeline.
Ideally, developers/devops would setup projects in TFS from the onset with whatever solution makes sense to execute these Selenium .SIDE files, but afterwards, the testers would manage adding/modifying those tests cases elsewhere.
The real goal here is to not have testers writing code, or checking in code. Only writing these UI Selenium tests, but having TFS execute those as part of CI.
Researching this on the internet drives me basically always to something that requires testers to write code.
I don't think it can automate testing without code, at lease, you need a test project containing your automated tests.
Generally, in Azure DevOps, we use Visual Studio Test task to run tests. This task supports using the following tests:
Test assembly: Use this option to specify one or more test assemblies that contain your tests. You can optionally specify a
filter criteria to select only specific tests.
Test plan: Use this option to run tests from your test plan that have an automated test method associated with it. To learn more about
how to associate tests with a test case work item, see Associate
automated tests with test cases.
Test run: Use this option when you are setting up an environment to run tests from test plans. This option should not be used when
running tests in a continuous integration/continuous deployment
(CI/CD) pipeline.
This was a question that I had as well, and I think I found an imperfect but better solution.
I wasn't able to get my Selenium IDE tests running with Jenkins, but I was able to get them to run with TeamCity, another CI.
I created a build step like the following :
Runner type: Command Line
Working Directory: where the selenium IDE .side file is located
Run: Custom Script
With the build script content that I usually use to run my Selenium IDE Tests, such as selenium-side-runner sidefile.side
I also added the following so I could output the results in Junitor another form: --output-directory=results --output-format=junit
You can also add the following so the tests are run headlessly, this only works in Chrome : -c "goog:chromeOptions.args=[--headless,--nogpu] browserName=chrome"
Finally, I also use --filter to run one test suite at a time, but that is optional too.
I then used another build step to export the results to our test manger, xray, but I think that is beyond the scope of this question.
The problem with this solution is that it runs directly from a users individual machine still, but this can be work around.
I have a scenario like this - Application written in hybrid languages (Python, C++ and Java majorly). There are around 100 test cases written in Robot framework to test the application. Now I want to see code coverage of my application. Is there any tool that can work in such scenario?
Thanks in advance.
The same way you would do outside robotframework: Using external code analysis tools. Let me elaborate.
In Python, you can use tools like Coverage (https://pypi.org/project/coverage) to run your testing suite while gathering coverage data. For example, if you usually run your robot test suites using:
robot suites
(supposing you have a "suites" directory with your .robot files) then you would run robot as a Python module over Coverage like this:
coverage run -m robot suites
And you could get your report with:
coverage report
You'll probably need to filter the report (--include option) if you're only interested in code inside a directory. For example, for a directory "myproy" you would do:
coverage report --include *myproy/*
You could use a similar strategy with other Python test coverage tools as long as you figure out how to tweak their execution command to run robot as a module. The same holds for C++ and Java code analysis tools; For example, check the following link for a guide that uses Java with JaCoCo and Maven: https://www.cnblogs.com/z1500592/p/6676646.html
I have a situation where we would like to run our Selenium test suite, but
at certain points in the process we would like to execute a shell command
on our machine.
Does Selenium have the capability of executing a shell command and to
receive a response and then to act on those results? Is this possible? If
so, how?
thanks in advance.
You tagged it TestNG meaning you write your Selenium tests in Java. The Selenium framework itself does not have a method to run shell scripts. But Java does, you can just execute commands in your tests and later use the output or input from/for Selenium actions.
Runtime.getRuntime().exec()
Read more:
In the Java documentation: https://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html#exec(java.lang.String[])
How to Run a Shell Command in Java: https://www.baeldung.com/run-shell-command-in-java
Using the output of the Shell command: java runtime.getruntime() getting output from executing a command line program
Selenium is just a library for interacting with web browsers. To strictly answer your question.. the answer is "No, it can not". However, you can easily integrate execution of shell scripts from your test code or framework. How you do that will depend on the language you are using.
Yes you can, but you have to use certain specific libraries along with it in addition to the language you choose eg. If you choose java, you either have to provide repo url via maven or gradle but if you just use python instead since it's highly interpreted, it can pull dynamic content that do pose limitation in Java Selenium, TestNG, PDFBox, POI lib etc. You can deploy a js library of your choice and include that function inside your python code to be invoked via the commands or action that you set your application to.
In short, use Python for purely less troubling dynamic meta extractions, string manipulation, piping output to any other file. Hope that helps.
Is there an open source tool available to control the running of BDD cucumber tests?
We are developing BDD cucumber tests and would like the option to control the tests when running them (start/stop/pause/restart) using an open source (or proprietary) test tool.
The short answer it, yes.
The somewhat longer answer is that it depends on your echo system.
If you are using Java, then any build tool will be sufficient. That is Maven, Gradle or similar. These are easy to integrate in your Continuous Integration, CI, environment. With a tool chain like that, you are able to execute Cucumber on every build and will always know if your system works or not.
Yes , but in small scope (Automation tests) and less process control related to run and control tests ,In high scope with multiple branches and projects i think you have to move to Jenkins with full control.
Following link describe the coparsion : https://www.saashub.com/compare-jenkins-vs-cucumber
I am creating selenium test to test a web app. I also want to be able to use these test cases as a way for non technical users to automate the first few steps of the process. ~Is this possible?
If you are using testng, generate command to run XML from the command line and then copy the same command to .bat or .cmd file. By a click any user can run those tests/suite.
Of course you can, Selenium automates browsers, it is not a 'test library' you can use it in any program. Just write java program where you use selenium.
You can also write tests (in JUnit or TestNG or any other test runner) and run tests programmatically.