I am running a performance/reliability/stress(P/S/R) testing script in my SUT(system under test) using Robot Framework and some internal libraries (e.g. s2l, os, bulletin, collection, datetime and some own in-house libraries), and which need to run for 60 days to measure the expected P/S/R parameters.I know after completing its 60 days execution (if the SUT is not interrupted by any system or networking issues), i will get log and report file.
But, i have a requirement of getting its weekly execution status as log file or report file.
Is there any way to do this in Robot Framework,i am using robot framework only for my testing.Is there any internal/external libraries available (apart from bulletin library) to do this efficiently.
Or, can i include a python script and include the script in the test ENV, if so how can i do this, Any suggestions.
My recommendation would be to rewrite the test so that it runs for one week. Then, schedule a job using jenkins or a python script or bash script that runs that test eight times. This gives you the benefit of a weekly report, and at the end you can use rebot to combine all of the reports into a single larger report.
Another option would be to use the listener interface to stream test results to some other process or file. Then, once a week you can create your own report from this data. For example, you could set up an elastic search server to store the results, and use kibana to view the results.
Related
Currently I have a powershell, post-build script that launches our Selenium tests. The time of day is checked and if between 6:45 and 8:00 AM, the full test suite runs. If not, its a normal CI build and only a small subset of tests runs.
We are switching to TestCafe and I have added Build Definition steps to install testcafe, install testcafe-reporter-junit and run the tests. I's like to move the test runs to regular steps instead of scripted, if possible, but I would need to know if I can condition the full suite test step to only run during the above mentioned time period. Is that possible with custom conditions?
There is no condition syntax that deals with dates and time. But you can run a small scripted step to set a variable, then use that variable in the condition.
I have to do the following in my test.
Login to the site
click on a button to upload an excel file. It will kick off a task
Wait for job to complete (takes 20-25 minutes)
Verify the output of job (Successful, Failed)
Excel file contains data based on which job will either return success or failure.
I am trying to do this using selenium but the problem is that in the framework we can execute a test for max 10 minutes else it kills the test.
So not sure how to go about automating this scenario.
One of the way i could see is
First Test --> Just upload the file and kick off the task
Second Test (depends on First Test) --> verify the results.
Have a you faced similar situation.
Could any of you please suggest the approach for this? or can suggest alternative tool that i could use to automate this scenario.
I am wondering about where to start in building a test framework here.
I create a vb.net application to display the list of projects available. Allow user to select the project, time and date when test needs to get executed.
Once the user decides the time and task, I want my system to schedule a task onto a remote machine where the test execution would happen at the specified time.
I am stuck at point two. any pointers or question is much appreciated.
I use testcomplete for automation.
I want my system to schedule a task onto a remote machine where the test execution would happen at the specified time
There's a Windows Task Scheduler and associated API that supports scheduling tasks at specific times. The API is aimed at C++ programmers.
You could use the Task Scheduler Managed Wrapper available on CodePlex for easy interop with VB.Net.
The task to execute could be copied to a network drive so that it is accessible from the remote machine.
For point 2 you'll have to call TestComplete from the command line as per these instructions:
http://support.smartbear.com/viewarticle/55587/
You can also call TestExecute from the command line, it's a cut down version of Test Complete that will run your tests. Your license may or may not include that.
Did you also consider taking a look at Jenkins for scheduling your test runs?
I am trying to create a unit test to run on two machines in Microsoft Test Manager 2010. In this test I want some client and server side test code to run simultaneously; the client side test being dependent on server side test working successfully.
When putting together a Test Suite in Test Manager, I want to be able to set both tests to have the same order value (so they run at the same time) but the validation prevents this; setting the order as shown below:
Is there any way I can achieve the simultaneous test execution I am after?
Sorry for the late answer... I've missed the notification about your answers to my question :-( Sorry for that!
In case you are still looking for solution, here my suggestion.
I suppose you have a test environment consisting of two machines (for server and client).
If so, you will not be able to run tests on both of them, or better to say you will not have enough control over running tests. Check How to Run automated tests on multiple computers at the same time
Actually I posted a related question to "Visual Studio Development Forum", you could check the answers I got here: Is it possible to run test on several virtual machines, which belong to the same environment, using build-deploy-test workflow
That all means you will end up creating two environments each consisting of one machine (one for server and one for client).
But then you will not be able to reference both environment in your build definition it you can only select one environment in DefaultLabTemplate.
That leads to the solution I can suggest:
Create two lab environments
Create three build definitions
the first one will only build your test code
the second one will deploy last successful build from the first one and start tests on the server environment
the third one will deploy last successful build from the first one and start tests on the client environment.
Run the first build definition automatically at night
Trigger the latter two simultaneously later.
It's not really nice, I know...
You will have to synchronize the build definition building the test code with the two build definitions running the tests.
I was thinking about setting up similar tests some months ago and it was the best solution I came up with...
Another option I have not tried yet could be:
Use a single test environment consisting of two machines and use different roles for them (server and client respectively).
In MTM create two Test Settings (one for the server role and one for the client role).
Create a bat file starting tests using tcm.exe tool (see How to: Run Automated Tests from the Command Line Using Tcm for more details).
You will need two tcm.exe calls, one for each Test Settings you have created.
Since a tcm.exe call just queues a test run an returns (more or less) immediately this bath file will start tests (more or less) simultaneously.
Create a build definition using DefaultLabTemplate.
This definition will:
build test code
deploy them to both machines in your environment
run your bath script as the last deployment step
(you will have to make sure this script is located on the build machine or deploy it there or make it accessible from the build machine)
As I've said, I have not tried it yet.
The disadvantage of this approach will be that you will not see the test part in the build log since the tests will not be started by means provided by DefaultLabTemplate. So the build will not fail when tests fail.
But you will still be able to see test outcomes in MTM and will have test results for each machine.
But depending on what is more important to you (having rest results or having build definition that fails if tests fail or having both) it could be a solution for you.
Yes, you can with modified TestSettings file.
http://blogs.msdn.com/b/vstsqualitytools/archive/2009/12/01/executing-unit-tests-in-parallel-on-a-multi-cpu-core-machine.aspx
I have a complex screen-scraping script that I've put together that uses Selenium2, the Selenium web driver and PHP binding script, so at the end of it all, I have a PHP script that drives Selenium, which in turn fetches a URL, parses some Javascript, fills out a form, blah blah blah, and then returns the HTML that is ultimately what I'm after. It all works great on my local computer (as a development and proof-of-concept environment).
So.
For production, I need this script to run automatically three times every day. I am trying to figure out if it would be better for me to set up everything on my server (meaning: figure out how to get Firefox for Linux going, then Java, then Selenium2, etc, etc... not trivial for me; Damn it Jim, I'm a coder, not a sysadmin!), or if I can use a 3rd-party Selenium testing service like Sauce Labs' OnDemand, or any of these other cloud-based Selenium services.
Those 3rd party solutions seem like they're all set up for "unit testing," which is totally not what I'm doing. I don't know about that stuff, or using PHPUnit, or doing tests with builds, or whatever. I just want to run my straightforward PHP script 3x/day and have it talk to Selenium to drive a browser and do my screen scraping.
Are one of those 3rd party solutions a good idea for what I'm trying to accomplish, or are they overkill/too far away from my (relatively simple) goal?
First, I want to let you know that I use Selenium with Ruby so I am assuming that running your php script will start up the selenium webdriver and run your tests... I will just explain how easily run your script 3 times a day without needing to be a sysadmin master.
Linux has an extremely stable and robust command called cron which is what you will need to use. It allows you to schedule actions to happen daily/hourly/whatever.
The first thing you want to do is to go to the directory with your script. I will refer to your script as script.php.
First thing is to make sure that the top line of your script is:
#!/usr/bin/php
In the directory you will execute the following command to make your file accessible by the system:
chmod +x script.php
Now set up your cron job with the following command:
crontab -e
Then put in your job:
00 4,12,20 * * * /home/sean/script.php
00 - Means at 00 minutes.
4,12,20 - Are the hours (it is a 24 hour clock.)
The first: * - Every day
The second: * - Every month
The third: * - Every Day of the week
So this script would run every day, every week, every month at 4,noon and 8pm.
Obviously change the directory to the script on your system and set the times to whenever you want the scraping to occur.
I hope this helps!
-Appended stuff for the java/firefox-
First off, take this all with a grain of salt since I am using Ruby :)
Okay to get java/firefox running you will probably want to grab the selenium standalone. You can grab it here.
Then to run the selenium server you just:
java -jar selenium-server-standalone-2.5.0.jar
You can run put the standalone server starting in the cron job and then close it in your script file.