I have a complex screen-scraping script that I've put together that uses Selenium2, the Selenium web driver and PHP binding script, so at the end of it all, I have a PHP script that drives Selenium, which in turn fetches a URL, parses some Javascript, fills out a form, blah blah blah, and then returns the HTML that is ultimately what I'm after. It all works great on my local computer (as a development and proof-of-concept environment).
So.
For production, I need this script to run automatically three times every day. I am trying to figure out if it would be better for me to set up everything on my server (meaning: figure out how to get Firefox for Linux going, then Java, then Selenium2, etc, etc... not trivial for me; Damn it Jim, I'm a coder, not a sysadmin!), or if I can use a 3rd-party Selenium testing service like Sauce Labs' OnDemand, or any of these other cloud-based Selenium services.
Those 3rd party solutions seem like they're all set up for "unit testing," which is totally not what I'm doing. I don't know about that stuff, or using PHPUnit, or doing tests with builds, or whatever. I just want to run my straightforward PHP script 3x/day and have it talk to Selenium to drive a browser and do my screen scraping.
Are one of those 3rd party solutions a good idea for what I'm trying to accomplish, or are they overkill/too far away from my (relatively simple) goal?
First, I want to let you know that I use Selenium with Ruby so I am assuming that running your php script will start up the selenium webdriver and run your tests... I will just explain how easily run your script 3 times a day without needing to be a sysadmin master.
Linux has an extremely stable and robust command called cron which is what you will need to use. It allows you to schedule actions to happen daily/hourly/whatever.
The first thing you want to do is to go to the directory with your script. I will refer to your script as script.php.
First thing is to make sure that the top line of your script is:
#!/usr/bin/php
In the directory you will execute the following command to make your file accessible by the system:
chmod +x script.php
Now set up your cron job with the following command:
crontab -e
Then put in your job:
00 4,12,20 * * * /home/sean/script.php
00 - Means at 00 minutes.
4,12,20 - Are the hours (it is a 24 hour clock.)
The first: * - Every day
The second: * - Every month
The third: * - Every Day of the week
So this script would run every day, every week, every month at 4,noon and 8pm.
Obviously change the directory to the script on your system and set the times to whenever you want the scraping to occur.
I hope this helps!
-Appended stuff for the java/firefox-
First off, take this all with a grain of salt since I am using Ruby :)
Okay to get java/firefox running you will probably want to grab the selenium standalone. You can grab it here.
Then to run the selenium server you just:
java -jar selenium-server-standalone-2.5.0.jar
You can run put the standalone server starting in the cron job and then close it in your script file.
Related
This is a bit high-level solution based question: I have a website, and what I need to do is login->navigate->click category->check whether the specific content has come; every 10 mins.
There are quite a few third-party services, but they only checking the website uptime, which is not what I need.
Currently, I am thinking of silly way: create a selenium UI test, and run it every 10 mins, which seems pretty complex.
Feels like need some tool, so that I can run some extra js on a particular web to test my functionality. Any good solution for this? even some js framework, or even third-party payable service that I can direct make use of?
Thanks.
The best way to do this is :
Create a selenium test - Single main method that does all the steps.
Export it as a runnable jar
Schedule the .jar execution using windows scheduler.
Before scheduling the jar, once just double click the .jar file and see if the steps run smoothly.
Alternately you can try using vbscript (.vbs) file also.
I have a small collection of integration tests that utilize selenium in a class. The idea is that these tests run every time there is a merge to the codebase, with the merge proceeding through the pipeline and having a series of tests running against the new code.
The thing is, these selenium tests have to run one at a time. They're using the browser to log into a website, and the account will just log out if more than one person tries to log into the account at once, it'll just log out, and the test will obviously fail, so I need these tests to run one at a time. I've tried using the #NotThreadSafe annotation, doesn't seem to have changed anything, and I've searched through for some sort of switch or parameter that defines how many tests run at once with no luck. These tests are using junit 4.12.
I am running a performance/reliability/stress(P/S/R) testing script in my SUT(system under test) using Robot Framework and some internal libraries (e.g. s2l, os, bulletin, collection, datetime and some own in-house libraries), and which need to run for 60 days to measure the expected P/S/R parameters.I know after completing its 60 days execution (if the SUT is not interrupted by any system or networking issues), i will get log and report file.
But, i have a requirement of getting its weekly execution status as log file or report file.
Is there any way to do this in Robot Framework,i am using robot framework only for my testing.Is there any internal/external libraries available (apart from bulletin library) to do this efficiently.
Or, can i include a python script and include the script in the test ENV, if so how can i do this, Any suggestions.
My recommendation would be to rewrite the test so that it runs for one week. Then, schedule a job using jenkins or a python script or bash script that runs that test eight times. This gives you the benefit of a weekly report, and at the end you can use rebot to combine all of the reports into a single larger report.
Another option would be to use the listener interface to stream test results to some other process or file. Then, once a week you can create your own report from this data. For example, you could set up an elastic search server to store the results, and use kibana to view the results.
I'm using Selenium IDE 2.3.0 to record actions in my web application and create tests.
Before every test I have to clear all cookies, load the main page, log in with a specific user and submit the login form. These ~10 commands are fix and every test case needs them, but I don't want to record or copy them from other tests every time.
Is there a way to configure how "empty" test cases are created?
I know I could create a prepare.html file or something and prepend it to a test suite. But I need to be able to run either a single test or all tests at once, so every test case must include the commands.
Ok I finally came up with a solution that suits me. I wrote custom commands setUpTest and tearDownTest, so I only have to add those two manually to each test.
I used this post to get started:
Adding custom commands to Selenium IDE
Selenium supports object-oriented design. You should create a class that takes those commands that you are referring to and always executes those, in each of the tests that you are executing you could then make a call to that class and the supporting method and then execute it.
A great resource for doing this is here.
I've actually looked through Selenium questions on here and didn't find quite what I was looking for.
Basically I have about 10 "use cases" for smoke testing my site. Basic things such as, can a user log in, can they register etc..
I want to set this up on an interval such as every 10 minutes run these tests.
Is this possible with Selenium Remote Control? Does anyone have a link/tutorial they could point me towards. I'm fairly confident this is possible but I'm just not 100% sure how to get it all set up and running.
Thanks in advance.
Yes this can be done with Selenium RC. I have some Selenium Tutorials on my site. I have set up a basic user experience monitor before using Selenium RC, C#/Nunit and Windows Scheduled tasks to start the job to check the speed of our web app through the day.
Since I recommend using Selenium RC you can use any language you want to write the test.
Dending on what kind of environment you're working with you could use something like JUnit and an automated build system like hudson. This gives you all sorts of notification infrastructure when something goes wrong.
I have known people to run a script like this against both test and production systems (with a fixed user). In test environments you can discover programming mistakes, in production you can assess the up-ness of your system at a far more interesting level than pings or process watching.
Take a look at New Relic's Synthetics product, which wraps Selenium and provides periodic runs with alerting.