I am currently running capybara specs that test different functionality of an app. When running my specs, it appears that the page is not being reached fast enough to test. I have three specs currently, and it takes 9 minutes to end up having them all fail with this same error. Here is the result of running the specs
Randomized with seed 38457
Capybara starting Puma...
* Version 3.11.4 , codename: Love Song
* Min threads: 0, max threads: 4
* Listening on tcp://127.0.0.1:50109
FFF
Failures:
1) Successful source is created
Got 0 failures and 2 other errors:
1.1) Failure/Error: visit ('/clients/new')
Net::ReadTimeout:
Net::ReadTimeout
# ./spec/qa/variables.rb:12:in `login_user'
# ./spec/qa/successful_source_spec.rb:7:in `block in (root)'
1.2) Failure/Error: #io.to_io.wait_readable(#read_timeout) or raise Net::ReadTimeout
Net::ReadTimeout:
Net::ReadTimeout
Also note that I am using selenium with chrome headless.
Is there anything I can do to have the page load faster so that I can test? This is also the first time these specs are being run.
I decided to go with a quick fix for this problem. It turns out that the rails server was not connecting to the page fast enough before the test would time out. I decided to change the default wait time in the spec_helper.rb to 120 seconds. It will take a lot longer the first time you run the specs, but they eventually will connect and run smoothly from then on. I do not think this is best practice, but it does provide a quick fix to begin testing your specs.
Related
Actually, I am creating automation testing for an e-commerce website. Actually, the website have function lazy load or something. I am testing it on UAT server. So, it will load the page slowly because the specification of the server. It takes more than 60 sec or more to load all the resources from the webpage. So, when I am trying to create selenium automation, it always waiting more than 60 sec to continue the next step (because waiting the page fully loaded). Please, someone give me tips how to continue run the test step after 10 seconds wait the page to load. It won't throw an exception, just continue the test step.
Not possible.
If you find some element and try execute some action while loading you will get stale element error + due loading issue you will have a lot of failed tests and it will take a lot more time to debug.
Automation means to execute fast and have reliable results.
It seems that this environment is not built for automation, you should request more resources.
As an alternative maybe you can use a headless driver or see if you can put the same build on a VM.
Why this is an issue: Selenium needs to wait for each request to be complete.For example when you request a page, if the page is not received entirely and the server still sending info then the request is not done, it is logical that you need a complete request in order to continue.
You should address this to your Project Manager/QA Lead and ask for advice/option on how to handle this.
Please note that these costs should be included/added in the automation price.You need to address this in a simple way:
good server -> automation runs smoothly and fast and the testing is
done faster
bad server -> unable to run automation since is not reliable and each
test has a high rate of failure => alternative X day(s) of
manual testing for each build
If this would be a coding issue like some delayed ajax request then you would have some solutions, devs could help, but if is an infrastructure/resources issue then if not depending on you, and you cannot solve it.
You could use try any type of wait implicit/explicit, explicit would throw some exception, but this is not a solution for poor resources.
I'm having a really tough time investigating the cause of a test failure. I'm a very experienced programmer and am well versed in general debugging techniques, but I'm new to Capybara and RSpec so I'm hoping there's some kind of facility I'm ignorant of that can help me.
In short, I have a test something like this:
expect { click('.fake_button'); sleep 1 }.to change { clicks.count }.by(1)
When the fake button is clicked, it triggers an AJAX call to the Rails app which, among other things, adds a click record to the database. I can think of dozens of things that could be causing this test to fail and have had only limited success getting information out of logs. The tests do not fail in development and it only fails sporadically in test. One of the differences of the test environment is that the tests are run on a server in our office against a server in the cloud, so there are network delays along with other possible issues.
This is very hard to diagnose because there's so little information coming out of the failed test and of course all the database information is thrown away by the time I read about the failure. I know clicks.count didn't change in the test and I can infer that click('.fake_button') succeeded, but due to server time sync issues I can't even be sure that the click happened on the right button or that the AJAX call fired.
What I'd like are some tools to help me follow this test case in the web server logs (maybe using automatic URL parameters, for example), detailed logging about what Capybara did, and a record of the web page as it was when the failure occurred, including cookie values. Can I get any of that? Anything like that?
Capybara simulates human actions. The test code does exactly what needed. It's something a real user should expect. I don't think you should complain the code.
I think it's okay to increase the wait time, say 1 to 2, due to your network latency, but it should not exceed a reasonable value otherwise the app does not work as real user expected.
To debug Capybara codes, there are three methods as I summarized:
Add "save_and_open_page" to the place you want to see result. Then a saved html page will appear during the test. (I forget if "launchy" gem should be added)
Temporarily set this test as JS to see how this test going.
scenario "a fake test", js: true do
# code here
end
By doing this a real browser will pop up and Capybara will show you step by step how it play the code.
Just run $ tail log/test.log to show what happened recently.
Building off what #Billy suggested, log/test.log was not giving me any useful information and I was already using js: true so I tried this:
begin
expect { click('.fake_button'); sleep 1 }.to change { clicks.count }.by(1)
rescue Exception => e
begin
timestamp = Time::now.strftime('%Y%m%d%H%M%S%L')
begin
screenshot_name = "tmp/capybara/capybara-screenshot-#{timestamp}.png"
$stderr.puts "Trying to save screenshot #{screenshot_name} due to test failure"
page.save_screenshot(screenshot_name)
rescue Exception => inner
$stderr.puts "Ignoring exception #{inner} while trying to save screenshot of test page"
end
begin
# Page saved by Capybara under tmp/capybara/ by default
save_page "capybara-html-#{timestamp}.html"
rescue Exception => inner
$stderr.puts "Ignoring exception #{inner} while trying to save HTML of failed test page"
end
ensure
raise e
end
end
Later I changed the test itself to take advantage of Capybara's AJAX synchronization features by doing something like this:
starting_count = clicks.count
click('.fake_button')
page.should have_css('.submitted') # Capybara is smart enough to wait for this to happen
clicks.count.should == starting_count + 1
Note that the CSS I'm looking for is something added to the page in JavaScript by the AJAX callback, so it showing up is a signal that the AJAX call completed.
The rescue blocks are important because the screenshot has a high failure rate from not having enough memory to render the full page and convert it to an image.
EDIT
Though I haven't tried it, a promising solution is Capybara::Screenshot which automatically saves the screenshot and HTML on any failure. Just reading the code it looks like it will have problems when the screenshot fails and I can't tell what state the page will be in by the time the screenshot is triggered, but it certainly looks like it's worth a try.
A nice way to debug tests is to use irb to watch what's actually happening in the browser. RSpec fails usually give decent information for simple cases, but for more complicated things I either split the case up until it is simple, or chuck it in irb for a live session to make sure its doing what it should do.
Make sure to use :selenium as your driver, and you should see firefox come up and be able to be driven by your irb session.
I'm seeing an error in some integration specs, using rspec, capybara, capybara-webkit and timecop.
Capybara::FrozenInTime:
time appears to be frozen, Capybara does not work with libraries which freeze time, consider using time travelling instead
The only gem that I know that freezes time is Timecop, but I'm not using it in the test case that fails.
Since the error occurs only sometimes I can't even know if its gone or not after changing something.
The end of the error message holds the solution:
consider using time travelling instead
Just change Timecop.freeze to Timecop.travel. Timecop.freeze breaks Capybara's auto-wait feature.
In addition, I would call Timecop.return in an after block, as it will be associated with the most recent travel block:
after :each do
Timecop.return
end
The solution I found was to add
before :each do
Timecop.return
end
in spec_helper.rb.
This way we garantee that the time is not frozen before each test, although the only ones that have this problem are the ones executed in a webdriver different from rack-test. In my case capybara-webkit.
Having a bit of an oddity with some capybara webkit-driver (:js => true) tests.
Tests run fine when they run on their own, but somehow in sequence, they fail.
For example, I have a request test that looks something like
describe "A", :js => true do
# tests here run fine
end
describe "B", :js => true do
# tests here fail
end
When I split describe B section into its own file, and run it using bundle exec rspec spec/requests/b_spec.rb however - tests run fine and pass.
Debugging this, it looks like when both sections are in the same file, somehow the webkit driver loads a 'dirty' browser session. Trying things like page.driver.reset! or Capybara.reset_sessions! or Capybara.reset! doesn't seem to have any effect...
When using spectator/spork this isn't a problem, since I can split the tests into different files and run them independently, but when running the full suite of tests using bundle exec rspec - these tests fail...
How can I reset the webkit driver / session properly between tests? Or I am chasing the wrong problem?
p.s. These tests are not hitting the database or altering state in any particular way, so I'm pretty sure it's some driver related problem.
Sometime it helps just writing the question for the solution to pop up.
The key for me was:
These tests are not hitting the database or altering state in any particular way, so I'm pretty sure it's some driver related problem.
Turns out there was a state-change. In my particular case, setting OmniAuth into test_mode, which required setting it back to false after the previous test was running...
I recently switched from Heroku's Bamboo stack to the Cedar one (Rails 3.1.4, Ruby 1.9.2, Thin gem for web server). Since then I keep getting 500 errors such as this, where it seems that the query is not acting right:
207 <13>1 2012-05-06T16:10:51+00:00 d. app web.1 - - ActiveRecord::StatementInvalid (Mysql::Error: : SELECT `foos`.* FROM `foos` WHERE `foos`.`id` = ? LIMIT 1)
It's not an error in the code though because the page eventually renders successfully (ie status 200) when I refresh the page. Sometimes it is 1 refresh, but can get up to 4 refreshes before I get a 200.
I thought it was the database because I was on ClearDB's free plan, but I upgraded to ClearDB's next plan with better I/O performance and it still happens
this never happened when I was on Bamboo
it happens on just about every page that does queries on the DB
it doesn't always happen, but I'd say it happens on at least 1 in 5 pages views
the model/query doesn't matter, the same error occurs (just indicating a different model/fields then the example above)
Do you get the same errors if you are in console heroku run console ? I've never seen this before. Try upgrading your Mysql gem, which one are you using http://api.rubyonrails.org/classes/ActiveRecord/StatementInvalid.html i think the correct one is mysql2 https://rubygems.org/gems/mysql2