Feature specs fail only on CircleCI or Codeship continuous integration services - selenium

Question

My very basic feature specs are passing just fine locally but failing on CircleCI and Codeship. The tests that are failing:
require 'spec_helper'
describe 'Authentication' do
describe "sign in" do
it "is the landing page for non-authenticated users" do
user = create(:user)
visit root_path
expect( page ).to have_content 'LOG IN' # On sign-in page
fill_in 'Email', with: user.email
fill_in "Password", with: user.password
click_button 'Sign in'
expect( current_path ).to eq user_path(user)
end
end
describe 'registration' do
it "allows new users to register" do
visit root_path
click_link 'Sign up'
fill_in 'Email', with: 'myfake#email.com'
fill_in 'Password', with: 'password'
fill_in 'Password confirmation', with: 'password'
fill_in 'First name', with: 'John'
fill_in 'Last name', with: 'Doe'
click_button "Sign up"
expect( current_path ).to include '/users/'
expect( page ).to have_content "Welcome! You have signed up successfully."
end
end
end
The tests both fail on the first lines where they set expectations of the pages (expect( page ).to have_content "LOG IN" and click_link "Sign up", respectively), with errors suggesting the page HTML is completely blank:
expected to find text "LOG IN" in ""
I saved screenshots on CircleCI, and they indeed show a completely blank page.
Here's where it gets interesting. I tried debugging the problem by running/watching the specs on Circle using a VNC. When I a) set driver: :selenium for the tests, b) add a sleep 1 or two to the tests before testing the page expectations, and c) manually run the test after SSHing into their servers with the VNC, I can see the tests run in Selenium (they open a browser in the VNC) and they pass perfectly.
Outside of the VNC, however, the tests fail consistently in both CI servers. With or without tons of sleeps and driver: :selenium. Any ideas what could be causing this discrepancy between the regular CircleCI/Codeship servers and their VCN/my local test environment? I got in touch with the folks at CircleCI, but they're stumped for the moment.
If relevant, I'm running Ruby 2.2.0, Rails 4.2, Capybara 2.4.4, Capybara-Webkit 1.4.1, and Selenium-Webdriver 2.44.0
Some potentially relevant files:
spec_helper.rb
ENV["RAILS_ENV"] = "test"
require File.expand_path("../../config/environment", __FILE__)
require "rspec/rails"
require "shoulda/matchers"
require "webmock/rspec"
require 'vcr'
Dir[Rails.root.join("spec/support/**/*.rb")].each { |file| require file }
module Features
include Warden::Test::Helpers
Warden.test_mode!
def sign_in(user)
login_as(user, scope: :user)
end
end
module Controllers
# Pre-parse controller responses for easy access
def response_body
body = JSON.parse(response.body)
body.is_a?(Hash) ? body.to_sh : body.map(&:to_sh)
end
end
module Mock
def disable_webmock(&block)
WebMock.disable!
yield
WebMock.enable!
end
end
RSpec.configure do |config|
config.expect_with :rspec do |c|
c.syntax = :expect
end
# Save a screenshot to CircleCI when a feature test fails
config.after(:each, :type => :feature) do |example|
if example.exception
artifact = save_page
puts "\"#{example.description}\" failed. Page saved to #{artifact}"
end
end
config.include Features, type: :feature
config.include Controllers, type: :controller
config.include Mock
config.include Formulaic::Dsl, type: :feature
config.infer_spec_type_from_file_location!
config.infer_base_class_for_anonymous_controllers = false
config.order = "random"
config.use_transactional_fixtures = false
end
RSpec::Matchers.define :hash_eq do |expected|
match do |actual|
actual.recursive_symbolize_keys == expected.recursive_symbolize_keys
end
end
VCR.configure do |c|
c.cassette_library_dir = 'spec/fixtures/vcr_cassettes'
c.hook_into :webmock
c.allow_http_connections_when_no_cassette = true
c.configure_rspec_metadata!
c.ignore_hosts '127.0.0.1', 'localhost:3000'
end
ActiveRecord::Migration.maintain_test_schema!
Capybara.javascript_driver = :webkit
if ENV['CIRCLE_ARTIFACTS']
Capybara.save_and_open_page_path = ENV['CIRCLE_ARTIFACTS']
end
WebMock.disable_net_connect!(allow_localhost: true)
database_cleaner.rb
RSpec.configure do |config|
config.use_transactional_fixtures = false
config.before(:suite) do
DatabaseCleaner.strategy = :truncation
end
config.before(:each) do
DatabaseCleaner.start
end
config.after(:each) do
DatabaseCleaner.clean
end
end

Answer 1

I can think of a couple of things to try:
If you want to get your specs to run headless, try using poltergeist rather than capybara-webkit. poltergeist has several advantages which I described here: https://stackoverflow.com/a/24108439/634576
Selenium might just be a distraction, but, if you want to get it to work, be sure that your CI environment has the right version of Firefox. Each version of selenium-webdriver seems to need a narrow range of versions of Firefox. On CircleCI, you can configure circle.yml to install a specific version of Firefox. Right now we use selenium-webdriver 2.46.2 and Firefox 39.0.3, installed with the following in circle.yml:
dependencies:
pre:
- sudo apt-get update; sudo apt-get install firefox=39.0.3+build2-0ubuntu0.12.04.1
I don't know about Codeship.

Answer 2

I have experienced instability of my Selenium tests on CircleCI until I came across this post https://discuss.circleci.com/t/selenium-tests-timing-out/7765/2
After adding:
machine:
environment:
DBUS_SESSION_BUS_ADDRESS: /dev/null
to my circle.yml, my tests have become stable.