Phoenix/Elixir - Invalid Parameter error with Arc Ecto cast_attachments - amazon-s3

I am trying to make an image upload function with Phoenix/elixir, and I am getting an error on the create action with my console giving me the following error message:
[debug] Processing by Callme.DoctorController.create/2
Parameters: %{"_csrf_token" => "AnpLGQsANzU1YmxQAwA8Oy4CA3VzAAAA45djZauss3TelOpHInPFFQ==", "_utf8" => "✓", "doctor" => %{"avatar" => %Plug.Upload{content_type: "image/jpeg", filename: "file000166349580.jpg", path: "C:\\Users\\Frank\\AppData\\Local\\Temp/plug-1470/multipart-62273-279000-1"}, "bio" => "m", "hiddeninfo" => "m", "name" => "m", "picture" => "m", "specialty" => "m"}}
Pipelines: [:browser]
[error] Task #PID<0.406.0> started from #PID<0.403.0> terminating
** (ArgumentError) argument error
(crypto) :crypto.sha256_mac_nif(["AWS4", nil], "20160801", 32)
(crypto) crypto.erl:1055: :crypto.sha256_mac/3
(ex_aws) lib/ex_aws/auth.ex:90: ExAws.Auth.signing_key/3
(ex_aws) lib/ex_aws/auth.ex:53: ExAws.Auth.signature/7
(ex_aws) lib/ex_aws/auth.ex:42: ExAws.Auth.auth_header/7
(ex_aws) lib/ex_aws/auth.ex:15: ExAws.Auth.headers/6
(ex_aws) lib/ex_aws/request.ex:31: ExAws.Request.request_and_retry/7
lib/arc/storage/s3.ex:14: Arc.Storage.S3.put/3
(elixir) lib/task/supervised.ex:89: Task.Supervised.do_apply/2
(elixir) lib/task/supervised.ex:40: Task.Supervised.reply/5
(stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Function: #Function<2.133421496/0 in Arc.Actions.Store.async_put_version/3>
Args: []
[error] Task #PID<0.407.0> started from #PID<0.403.0> terminating
** (ArgumentError) argument error
(crypto) :crypto.sha256_mac_nif(["AWS4", nil], "20160801", 32)
(crypto) crypto.erl:1055: :crypto.sha256_mac/3
(ex_aws) lib/ex_aws/auth.ex:90: ExAws.Auth.signing_key/3
(ex_aws) lib/ex_aws/auth.ex:53: ExAws.Auth.signature/7
(ex_aws) lib/ex_aws/auth.ex:42: ExAws.Auth.auth_header/7
(ex_aws) lib/ex_aws/auth.ex:15: ExAws.Auth.headers/6
(ex_aws) lib/ex_aws/request.ex:31: ExAws.Request.request_and_retry/7
lib/arc/storage/s3.ex:14: Arc.Storage.S3.put/3
(elixir) lib/task/supervised.ex:89: Task.Supervised.do_apply/2
(elixir) lib/task/supervised.ex:40: Task.Supervised.reply/5
(stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Function: #Function<2.133421496/0 in Arc.Actions.Store.async_put_version/3>
Args: []
[error] Ranch protocol #PID<0.403.0> (:cowboy_protocol) of listener Callme.Endpoint.HTTP terminated
** (exit) an exception was raised:
** (ArgumentError) argument error
(crypto) :crypto.sha256_mac_nif(["AWS4", nil], "20160801", 32)
(crypto) crypto.erl:1055: :crypto.sha256_mac/3
(ex_aws) lib/ex_aws/auth.ex:90: ExAws.Auth.signing_key/3
(ex_aws) lib/ex_aws/auth.ex:53: ExAws.Auth.signature/7
(ex_aws) lib/ex_aws/auth.ex:42: ExAws.Auth.auth_header/7
(ex_aws) lib/ex_aws/auth.ex:15: ExAws.Auth.headers/6
(ex_aws) lib/ex_aws/request.ex:31: ExAws.Request.request_and_retry/7
lib/arc/storage/s3.ex:14: Arc.Storage.S3.put/3
(elixir) lib/task/supervised.ex:89: Task.Supervised.do_apply/2
(elixir) lib/task/supervised.ex:40: Task.Supervised.reply/5
(stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Here is the the create action:
def create(conn, %{"doctor" => doctor_params}) do
changeset = Doctor.changeset(%Doctor{}, doctor_params)
case Repo.insert(changeset) do
{:ok, _doctor} ->
conn
|> put_flash(:info, "Doctor created successfully.")
|> redirect(to: doctor_path(conn, :index))
{:error, changeset} ->
render(conn, "new.html", changeset: changeset)
end
And here is my doctor.ex file:
defmodule Callme.Doctor do
use Callme.Web, :model
use Arc.Ecto.Model
schema "doctors" do
field :name, :string
field :hiddeninfo, :string
field :bio, :string
field :specialty, :string
field :picture, :string
field :avatar, Callme.Avatar.Type
has_many :services, Callme.Service
timestamps
end
#required_fields ~w(name hiddeninfo bio specialty picture )
#optional_fields ~w()
#required_file_fields ~w()
#optional_file_fields ~w(avatar)
#doc """
Creates a changeset based on the `model` and `params`.
If no params are provided, an invalid changeset is returned
with no validation performed.
"""
def changeset(model, params \\ :empty) do
model
|> cast(params, #required_fields, #optional_fields)
|> cast_attachments(params, #required_file_fields, #optional_file_fields)
end
end
Here is the Avatar.ex file:
defmodule Callme.Avatar do
use Arc.Definition
use Arc.Ecto.Definition
# Include ecto support (requires package arc_ecto installed):
# use Arc.Ecto.Definition
#versions [:original, :thumb]
#extension_whitelist ~w(.jpg .jpeg .gif .png)
def acl(:thumb, _), do: :public_read
def validate({file, _}) do
file_extension = file.file_name |> Path.extname |> String.downcase
Enum.member?(#extension_whitelist, file_extension)
end
def transform(:thumb, _) do
{ :noaction }
end
def filename(version, _) do
version
end
def storage_dir(_, {file, user}) do
"uploads/avatars/#{user.id}"
end
def default_url(:thumb) do
"https://placehold.it/100x100"
end
end
My mix file:
defmodule Callme.Mixfile do
use Mix.Project
def project do
[app: :callme,
version: "0.0.1",
elixir: "~> 1.0",
elixirc_paths: elixirc_paths(Mix.env),
compilers: [:phoenix, :gettext] ++ Mix.compilers,
build_embedded: Mix.env == :prod,
start_permanent: Mix.env == :prod,
aliases: aliases,
deps: deps]
end
# Configuration for the OTP application.
#
# Type `mix help compile.app` for more information.
def application do
[mod: {Callme, []},
applications: [:phoenix, :phoenix_html, :cowboy, :ex_aws, :httpoison, :logger, :gettext,
:phoenix_ecto, :postgrex]]
end
# Specifies which paths to compile per environment.
defp elixirc_paths(:test), do: ["lib", "web", "test/support"]
defp elixirc_paths(_), do: ["lib", "web"]
# Specifies your project dependencies.
#
# Type `mix help deps` for examples and options.
defp deps do
[{:phoenix, "~> 1.1.4"},
{:postgrex, ">= 0.0.0"},
{:ecto, "~>1.1.2", override: true},
{:phoenix_ecto, "~> 2.0"},
{:phoenix_html, "~> 2.4"},
{:phoenix_live_reload, "~> 1.0", only: :dev},
{:gettext, "~> 0.9"},
{:comeonin, "~> 1.0"},
{:mailgun, "~> 0.1.2"},
{:poison, "~> 2.1", override: true},
{:cowboy, "~> 1.0"},
{:arc, "~> 0.5.2"},
{:ex_aws, "~> 0.4.10"},
{:arc_ecto, "~> 0.3.2"},
{:httpoison, "~> 0.7"}]
end
# Aliases are shortcut or tasks specific to the current project.
# For example, to create, migrate and run the seeds file at once:
# $ mix ecto.setup
#
# See the documentation for `Mix` for more info on aliases.
defp aliases do
["ecto.setup": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"],
"ecto.reset": ["ecto.drop", "ecto.setup"]]
end
end
and my config file:
use Mix.Config
# Configures the endpoint
config :callme, Callme.Endpoint,
url: [host: "localhost"],
root: Path.dirname(__DIR__),
secret_key_base: "jTrdsYnbNCqUuRuBqUIg0p6YKgUG3paPiZgB0ivCF45uWfUhZAd/zxuOiOe0GVKe",
render_errors: [accepts: ~w(html json)],
pubsub: [name: Callme.PubSub,
adapter: Phoenix.PubSub.PG2]
# Configures Elixir's Logger
config :logger, :console,
format: "$time $metadata[$level] $message\n",
metadata: [:request_id]
config :arc,
bucket: "callme"
config :ex_aws,
access_key_id: System.get_env("AWS_ACCESS_KEY"),
secret_access_key: System.get_env("AWS_SECRET_ACCESS_KEY")
#.env library
# Import environment specific config. This must remain at the bottom
# of this file so it overrides the configuration defined above.
import_config "#{Mix.env}.exs"
import_config "config.secret.exs"
# Configure phoenix generators
config :phoenix, :generators,
migration: true,
binary_id: false
config :callme, Callme.Repo,
adapter: Ecto.Adapters.Postgres,
username: "postgres",
password: "postgres",
database: "callme_dev",
hostname: "localhost",
pool_size: 10
And also, here is a link to the github repository because I can't think of any other files that might have errors on them.
I hope we can figure this out. This has been literally an entire week now of trying to figure what is wrong with this thing and I feel like I am taking crazy pills.

Related

Rswag: Authorization header appeared in query parameter

Version: rswag (2.0.5), rspec (3.8.0)
Environment: Rails 5.2.3, Ruby 2.4.5
It is my first time to use it, was stuck in authorization header for a day.
Here is what I did:
# in spec/swagger_helper.rb
config.swagger_docs = {
'api/v1/swagger.json' => {
swagger: '2.0',
info: {
title: 'API V1',
version: 'v1'
},
paths: {},
securityDefinitions: {
JWT: {
description: 'the jwt for API auth',
type: :apiKey,
name: 'Authorization',
in: :header
}
}
}
}
# in spec/integration/api/v1/nodes_spec.rb
path '/api/v1/nodes' do
get 'Get all servers' do
tags TAGS_NODE
produces 'application/json'
security [JWT: {}]
parameter name: :searchString, in: :query, type: :string
parameter name: :searchColumn, in: :query, type: :string
#parameter name: 'Authorization', :in => :header, :type => :string
let(:nodes) { create_list(:node_list, 32) }
response '200', 'Servers found' do
let(:'Authorization') { "Bearer #{gen_jwt}" }
let(:searchString) { 'test' }
let(:searchColumn) { ';Name;' }
run_test! do |repsonse|
data = JSON.parse(response.body)
puts data
end
end
end
end
Expected: The 'Bearer ....' is set in 'Authorization' header
Actual: In the test log, I found:
[INFO] [2019-09-29 01:11:13 UTC] [anony] [no session] [no req] [other other]Started GET "/api/v1/nodes?searchString=test&searchColumn=;Name;&params&headers[HTTP_AUTHORIZATION]=Bearer+eyJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1Njk3NjI2NzMsImVtYWlsIjoidGVzdEBpYm0uY29tIiwib3JnYW5pemF0aW9ucyI6WyJ0ZXN0X29yZzEiLCJvcmcyIl0sInJvbGVzIjpbImNjX2NvbnNvbGVfYWRtaW4iLCJyb2xlMiJdLCJpbnZlbnRvcnkiOlsidGVzdF9pbnYiXX0.eenTMWUy6kSHO58_kLKoKWmNjvQ9i5TU9ex4Ou-ausE&headers[HTTP_ACCEPT]=application%2Fjson" for 127.0.0.1 at 2019-09-29 01:11:13 +0000
[INFO] [2019-09-29 01:11:13 UTC] [anony] [no session] [no req] [other other]Processing by Api::V1::NodesController#index as HTML
......
[DEBUG] [2019-09-29 01:11:13 UTC] [anony] [no session] [no req] [other other]Auth by JWT token....
[ERROR] [2019-09-29 01:11:13 UTC] [anony] [no session] [no req] [other other]No JWT token included in request
As marked as bold in the log, the 'Authorization' as well as 'Accept' headers are appeared in query parameters, which are supposed to be http headers, so that no JWT token can be retrieved from header in code.
I also tried not to use securityDefinition, but specify a parameter in header as following: parameter name: 'Authorization', :in => :header, :type => :string. It did not work either.
Not sure any configuration I missed, or something wrong I did? Thanks!
Update: it seems to be related to other gems conflict? I had another try to create a new Rails 5 api only app, add rspec and rswag gems only, and run with a simple test case, it worked!
Here is my Gemfile:
source 'https://rubygems.org'
git_source(:github) { |repo| "https://github.com/#{repo}.git" }
ruby '2.4.5'
gem 'rails', '5.2.3'
gem 'puma', '3.11'
gem 'bootsnap', '1.1.0', require: false
group :development, :test do
# Call 'byebug' anywhere in the code to stop execution and get a debugger console
gem 'byebug', platforms: [:mri, :mingw, :x64_mingw]
end
group :development do
gem 'listen', '>= 3.0.5', '< 3.2'
end
group :test do
# Test framework
gem "rspec-rails"
gem "database_cleaner", '1.6.0'
gem "simplecov"
gem "simplecov-rcov"
gem "factory_bot_rails", '5.1.0'
gem "ci_reporter_rspec"
gem "faker"
end
# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby]
gem 'pg', '1.1.4'
gem 'delayed_job_active_record', '4.1.3'
gem 'delayed_job_worker_pool', '0.2.3'
gem 'dalli', '2.7.8'
gem 'ruby-kafka', '0.7.5'
gem 'active_model_serializers', '0.10.10'
gem 'will_paginate', '3.1.7'
gem 'rest-client', '1.8.0'
gem 'symmetric-encryption', '4.3.0', require: false
gem 'unicorn', '5.2.0'
gem 'rubyzip', '1.2.2'
gem 'jwt', '2.2.1'
gem 'rubyXL', '3.3.30'
gem 'apartment', '2.2.1'
gem 'rswag', '2.0.5'
[Resolved]
Seems not working with Rack::Test::Methods
It worked after remove the line 'include Rack::Test::Methods" in a helper file, which was added previously to use 'get' to test the API.
Seems not working with Rack::Test::Methods
It worked after remove the line 'include Rack::Test::Methods" in a helper file, which was added previously to use 'get' to test the API.

phoenix module ExAws.S3.Client is not loaded and could not be found

I use the library https://github.com/CargoSense/ex_aws, and I faced such problem:
== Compilation error on file web/models/s3.ex ==
** (CompileError) web/models/s3.ex:2: module ExAws.S3.Client is not loaded and could not be found
(elixir) expanding macro: Kernel.use/2
web/models/s3.ex:2: Minion.S3 (module)
(elixir) lib/kernel/parallel_compiler.ex:116: anonymous fn/4 in Kernel.ParallelCompiler.spawn_compilers/1
in my mix.exs:
def application do
[mod: {Minion, []},
applications: [:phoenix, ... :ex_aws, :httpoison, :poison]]
end
...
defp deps do
[{:phoenix, "~> 1.2.1"},
...
{:ex_aws, "~> 1.0.0-beta0"},
{:poison, "~> 2.0"},
{:httpoison, "~> 0.8"}]
end
my config.exs:
config :minion, :ex_aws,
access_key_id: "...",
secret_access_key: "...",
region: "us-east-1",
s3: [
scheme: "http://",
host: "...",
region: "us-east-1"
]
my s3.ex:
defmodule Minion.S3 do
use ExAws.S3.Client, otp_app: :minion
end
I will be grateful for any help to fix this problem
:ex_awsin version >= 1.0.0 does not use the old (pre 1.0.0) style where you needed to do as in your module Minion.S3. That's why ExAws.S3.Client does not exists. Instead, try:
Remove your file s3.ex
In config.exs, change from config :minion, :ex_aws, to config :ex_aws,
Update your requests to the new style: S3.list_buckets |> ExAws.request #=> {:ok, response} instead of old style Minion.S3.list_buckets

Uploading images to amazon S3 using Phoenix/elixir and Arc

Hey I am having problems getting Arc to upload images to my amazon S3 account. I don't know if the problem is configuring the app to connect to the account properly or how to set the upload file correctly. I have tried a few ways of doing things and have not had success.
First, I tried hard setting my key ID and Secret access key in a secret.config.exs file like so:
config :ex_aws,
access_key_id: "My key ID was here",
secret_access_key: "My secret access key was here",
region: "us-west-2",
s3: [
scheme: "https://",
host: "s3.amazonaws.com",
region: "us-west-2"
]
But no success. I tried again to set them in the config file using system environment variables like this:
config :arc,
bucket: "callmemd"
config :ex_aws,
access_key_id: [{:system, "AWS_ACCESS_KEY_ID"}, :instance_role],
secret_access_key: [{:system, "AWS_SECRET_ACCESS_KEY"}, :instance_role]
But, I am not sure I did that correctly. This is the code in the controller I am trying to use to call the upload with the uploaded file being saved properly in a param named avatar as an upload plug:
def create(conn, %{"doctor" => doctor_params}) do
avatar = doctor_params["avatar"]
Myapp.Avatar.store({avatar.filename, avatar.path})
I have also tried the following different configurations and many other variations with no luck:
Callme.Avatar.store({doctor_params["avatar"], doctor_params["id"]})
Callme.Avatar.store({%Plug.Upload{doctor_params}, doctor_params["id"]})
Callme.Avatar.store({%Plug.Upload{}, doctor_params["id"]})
Callme.Avatar.store({doctor_params["avatar"], doctor_params["id"]})
Here are my dependencies
defp deps do
[{:phoenix, "~> 1.1.4"},
{:postgrex, ">= 0.0.0"},
{:phoenix_ecto, "~> 2.0"},
{:phoenix_html, "~> 2.4"},
{:phoenix_live_reload, "~> 1.0", only: :dev},
{:gettext, "~> 0.9"},
{:comeonin, "~> 1.0"},
{:mailgun, "~> 0.1.2"},
{:poison, "~> 2.1", override: true},
{:cowboy, "~> 1.0"},
{:arc, "~> 0.5.2"},
{:ex_aws, "~> 0.4.10"},
{:httpoison, "~> 0.7"}]
end
Mix file:
def application do
[mod: {Callme, []},
applications: [:phoenix, :phoenix_html, :cowboy, :ex_aws, :httpoison, :logger, :gettext, :phoenix_ecto, :postgrex]]
end
Log when Create action is invoked successfully:
[info] POST /doctors
[debug] SELECT u0."id", u0."name", u0."email", u0."crypted_password", u0."admin", u0."inserted_at", u0."updated_at" FROM "users" AS u0 WHERE (u0."id" = $1) [26] OK query=0.0ms
[debug] Processing by Callme.DoctorController.create/2
Parameters: %{"_csrf_token" => "XxEtCRshOjg5Wj1ZLwIzWmYzPwYBJgAAmtGqxfcHa3q1mOF8WfGDLQ==", "_utf8" => "✓", "doctor" => %{"avatar" => %Plug.Upload{content_type: "image/jpeg", filename: "beta.jpg", path: "C:\\Users\\Frank\\AppData\\Local\\Temp/plug-1469/multipart-622422-206000-2"}, "bio" => "bio", "hiddeninfo" => "hiddeninfo", "name" => "name", "picture" => "picture", "specialty" => "specialty"}}
Pipelines: [:browser]
[debug] BEGIN [] OK query=0.0ms
[debug] INSERT INTO "doctors" ("inserted_at", "updated_at", "bio", "hiddeninfo", "name", "picture", "specialty") VALUES ($1, $2, $3, $4, $5, $6, $7) RETURNING "id" [{{2016, 7, 27}, {12, 27, 2, 0}}, {{2016, 7, 27}, {12, 27, 2, 0}}, "bio", "hiddeninfo", "name", "picture", "specialty"] OK query=0.0ms
[debug] COMMIT [] OK query=16.0ms
[info] Sent 302 in 32ms
[info] GET /doctors
[debug] SELECT u0."id", u0."name", u0."email", u0."crypted_password", u0."admin", u0."inserted_at", u0."updated_at" FROM "users" AS u0 WHERE (u0."id" = $1) [26] OK query=0.0ms
[debug] Processing by Callme.DoctorController.index/2
Parameters: %{}
Pipelines: [:browser]
[debug] SELECT d0."id", d0."name", d0."hiddeninfo", d0."bio", d0."specialty", d0."picture", d0."inserted_at", d0."updated_at" FROM "doctors" AS d0 [] OK query=0.0ms
I adjusted the code a little bit and now I am getting the error as follows
Request: POST /doctors
Jul 28 10:28:37 callmemd app/web.1: ** (exit) an exception was raised:
Jul 28 10:28:37 callmemd app/web.1: ** (KeyError) key :id not found in: %Plug.Upload{content_type: "image/jpeg", filename: "file000166349580.jpg", path: "/tmp/plug-1469/multipart-726916-266069-2"}
Jul 28 10:28:37 callmemd app/web.1: (callme) web/controllers/doctor_controller.ex:21: Callme.DoctorController.create/2
Jul 28 10:28:37 callmemd app/web.1: (callme) web/controllers/doctor_controller.ex:1: Callme.DoctorController.action/2
Jul 28 10:28:37 callmemd app/web.1: (callme) web/controllers/doctor_controller.ex:1: Callme.DoctorController.phoenix_controller_pipeline/2
Jul 28 10:28:37 callmemd app/web.1: (callme) lib/phoenix/router.ex:261: Callme.Router.dispatch/2
Jul 28 10:28:37 callmemd app/web.1: (callme) web/router.ex:1: Callme.Router.do_call/2
Jul 28 10:28:37 callmemd app/web.1: (callme) lib/callme/endpoint.ex:1: Callme.Endpoint.phoenix_pipeline/1
Jul 28 10:28:37 callmemd app/web.1: (callme) lib/phoenix/endpoint/render_errors.ex:34: Callme.Endpoint.call/2
One problem was the variable for the amazon ID was named incorrectly for my system variable which I changed. Now I am trying with the controller code being as follows:
def create(conn, %{"doctor" => doctor_params}) do
changeset = Doctor.changeset(%Doctor{}, doctor_params)
avatar = doctor_params["avatar"]
Callme.Avatar.store({%Plug.Upload{}, id: avatar.id})

problems with Scraping a Website with Elixir

I'm trying to get a simple hound test working with my app, I figured out its an error with selenium. This is the code:
In mix.exs:
defmodule Scraper.Mixfile do
use Mix.Project
def project do
[app: :scraper,
version: "0.0.1",
elixir: "~> 1.0",
build_embedded: Mix.env == :prod,
start_permanent: Mix.env == :prod,
deps: deps]
end
# Configuration for the OTP application
#
# Type `mix help compile.app` for more information
def application do
[applications: [:logger, :httpoison, :hound]]
end
# Dependencies can be Hex packages:
#
# {:mydep, "~> 0.3.0"}
#
# Or git/path repositories:
#
# {:mydep, git: "https://github.com/elixir-lang/mydep.git", tag: "0.1.0"}
#
# Type `mix help deps` for more examples and options
defp deps do
[
{:httpoison, "~> 0.7"},
{:floki, "~> 0.7"},
{:hound, "~> 0.7"}
]
end
end
In lib/scraper.ex
defmodule Example do
use Hound.Helpers
def run do
Hound.start_session
IO.inspect "Iniciando"
navigate_to "http://akash.im"
IO.inspect page_title()
Hound.end_session
end
end
In config/config.exs
# This file is responsible for configuring your application
# and its dependencies with the aid of the Mix.Config module.
use Mix.Config
# This configuration is loaded before any dependency and is restricted
# to this project. If another project depends on this project, this
# file won't be loaded nor affect the parent project. For this reason,
# if you want to provide default values for your application for third-
# party users, it should be done in your mix.exs file.
# Sample configuration:
#
# config :logger, :console,
# level: :info,
# format: "$date $time [$level] $metadata$message\n",
# metadata: [:user_id]
# It is also possible to import configuration files, relative to this
# directory. For example, you can emulate configuration per environment
# by uncommenting the line below and defining dev.exs, test.exs and such.
# Configuration from the imported file will override the ones defined
# here (which is why it is important to import them last).
#
# import_config "#{Mix.env}.exs"
# Define how long the application will wait between failed attempts (in miliseconds)
config :hound, retry_time: 100000
# Start with selenium driver (default)
config :hound, driver: "selenium"
Starting a webdriver server
java -jar selenium-server-standalone-2.45.0.jar
Run app:
/scraper$ iex -S mix
Erlang/OTP 18 [erts-7.1] [source] [64-bit] [smp:2:2] [async-threads:10] [hipe] [kernel-poll:false]
Interactive Elixir (1.0.5) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Example.run
** (exit) exited in: :gen_server.call(Hound.SessionServer, {:find_or_create_session, #PID<0.148.0>}, 60000)
** (EXIT) an exception was raised:
** (MatchError) no match of right hand side value: {:error, %HTTPoison.Error{id: nil, reason: :timeout}}
(hound) lib/hound/request_utils.ex:43: Hound.RequestUtils.send_req/4
(hound) lib/hound/session_server.ex:22: Hound.SessionServer.handle_call/3
(stdlib) gen_server.erl:629: :gen_server.try_handle_call/4
(stdlib) gen_server.erl:661: :gen_server.handle_msg/5
(stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
11:26:13.971 [error] GenServer Hound.SessionServer terminating
Last message: {:find_or_create_session, #PID<0.148.0>}
State: #HashDict<[]>
** (exit) an exception was raised:
** (MatchError) no match of right hand side value: {:error, %HTTPoison.Error{id: nil, reason: :timeout}}
(hound) lib/hound/request_utils.ex:43: Hound.RequestUtils.send_req/4
(hound) lib/hound/session_server.ex:22: Hound.SessionServer.handle_call/3
(stdlib) gen_server.erl:629: :gen_server.try_handle_call/4
(stdlib) gen_server.erl:661: :gen_server.handle_msg/5
(stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
(stdlib) gen_server.erl:212: :gen_server.call/3
(scraper) lib/scraper.ex:37: Example.run/0
iex(1)>
The request timed out in this case, as can be seen from the line
** (MatchError) no match of right hand side value: {:error, %HTTPoison.Error{id: nil, reason: :timeout}}
If you look at the stack trace, it indicates the error is at
(hound) lib/hound/request_utils.ex:43: Hound.RequestUtils.send_req/4
And if you open up hound source, on line 43 of lib/hound/request_utils.ex you see
case type do
:get ->
{:ok, resp} = HTTPoison.get(url, headers, #http_options)
:post ->
{:ok, resp} = HTTPoison.post(url, body, headers, #http_options)
:delete ->
{:ok, resp} = HTTPoison.delete(url, headers, #http_options)
end
This code expects a response, and crashes otherwise. There's a timeout error in your case, causing the crash.
Please check if the website up and reachable when you run the test, and retry.

vagrant provisining don't work on Windows but work on Ubuntu

I have vagrant/puppet provision script and see there is a problem when my host machine is Windows (when hosting on Ubuntu doesn't seems to have this problem). I can see that apache::concat working with apache ports.conf file have a problem. I'm using guest box same on both places (ubuntu server 14.04). As far as I know puppet is running on guest machine.
My question is why I have this warnings and errors during provisioning?
==> default: Running provisioner: puppet...
==> default: Running Puppet with default.pp...
==> default: stdin: is not a tty
==> default: Notice: Compiled catalog for eclectic.vm in environment production in 7.06 seconds
==> default: Notice: /Stage[main]/Main/Exec[apt-update]/returns: executed successfully
==> default: Notice: /Stage[main]/Main/Exec[git-checkout-drush]/returns: executed successfully
==> default: Notice: /Stage[main]/Main/File[/var/www/vhosts/eclectic.ca]/mode: mode changed '0777' to '0775'
==> default: Notice: /Stage[main]/Main/Apache::Vhost[ca.eclectic.vm]/File[/var/www/vhosts/eclectic.ca/public_html]/owner: owner changed 'www-data' to 'root'
==> default: Notice: /Stage[main]/Main/Apache::Vhost[ca.eclectic.vm]/File[/var/www/vhosts/eclectic.ca/public_html]/group: group changed 'vagrant' to 'root'
==> default: Error: /Stage[main]/Apache/Concat[/etc/apache2/ports.conf]/Exec[concat_/etc/apache2/ports.conf]: Could not evaluate: /usr/bin/env: ruby
: No such file or directory
==> default: Notice: /Stage[main]/Apache/Concat[/etc/apache2/ports.conf]/File[/etc/apache2/ports.conf]: Dependency Exec[concat_/etc/apache2/ports.conf] has failures: true
==> default: Warning: /Stage[main]/Apache/Concat[/etc/apache2/ports.conf]/File[/etc/apache2/ports.conf]: Skipping because of failed dependencies
==> default: Notice: /Stage[main]/Main/Exec[composer-install-drush]/returns: executed successfully
==> default: Error: /Stage[main]/Main/Apache::Vhost[ssl-ca.eclectic.vm]/Concat[25-ssl-ca.eclectic.vm.conf]/Exec[concat_25-ssl-ca.eclectic.vm.conf]: Could not evaluate: /usr/bin/env: ruby
: No such file or directory
==> default: Notice: /Stage[main]/Main/Apache::Vhost[ssl-ca.eclectic.vm]/Concat[25-ssl-ca.eclectic.vm.conf]/File[25-ssl-ca.eclectic.vm.conf]: Dependency Exec[concat_25-ssl-ca.eclectic.vm.conf] has failures: true
==> default: Warning: /Stage[main]/Main/Apache::Vhost[ssl-ca.eclectic.vm]/Concat[25-ssl-ca.eclectic.vm.conf]/File[25-ssl-ca.eclectic.vm.conf]: Skipping because of failed dependencies
==> default: Notice: /Stage[main]/Main/Apache::Vhost[ssl-ca.eclectic.vm]/File[25-ssl-ca.eclectic.vm.conf symlink]: Dependency Exec[concat_25-ssl-ca.eclectic.vm.conf] has failures: true
==> default: Warning: /Stage[main]/Main/Apache::Vhost[ssl-ca.eclectic.vm]/File[25-ssl-ca.eclectic.vm.conf symlink]: Skipping because of failed dependencies
==> default: Error: /Stage[main]/Main/Apache::Vhost[ca.eclectic.vm]/Concat[25-ca.eclectic.vm.conf]/Exec[concat_25-ca.eclectic.vm.conf]: Could not evaluate: /usr/bin/env: ruby
: No such file or directory
==> default: Notice: /Stage[main]/Main/Apache::Vhost[ca.eclectic.vm]/Concat[25-ca.eclectic.vm.conf]/File[25-ca.eclectic.vm.conf]: Dependency Exec[concat_25-ca.eclectic.vm.conf] has failures: true
==> default: Warning: /Stage[main]/Main/Apache::Vhost[ca.eclectic.vm]/Concat[25-ca.eclectic.vm.conf]/File[25-ca.eclectic.vm.conf]: Skipping because of failed dependencies
==> default: Notice: /Stage[main]/Main/Apache::Vhost[ca.eclectic.vm]/File[25-ca.eclectic.vm.conf symlink]: Dependency Exec[concat_25-ca.eclectic.vm.conf] has failures: true
==> default: Warning: /Stage[main]/Main/Apache::Vhost[ca.eclectic.vm]/File[25-ca.eclectic.vm.conf symlink]: Skipping because of failed dependencies
==> default: Notice: /Stage[main]/Apache::Service/Service[httpd]: Dependency Exec[concat_/etc/apache2/ports.conf] has failures: true
==> default: Notice: /Stage[main]/Apache::Service/Service[httpd]: Dependency Exec[concat_25-ca.eclectic.vm.conf] has failures: true
==> default: Notice: /Stage[main]/Apache::Service/Service[httpd]: Dependency Exec[concat_25-ssl-ca.eclectic.vm.conf] has failures: true
==> default: Warning: /Stage[main]/Apache::Service/Service[httpd]: Skipping because of failed dependencies
==> default: Notice: Finished catalog run in 18.24 seconds
My puppet file
# execute 'apt-get update'
exec { 'apt-update':
command => 'apt-get update',
path => ["/usr/bin"],
}
# Install and configure apache2
class { 'apache': # use the "apache" module
default_vhost => false, # don't use the default vhost
default_mods => false, # don't load default mods
mpm_module => 'prefork', # use the "prefork" mpm_module
service_ensure => 'running',
}
include apache::mod::php
include apache::mod::rewrite
# Enable not define apache mods
# https://forge.puppetlabs.com/puppetlabs/apache#defined-type-apachemod
apache::mod { 'access_compat': }
# http eclectic version
apache::vhost { 'com.domain.vm':
port => '80',
docadmin => '/var/www/vhosts/domain.com/public_html',
override => ['all'],
serveraliases => [
'www.com.domain.vm',
],
}
# https eclectic version
apache::vhost { 'ssl-com.domain.vm':
servername => 'com.domain.vm',
port => '443',
docadmin => '/var/www/vhosts/domain.com/public_html',
override => ['all'],
ssl => true,
serveraliases => [
'www.com.domain.vm',
],
}
# Setting up custom web eclectic direcotry
file { [
"/var/www",
"/var/www/vhosts",
"/var/www/vhosts/domain.com"
]:
ensure => "directory",
owner => "www-data",
group => "vagrant",
mode => 775,
}
# Add virtual host domain in /etc/hosts
host { 'com.domain.vm':
ensure => present,
ip => '127.0.0.1',
host_aliases => 'www.com.domain.vm',
}
# install mysql-server package
package { 'mysql-server':
require => Exec['apt-update'], # require 'apt-update' before installing
ensure => installed,
}
# ensure mysql service is running
service { 'mysql':
ensure => running,
require => Package['mysql-server'],
}
$packages = [
'vim',
'nano',
'htop',
'tree',
'pv',
'wget',
'curl',
'git',
'php5',
'php5-cli',
'php5-gd',
'php5-curl',
'php5-mysql',
'php5-mcrypt',
]
package { $packages:
require => Exec['apt-update'], # require 'apt-update' before installing
ensure => installed,
}
exec { 'set-mysql-password':
command => 'mysqladmin -u admin password "admin"',
path => ["/usr/bin"],
logoutput => true,
unless => 'mysqladmin -uadmin -padmin version',
require => Service['mysql'],
}
package { 'php-pear':
require => Exec['apt-update'], # require 'apt-update' before installing
ensure => installed,
notify => [
Exec['pear-discover-channel-phing'],
Exec['pear-install-Console_Table'],
Exec['pear-install-Console_Color2'],
]
}
# http://puppet-php.readthedocs.org/en/latest/composer.html#installation
# Install to different destination
class { 'php::composer':
destination => '/usr/bin/composer',
notify => Exec['composer-install-drush']
}
exec { "pear-discover-channel-phing":
command => "pear channel-discover pear.phing.info",
path => ["/usr/bin"],
logoutput => true,
unless => 'pear channel-info pear.phing.info',
require => Package['php-pear'],
notify => Exec['pear-install-phing'],
}
exec { "pear-install-phing":
command => "pear install phing/phing",
unless => "which phing",
path => ["/usr/bin"],
logoutput => false,
require => Exec['pear-discover-channel-phing'],
}
exec { "pear-install-Console_Table":
command => "pear install Console_Table",
unless => "pear info Console_Table",
path => ["/usr/bin"],
logoutput => true,
}
exec { "pear-install-Console_Color2":
command => "pear install Console_Color2-0.1.2",
unless => "pear info Console_Color2-0.1.2",
path => ["/usr/bin"],
logoutput => true,
}
####################################
# Install drush
exec { "git-clone-drush":
command => "git clone https://github.com/drush-ops/drush.git /usr/local/src/drush",
path => ["/usr/bin"],
unless => 'test -d /usr/local/src/drush',
notify => Exec['git-checkout-drush']
}
exec { "git-checkout-drush":
command => "git checkout 6.6.0",
cwd => "/usr/local/src/drush",
path => ["/usr/bin"],
require => Exec['git-clone-drush'],
notify => File['/usr/bin/drush']
}
file { "/usr/bin/drush":
ensure => 'link',
target => '/usr/local/src/drush/drush',
require => Exec['git-checkout-drush'],
notify => Exec['composer-install-drush']
}
exec { "composer-install-drush":
environment => [ "COMPOSER_HOME=/usr/local/src/drush" ],
command => "composer install",
cwd => "/usr/local/src/drush",
path => ["/usr/bin"],
require => File['/usr/bin/drush']
}
# Copy ssh config vagrant file
file { '/home/vagrant/.ssh/config':
ensure => present,
mode => 600,
source => "puppet:///files/ssh/config/default-config",
}
Your problem is probably the same as mine, on which I spent the last couple of days. In my case the problem was with the windows git program.
When you install git under Windows it asks whether you would like to configure CRLF<->LF conversion - there are 3 options:
default, Checkout Windows-style, commit Unix-style line endings - that means convert LF to CRLF when checking out, and convert CRLF to LF when committing,
Checkout as-is, commit Unix-style line encodings - do not convert on check-out, and convert CRLF to LF when committing,
Checkout as-is, commit as-is - do not convert at all.
At first I chose (1), so git converted LF to CRLF when checking out one of the modules, icinga-vagrant\modules\concat, and specifically the script file icinga-vagrant\modules\concat\files\concatfragments.rb. The script then would be copied (without conversion) to the target Linux virtual machine, and wouldn't run there because /usr/bin/env didn't understand what "ruby<CR>" meant, and that's why the error message had a newline in it:
... Could not evaluate: /usr/bin/env: ruby
: No such file or directory
The resolution was to reinstall git with option nr (2), although it can probably be changed without reinstallation - I was a bit lazy. Now, git checks the script as is, with only LF at line ends, and so runs without problems on Linux.
Did you include module stdlib in puppet? concat is one of functions in it. Can you show me the Puppetfile if you manage forge module by librarian-puppet
Seems the dependence is the issue. Should fix your issue after install with it.