ExAws not working on production release (Phoenix/Arc/ExAws) - amazon-s3

Question

I've been trying to create an uploader for avatars, but I've been facing some issues, the code works perfectly on dev, but on the production release, I get this error:
=SUPERVISOR REPORT==== 8-Jul-2016::22:35:41 ===
Supervisor: {local,'Elixir.ExAws.Supervisor'}
Context: child_terminated
Reason: {#{'__exception__' => true,
'__struct__' => 'Elixir.Poison.SyntaxError',
message => <<"Unexpected token: <">>,
token => <<"<">>},
[{'Elixir.Poison.Parser','parse!',2,
[{file,"lib/poison/parser.ex"},{line,56}]},
{'Elixir.Poison','decode!',2,
[{file,"lib/poison.ex"},{line,83}]},
{'Elixir.ExAws.InstanceMeta',security_credentials,1,
[{file,"lib/ex_aws/instance_meta.ex"},{line,16}]},
{'Elixir.ExAws.Config.AuthCache',refresh_config,2,
[{file,"lib/ex_aws/config/auth_cache.ex"},{line,37}]},
{'Elixir.ExAws.Config.AuthCache',handle_call,3,
[{file,"lib/ex_aws/config/auth_cache.ex"},{line,27}]},
{gen_server,try_handle_call,4,
[{file,"gen_server.erl"},{line,615}]},
{gen_server,handle_msg,5,
[{file,"gen_server.erl"},{line,647}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,247}]}]}
Offender: [{pid,<0.2009.0>},
{id,'Elixir.ExAws.Config.AuthCache'},
{mfargs,
{'Elixir.ExAws.Config.AuthCache',start_link,
[[{name,'Elixir.ExAws.Config.AuthCache'}]]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,worker}]
22:35:41.624 [error] Ranch protocol #PID<0.2011.0> (:cowboy_protocol) of listener MyApp.Endpoint.HTTP terminated
** (exit) exited in: GenServer.call(ExAws.Config.AuthCache, {:refresh_config, %ExAws.S3{config: %{access_key_id: [{:system, "AWS_ACCESS_KEY_ID"}, :instance_role], host: %{"ap-northeast-1" =>
"s3-ap-northeast-1.amazonaws.com", "ap-southeast-1" => "s3-ap-southeast-1.amazonaws.com", "ap-southeast-2" => "s3-ap-southeast-2.amazonaws.com", "eu-central-1" => "s3-eu-central-1.amazonaws.c
om", "eu-west-1" => "s3-eu-west-1.amazonaws.com", "sa-east-1" => "s3-sa-east-1.amazonaws.com", "us-east-1" => "s3.amazonaws.com", "us-west-1" => "s3-us-west-1.amazonaws.com", "us-west-2" => "
s3-us-west-2.amazonaws.com"}, http_client: ExAws.Request.HTTPoison, json_codec: Poison, region: "us-east-1", scheme: "https://", secret_access_key: [{:system, "AWS_SECRET_ACCESS_KEY"}, :insta
nce_role]}, service: :s3}}, 5000)
** (EXIT) an exception was raised:
** (Poison.SyntaxError) Unexpected token: <
(poison) lib/poison/parser.ex:56: Poison.Parser.parse!/2
(poison) lib/poison.ex:83: Poison.decode!/2
(ex_aws) lib/ex_aws/instance_meta.ex:16: ExAws.InstanceMeta.security_credentials/1
(ex_aws) lib/ex_aws/config/auth_cache.ex:37: ExAws.Config.AuthCache.refresh_config/2
(ex_aws) lib/ex_aws/config/auth_cache.ex:27: ExAws.Config.AuthCache.handle_call/3
(stdlib) gen_server.erl:615: :gen_server.try_handle_call/4
(stdlib) gen_server.erl:647: :gen_server.handle_msg/5
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Thanks.

Answer 1

Your stack trace says that Poison, a dep that ExAws uses, failed to decode a json. If you check the module that made that call (lib/ex_aws/instance_meta.ex, you can read it here), you'll find that it communicates with Amazon to check your credentials.
The most probable thing is that your configuration is incorrect and Amazon is sending you an error message formatted as an XML or HTML (both begin with <).
ExAws config depends on env vars, so that's also probably why it works on dev, but not on production.