APNS through socket on GCE stops working after a few hours - ssl

Question

Here's a interesting problem I started facing since migrating from Heroku to Google Container Engine:
Since moving to GCE, after a few hours after server start/restart/deploy, out of nowhere, my Elixir application can't deliver push notifications to APNS any longer. I'm using the apns4ex library. Here is roughly what I found out so far:
Internally on init, the library opens a :ssl (erlang) socket to APNS and keeps recycling that inside a GenServer process
def connect_socket(host, port, opts, timeout_seconds) do
address = "#{host}:#{port}"
case :ssl.connect(host, port, opts, timeout_seconds * 1000) do
{:ok, socket} ->
APNS.Logger.debug("successfully connected to #{address}")
{:ok, socket}
{:error, reason} ->
APNS.Logger.error("failed to connect to push socket #{address}, reason given: #{inspect(reason)}")
{:error, {:connection_failed, address}}
end
end
Now, from hour x, after attempting to send a message, the library starts receiving the :ssl_closed message/callback to indicate that the SSL connection got closed
def handle_info({:ssl_closed, socket}, %{socket_apple: socket} = state) do
APNS.Logger.debug("ssl socket closed, returning :connect")
{:connect, {:error, "ssl_closed"}, %{state | socket_apple: nil}}
end
How it handles this is that it just let's the connection close and returns :connect, which will then re-connect to APNS (here)
Once push notifications stop working, the debug log always reports the following pattern on every message.
Attempt to send the message
Report "success sending" (nothing is being delivered to the phones. This message is caused by :ssl.send reporting :ok)
Then receive a ssl socket close message
Reconnect to gateway.push.apple.com (:ssl.connect returns :ok)
Repeat
send_package code:
def send_package(socket, packet) do
result = :ssl.send(socket, [packet])
case result do
:ok ->
APNS.Logger.debug("success sending ssl package")
{:error, reason} ->
APNS.Logger.warn("error #{reason} sending ssl package")
end
result
end
In contrast, on successful sending it stops at point 2.
Here is some raw log output from my app when sending a push (notice the last 9 lines showing the pattern I described)
01:41:14.820 request_id=fecds3h3s1so2825c44qfestvvvpv707 [debug] [APNS] #PID<0.20135.97> 23303051:1ad798 sending in poolboy transaction :myapp
01:41:14.821 request_id=fecds3h3s1so2825c44qfestvvvpv707 [debug] [APNS] #PID<0.20135.97> 23303051:1ad798 sending message
01:41:14.821 request_id=fecds3h3s1so2825c44qfestvvvpv707 [debug] [APNS] #PID<0.20135.97> 62064556:b12e98 sending in poolboy transaction :myapp
01:41:14.821 [debug] [APNS] #PID<0.349.0> 23303051:1ad798 handling cast :send
01:41:14.821 [debug] [APNS] #PID<0.349.0> 23303051:1ad798 message's payload looks good
01:41:14.821 request_id=fecds3h3s1so2825c44qfestvvvpv707 [debug] [APNS] #PID<0.20135.97> 62064556:b12e98 sending message
01:41:14.821 request_id=fecds3h3s1so2825c44qfestvvvpv707 [debug] [APNS] #PID<0.20135.97> 19048099:b3ed8e sending in poolboy transaction :myapp
01:41:14.822 [debug] [APNS] #PID<0.349.0> success sending ssl package
01:41:14.822 [debug] [APNS] #PID<0.349.0> 23303051:1ad798 success sending
01:41:14.822 [debug] [APNS] #PID<0.349.0> 23303051:1ad798 handle call :send received :ok
01:41:14.822 [debug] [APNS] #PID<0.348.0> 62064556:b12e98 handling cast :send
01:41:14.822 [debug] [APNS] #PID<0.348.0> 62064556:b12e98 message's payload looks good
01:41:14.823 request_id=fecds3h3s1so2825c44qfestvvvpv707 [debug] [APNS] #PID<0.20135.97> 19048099:b3ed8e sending message
01:41:14.823 request_id=fecds3h3s1so2825c44qfestvvvpv707 [info] Sent 200 in 22ms
01:41:14.823 [debug] [APNS] #PID<0.348.0> success sending ssl package
01:41:14.823 [debug] [APNS] #PID<0.348.0> 62064556:b12e98 success sending
01:41:14.823 [debug] [APNS] #PID<0.348.0> 62064556:b12e98 handle call :send received :ok
01:41:14.823 [debug] [APNS] #PID<0.347.0> 19048099:b3ed8e handling cast :send
01:41:14.824 [debug] [APNS] #PID<0.347.0> 19048099:b3ed8e message's payload looks good
01:41:14.824 [debug] [APNS] #PID<0.347.0> success sending ssl package
01:41:14.824 [debug] [APNS] #PID<0.347.0> 19048099:b3ed8e success sending
01:41:14.824 [debug] [APNS] #PID<0.347.0> 19048099:b3ed8e handle call :send received :ok
01:41:15.027 [debug] [APNS] #PID<0.348.0> ssl socket closed, returning :connect
01:41:15.029 [debug] [APNS] #PID<0.347.0> ssl socket closed, returning :connect
01:41:15.043 [debug] [APNS] #PID<0.349.0> ssl socket closed, returning :connect
01:41:15.207 [debug] [APNS] #PID<0.348.0> successfully connected to gateway.push.apple.com:2195
01:41:15.207 [debug] [APNS] #PID<0.348.0> successfully connected to socket
01:41:15.209 [debug] [APNS] #PID<0.347.0> successfully connected to gateway.push.apple.com:2195
01:41:15.209 [debug] [APNS] #PID<0.347.0> successfully connected to socket
01:41:15.214 [debug] [APNS] #PID<0.349.0> successfully connected to gateway.push.apple.com:2195
01:41:15.214 [debug] [APNS] #PID<0.349.0> successfully connected to socket
One theory is that GCE is closing the connection for being idle but this doesn't explain why another message after reconnect immediately results in the same pattern. Also why does the socket only close after sending with :ssl.send?

Answer 1

I have same issue with apns4erl, when socket close after trying send message, but problem was on my side, do not remember, it was either in the wrong certificate file or malformed messages