How to setup Phoenix PubSub subscriber callbacks - redis

I've got a fairly straightforward requirement centered around 2 services (for now) built in Phoenix:
ServiceA is responsible for registering users. When a user is registered, ServiceA broadcasts a message with info about the newly created user. This is being done using the following code in a Controller action right now:
ServiceA.Endpoint.broadcast("activity:all", "new:user", %{email: "test#test.com"})
ServiceB is responsible for listening out for all of these activity broadcasts and doing something with them (essentially building up a feed of activity).
I've hit a stumbling block in that I can see ServiceA broadcasting the message to Redis (using Phoenix.PubSub.Redis), but don't fully understand how to get the subscriber on ServiceB to process it...
The following piece of code is as far as I've managed to get, which does something when a message is broadcast and then raises an exception.
Partial Subscriber Module
defmodule ServiceB.UserSubscriber do
def start_link do
sub = spawn_link &(process_feed/0)
ServiceB.Endpoint.subscribe(:user_pubsub, "activity:all")
{:ok, sub}
end
def process_feed do
receive do
params ->
IO.inspect "processing goes here..."
end
process_feed
end
end
Exception
[error] GenServer :user_pubsub terminating
** (FunctionClauseError) no function clause matching in Phoenix.PubSub.RedisServer.handle_info/2
I'm guessing I've missed a whole load of GenServer work somewhere, but can't seem to find anything online that suggests where.

The problem (as expected) was that my Subscriber module wasn't implemented as a GenServer but I was trying to replicate the same functionality (and badly!). Updating my Subscriber Model as follows has done the trick:
defmodule SubscriberService.ActivitySubscriber do
use GenServer
def start_link(channel) do
GenServer.start_link(__MODULE__, channel)
end
def init(channel) do
pid = self
ref = SubscriberService.Endpoint.subscribe(pid, channel)
{:ok, {pid, channel, ref}}
end
def handle_info(%{event: "new:user"} = message, state) do
IO.inspect "#######################"
IO.inspect "New User - Received Message:"
IO.inspect message
IO.inspect "#######################"
{:noreply, state}
end
def handle_info(message, state) do
IO.inspect "#######################"
IO.inspect "Catch All - Received Message:"
IO.inspect message
IO.inspect "#######################"
{:noreply, state}
end
end
As you can see, init/1 triggers the subscription, and the handle_info/2 functions receive the incoming messages.
If you want to see how it works in all its glory (both Publisher and Subscriber services), take a look at the repo.

Related

SOLVED Handle multiple AMQP Pattern inside one Rails Service using Bunny gem

SOLVED PROBLEM.
Just create 2 different queues, like rpc.queue and pubsub.queue. Then you can use multiple messaging pattern in one service without any problem.
I create one rails service using Bunny and ConnectionPool Gem. This service "in my mind (cause not yet implemented)" handle multiple RMQ pattern such as Direct Messaging and RPC. These patterns initialized with different object of Connection Class and defined inside initalizer folder.
Initializer looks like this:
# RMQ Initializer for RabbitMQ Connection
class RMQ
include Lontara::RMQ
# NOTE: Call 2 Server caused errors
def self.start(url:, queue:, rpc_exchange:, pubsub_exchange:)
# Then start the consumer and subscriber
Server::RPCConsumer.new(Connection.new(url:), queue:, exchange: rpc_exchange).consume
Server::Subscriber.new(Connection.new(url:), queue:, exchange: pubsub_exchange).subscribe
end
end
RMQ.start(
url: ENV.fetch('RABBITMQ_URL', 'amqp://guest:guest#rmqserver:5672'),
queue: ENV.fetch('RABBITMQ_QUEUE_VOUCHER', 'lontara-dev.voucher'),
rpc_exchange: ENV.fetch('RABBITMQ_EXCHANGE_RPC', 'lontara-dev.rpc'),
pubsub_exchange: ENV.fetch('RABBITMQ_EXCHANGE_PUBSUB', 'lontara-dev.pubsub')
)
and Connection class:
module Lontara
module RMQ
# Class Connection initializing the connection to RabbitMQ.
class Connection
def initialize(url: ENV['RABBITMQ_URL'])
#connection = Bunny.new(url)
connection.start
#channel = channel_pool.with(&:create_channel)
yield self if block_given?
end
def close
channel.close
connection.close
end
attr_reader :connection, :channel
private
def channel_pool
#channel_pool ||= ConnectionPool.new { #connection }
end
end
end
end
The problem goes whenever these 2 Server:: (RPC and Subscriber) activated. Impacted only when use RPC as messaging, the problem is RPC Publisher does not get response from Consumer.
These steps (when RPC produce error) are:
Run Rails server
Open new terminal, and open rails console in same project
Create Request to Consumer using RPCPublisher
Publisher get response. Then send request again... On this step not get response.
Job is pending, i push ctrl+c to terminate job. Send request again, and get response...
Try again like step 4, and error...
But, if Server::Publisher not initialized on initializer, nothing error happened.
I assumed this error happened cause of thread... But i don't really get helped from other articles on internet.
My expectation is so simple:
RPC Connection requested for Get related (because RPC can reply this request) or other action requires response. And Pub/Sub (Direct) request for Create, Update, Delete since this type didn't need it.
Your answer really help me... Thankyou !

Consolidate response of API's though cast method of Genserver

I am trying to call one API 10 times asynchronously though cast method of Genserver. Can someone guide me how I can collect the responses of 10 API and consolidate in one list of tuples?
defmodule DataMonitor.RuleReceiver do
use GenServer
require Logger
alias DataMonitor.{
ProcessRuleSet
}
def init(state) do
{:ok, state}
end
def start_link do
GenServer.start_link(__MODULE__, [])
end
def process_rule_set(receiver_pid, {rule_set, company_id, auth_headers}) do
GenServer.cast(receiver_pid, {rule_set, company_id, auth_headers})
end
def handle_cast({rule_set, company_id, auth_headers}, state) do
result = ProcessRuleSet.process_rule_set(rule_set, company_id, auth_headers)
{:noreply, state}
end
end
Sender/Caller module
defmodule DataMonitor.RuleSender do
def perform() do
Enum.each(rule_sets, fn rule_set ->
{:ok, pid} = RuleReceiver.start_link()
RuleReceiver.process_rule_set(pid, {rule_set, company_id, auth_headers})
end)
end
end
though GenServer.cast/2 [...] how I can collect the responses
You cannot per se. Cast requests are asynchronous and return no response. The only way to collect the data from casts, would be to introduce a store (like an Agent, or ets, or whatever,) and store the values directly from casts. This solution has the obvious drawback: the workflow of it would be undetermined, one might not assume all the 10 responses are processed and stored at any time. That said, casts under some circumstances might even be lost and unprocessed and you have no chance to get notified about that. I have never met such a case, but it’s considered to be legit.
So, in this particular case, you probably should just use GenServer.call/2 instead of cast, and collect responses directly in the iteration with Enum.map/2:
def perform() do
Enum.map(rule_sets, fn rule_set ->
{:ok, pid} = RuleReceiver.start_link()
{pid, RuleReceiver.process_rule_set_call(...)}
end)
end
As the result you’ll have 10 tuples {pid, response}. I am not sure what is the reason to create 10 GenServers here and why would not you use named GenServers to not bother with pids, but this is obviously out of scope here.

RabbitMQ routing key does not route

i'm trying to do a simple message queue with RabitMQ i push a message with create_message
and then trying to get the message by the routing key.
it works great when the routing key is the same. the problem is when the routing key is different i keep on getting the message with the wrong routing key:
for example
def callback(ch, method, properties, body):
print("%r:%r" % (method.routing_key, body))
def create_message(self):
connection = pika.BlockingConnection(pika.ConnectionParameters(
'localhost'))
channel = connection.channel()
channel.exchange_declare(exchange='www')
channel.queue_declare(queue='hello')
channel.basic_publish(exchange='www',
routing_key="11",
body='Hello World1111!')
connection.close()
self.get_analysis_task_celery()
def get_message(self):
connection = pika.BlockingConnection(pika.ConnectionParameters(
'localhost'))
channel = connection.channel()
channel.exchange_declare(exchange='www')
timeout = 1
connection.add_timeout(timeout, on_timeout)
channel.queue_bind(exchange="www", queue="hello", routing_key="10")
channel.basic_consume(callback,
queue='hello',
no_ack=True,
consumer_tag= "11")
channel.start_consuming()
example for my output: '11':'Hello World1111!'
what am i doing wrong?
tnx for the help
this is a total guess, since i can't see your rabbitmq server..
if you open the RabbitMQ management website and look at your exchange, you will probably see that the exchange is bound to the queue for routing key 10 and 11, both of which are bound to the same queue.
since both go to the same queue, your message will always be delivered to that queue, the consumer will always pick up the message
again, i'm guessing since i can't see your server. but check the server to make sure you don't have leftover / extra bindings

Can a process have two receive blocks

My working environment is Erlang.
Can a process having 2 different functions have two receive blocks in the different functions.
receive
....
end.
request()
PID!(message)
%% Can i get the reply back here instead of the receive block above?
Yes, you can have many receive expressions. When one is evaluated it will take the first matching message out of the message queue/mailbox (take your pick of names) and leave the rest for the next receive. Sending a message, using the Pid ! Message syntax (the only way), is completely asynchronous and just adds the message to the end of the receiving processes message queue. receive is the only way to receive messages i.e. take them out of the message queue. You can never put them back.
There is no built-in synchronous message passing in Erlang, it is one by sending two messages:
The "requesting" process sends a message to the receiving process and then goes into a receive to wait for the reply.
The "receiving" process will itself get the message in a receive, process it, send the reply message back the the "requesting" process and then go into a receive to sit and wait for the next message.
Remember there is no inherent connection between processes, ever, and all communication is done using asynchronous message sending and receive.
So in reply to your second question: you can ONLY get a reply back in a receive expression. That is the only way!
Sorry to be a bit pedantic but Erlang doesn't have either blocks or statements. It is a functional language and only has expressions which always return a value even if the return value is sometimes ignored.
Of couse, you can.
Receives messages sent to the process using the send operator (!). The
patterns Pattern are sequentially matched against the first message in
time order in the mailbox, then the second, and so on. If a match
succeeds and the optional guard sequence GuardSeq is true, the
corresponding Body is evaluated. The matching message is consumed,
that is removed from the mailbox, while any other messages in the
mailbox remain unchanged.
The following code is from supervisor2.erl of rabbitmq project. It even uses nested receive statement. I have marked nested receive below.
terminate_simple_children(Child, Dynamics, SupName) ->
Pids = dict:fold(fun (Pid, _Args, Pids) ->
erlang:monitor(process, Pid),
unlink(Pid),
exit(Pid, child_exit_reason(Child)),
[Pid | Pids]
end, [], Dynamics),
TimeoutMsg = {timeout, make_ref()},
TRef = timeout_start(Child, TimeoutMsg),
{Replies, Timedout} =
lists:foldl(
fun (_Pid, {Replies, Timedout}) ->
{Reply, Timedout1} =
receive %% attention here
TimeoutMsg ->
Remaining = Pids -- [P || {P, _} <- Replies],
[exit(P, kill) || P <- Remaining],
receive {'DOWN', _MRef, process, Pid, Reason} -> %%attention here
{{error, Reason}, true}
end;
{'DOWN', _MRef, process, Pid, Reason} ->
{child_res(Child, Reason, Timedout), Timedout};
{'EXIT', Pid, Reason} ->
receive {'DOWN', _MRef, process, Pid, _} ->
{{error, Reason}, Timedout}
end
end,
{[{Pid, Reply} | Replies], Timedout1}
end, {[], false}, Pids),
timeout_stop(Child, TRef, TimeoutMsg, Timedout),
ReportError = shutdown_error_reporter(SupName),
[case Reply of
{_Pid, ok} -> ok;
{Pid, {error, R}} -> ReportError(R, Child#child{pid = Pid})
end || Reply <- Replies],
ok.

How do TLS connections in EventMachine work?

I have a custom Protobuf-based protocol that I've implemented as an EventMachine protocol and I'd like to use it over a secure connection between the server and clients. Each time I send a message from a client to the server, I prepend the message with a 4-byte integer representing the size of the Protobuf serialized string to be sent such that the server knows how many bytes to read off the wire before parsing the data back into a Protobuf message.
I'm calling start_tls in the post_init callback method in both the client and server protocol handlers, with the one in the server handler being passed the server's private key and certificate. There seems to be no errors happening at this stage, based on log messages I'm printing out.
Where I get into trouble is when I begin parsing data in the receive_data callback in the server's handler code... I read 4 bytes of data off the wire and unpack it to an integer, but the integer that gets unpacked is not the same integer I send from the client (i.e. I'm sending 17, but receiving 134222349).
Note that this does not happen when I don't use TLS... everything works fine if I remove the start_tls calls in both the client and server code.
Is it the case that SSL/TLS data gets passed to the receive_data callback when TLS is used? If so, how do I know when data from the client begins? I can't seem to find any example code that discusses this use case...
OK, so via a cross-post to the EventMachine Google Group I figured out what my problem was here. Essentially, I was trying to send data from the client to the server before the TLS handshake was done because I wasn't waiting until the ssl_handshake_completed callback was called.
Here's the code I got to work, just in case anyone comes across this post in the future. :)
Handler code for the server-side:
require 'eventmachine'
class ServerHandler < EM::Connection
def post_init
start_tls :private_key_file => 'server.key', :cert_chain_file => 'server.crt', :verify_peer => false
end
def receive_data(data)
puts "Received data in server: #{data}"
send_data(data)
end
end
Handler code for the client-side:
require 'eventmachine'
class ClientHandler < EM::Connection
def connection_completed
start_tls
end
def receive_data(data)
puts "Received data in client: #{data}"
end
def ssl_handshake_completed
send_data('Hello World! - 12345')
end
end
Code to start server:
EventMachine.run do
puts 'Starting server...'
EventMachine.start_server('127.0.0.1', 45123, ServerHandler)
end
Code to start client:
EventMachine.run do
puts 'Starting client...'
EventMachine.connect('127.0.0.1', 45123, ClientHandler)
end