Why do I need to define queue name while creating microservice? - rabbitmq

I want to have a hybrid NestJS app, HTTP + RabbitMQ. But don't get if I should create different microservices for each queue.
I have followed NestJS' RabbitMQ guide (https://docs.nestjs.com/microservices/rabbitmq) and GitHub example (https://github.com/nestjs/nest/tree/master/sample/03-microservices).
main.ts
app.connectMicroservice({
transport: Transport.RMQ,
options: {
urls: [`amqp://user:user#hostname:5672`],
queue: "cats_queue",
queueOptions: { durable: false },
prefetchCount: 1,
}
});
...
await app.startAllMicroservicesAsync();
app.module.ts (module import)
ClientsModule.register([{
name: "CATS_QUEUE", transport: Transport.RMQ, options: {
urls: [`amqp://user:user#hostname:5672`],
queue: "cats_queue",
queueOptions: { durable: false },
prefetchCount: 1
}
}])
app.controller.ts
constructor(
#Inject("CATS_QUEUE") private readonly client: ClientProxy
) {
}
#Get("mq")
mq(): Observable<number> {
const pattern = { cmd: "sum" };
const data = [1, 2, 3, 4, 5];
return this.client.send<number>(pattern, data);
}
#MessagePattern({ cmd: "sum" })
sum(data: number[]): number {
console.log("MESSAGE RECEIVED : " + data.toString());
return (data || []).reduce((a, b) => a + b);
}
As I understood, I need to define ClientsModule.register() for each queue in app.module.ts. But why I also need to define RabbitMQ queue name while creating microservice? Do I need to create different microservices for each queue?

I'm not a NestJS user, but it seems logical that when you use queues to send messages between your microservices, you will need a queue for each microservice (not the other way around).
Queues are used to send (TO) and receive(Process) messages between independent components asynchronously.
Does that make sense?

Related

Nest.js RabbitMQ not sending/receiving message

I have two microservices with Nest.js both of them connected to RabbitMQ service (one is the publisher and one is the receiver)
From the publisher I am trying to send a message to the receiver and seems that its not doing anything at all
Publisher :
auth.module.ts :
imports:[ClientsModule.registerAsync([
{
name: 'RMQ_SERVICE',
imports: [ConfigModule],
useFactory: (configService: ConfigService) => ({
transport: Transport.RMQ,
options: {
urls: [`amqp://${configService.get('RMQ_HOST')}:5672`],
queue: 'api-queue',
queueOptions: {
durable: false,
},
},
}),
inject: [ConfigService],
},
]),
]
auth.service.ts :
#Inject('RMQ_SERVICE') private readonly client: ClientProxy,
and using it like that :
this.client.send({ cmd: 'create-user-data' },{});
Receiver :
main.ts :
app.connectMicroservice<MicroserviceOptions>({
transport: Transport.RMQ,
options: {
noAck: false,
urls: [`amqp://${process.env.RMQ_HOST}:5672`],
queue: 'api-queue',
queueOptions: {
durable: false,
},
},
});
await app.startAllMicroservices();
users-data.controler.ts :
#MessagePattern({ cmd: 'create-user-data'})
async createUserData() {
console.log('create-user-data');
}
cant see any errors also i have rabbitmq web monitor and cannot see there any messages
any idea what wrong ?
if ill use emit and EventPattern its working i dont understand why ?
ok so after long digging i just needed to do
const result = await this.client.send(
{ cmd: 'create-user-data' },
{
userId: user.id,
provider,
...socialUser,
},
);
await result.subscribe();
and i received the message on the receiver

MassTransit consumed on single server

I am struggling to configure MassTransit with RabbitMQ to publish messages and subscribe to the queue. I think that its a simple configuration change that needs to be done. I have multiple services connected but the messages get consumed alternatively and never gets delivered / consumed on the other server.
I would like each message to get delivered to every connection / subscriber.
I am running this on ASP.net core 6 on the latest version of MassTransit.
services.TryAddSingleton(KebabCaseEndpointNameFormatter.Instance);
services.AddMassTransit(cfg =>
{
cfg.AddBus(context => Bus.Factory.CreateUsingRabbitMq(c =>
{
c.Host(connectionString, c =>
{
c.Heartbeat(10);
});
c.ConfigureEndpoints(
context,
KebabCaseEndpointNameFormatter.Instance);
c.Publish<VideoManagerResultEvent>(x =>
{
x.BindQueue("result", "video-msgs");
x.ExchangeType = Fanout;
});
c.ReceiveEndpoint("result:video-msgs", e =>
{
e.Consumer<VideoManagerResultConsumer>();
});
}));
// Request clients / DTO
RegisterRequestClients(cfg);
});
services.AddMassTransitHostedService();
}
private static void RegisterRequestClients(IServiceCollectionBusConfigurator cfg)
{
cfg.AddRequestClient<VideoManagerResultEvent>();
}
// consumer
public class VideoManagerResultConsumer : BaseConsumer<VideoManagerResultEvent>
{
public override async Task Consume(ConsumeContext<VideoManagerResultEvent> context)
{
Logger.Debug("Consumed video event");
await context.RespondAsync(new GenericResponse());
}
}
I call "SendMessage()" to publish a message to RabbitMQ.
// constructor and DI
public EventBusV2(IPublishEndpoint publishEndpoint)
{
_publishEndpoint = publishEndpoint;
}
public async Task SendMessage()
{
await _publishEndpoint.Publish<VideoManagerResultEvent>(msg);
}
To add to the original question the diagram display what is currently happening.
Diagram 1 - The request or message gets published to the eventbus but only gets delivered to the one instance of the consumer.
Diagram 2 - The required result the message gets published to both instances of the consumer.
The RaabitMQ Queue
The only configuration you should need is the following:
services.AddMassTransit(cfg =>
{
cfg.AddConsumer<VideoManagerResultConsumer>()
.Endpoint(e => e.InstanceId = "Web1"); // or 2, etc.
cfg.SetKebabCaseEndpointNameFormatter();
cfg.UsingRabbitMq((context, c) =>
{
c.Host(connectionString, c =>
{
c.Heartbeat(10);
});
c.ConfigureEndpoints(context);
});
// Request clients / DTO
RegisterRequestClients(cfg);
});
services.AddMassTransitHostedService();
Any published messages of type VideoManagerResultEvent will end up on the queue video-manager-result based upon the consumer name.

How to export data received by event in service worker to vue components?

I am building a notification system with Pusher. Currently I have a service worker registered with Pusher and I can receive "notifications" sent from my backend, but I can only show them in console:
importScripts("https://js.pusher.com/beams/service-worker.js");
PusherPushNotifications.onNotificationReceived = ({ pushEvent, payload }) => {
pushEvent.waitUntil(
self.registration.showNotification(payload.notification.title, {
body: payload.notification.body,
icon: payload.notification.icon,
data: payload.data
})
);
let notification = `Data recieved from notification ${payload.data.message}`;
console.log(notification);
};
I want to export the variable "notifications" to my vue components to manipulate the information coming from the backend.
I have tried to export, but it didn't work.
The service worker is placed in the "public" folder.
How can I do it?
Service workers communicate with a page only through messages.
function postMsg(message) {
return self.clients.matchAll().then(function(clients) {
clients.forEach(function(client) {
client.postMessage(message)
});
});
}
And then you can listen to the message inside your page :
navigator.serviceWorker.onmessage = function (evt) {
const message = evt.data
if (message.type === 'notification') {
doSomething(message)
}
}
I used Broadcast Channel to be able to send the notifications to my vue components.
Create a new instance of BroadcastChannel. Name it (in this case, the name of the Broadcast Channel is 'sw-messages') and use the method "postMessage" to send the message:
importScripts("https://js.pusher.com/beams/service-worker.js");
const channel = new BroadcastChannel('sw-messages');
PusherPushNotifications.onNotificationReceived = ({ pushEvent, payload }) => {
pushEvent.waitUntil(
self.registration.showNotification(payload.notification.title, {
body: payload.notification.body,
icon: payload.notification.icon,
data: payload.data
})
);
channel.postMessage({ title: payload.data});
};
In the vue component, I create (again) a new BroadcastChannel instance and then put an event handler, like this:
const channel = new BroadcastChannel("sw-messages");
channel.onmessage = function (event) {
this.pushNotification = event.data;
console.log(this.pushNotification.title);
}

What's the right way to publish Redis message from a Redis based NestJS Microservice

I built a sample Redis based Microservice with NestJS. It's fantastic and works great. In my microservice, after processing the message received over Redis (pub/sub), we publish our result to another Redis channel for a different microservice to pick.
What's the right way to publish? Are there any samples?
For my work, I used Redis package and published it (as opposed to ClientProxyFactory). Works fine and gets the job done.
import {
ClientProxy,
ClientProxyFactory,
Transport,
} from '#nestjs/microservices';
import { Injectable, Logger } from '#nestjs/common';
import * as redis from 'redis';
import { NVResponseDTO } from "../dto/nv.dto";
#Injectable()
export class NVPersistService {
logger = new Logger('NVPersistService');
private client: redis.RedisClient;
constructor() {
this.client = redis.createClient({port: 6379, host: 'localhost'});
this.logger.log('Successfully created client for publish');
}
async publish(result: NVResponseDTO) {
const channel = 'persistence';
try {
await this.client.publish(channel, JSON.stringify(result));
this.logger.log(`Message sent`);
} catch (e) {
this.logger.error(e);
}
}
}
But is this the way to do it or should I use something like below
this.client = ClientProxyFactory.create({
transport: Transport.REDIS,
options: {
url: 'redis://localhost:6379',
}
});
await this.client.connect();
const channel = 'persistence';
const status = await this.client.send<string, NVResponseDTO>(channel, result);
this.logger.log(`Status of send - ${status}`);
Note: Above code did not work for me, hence used Redis directly. Any guidance would be much appreciated

Node.js + socket.io + node-amqp and queue binginds when "re" connecting thru socket.io

I have one scenario which is very close to this sample:
One main screen:
this screen (client side) will connect to the socket.io server thru server:9090/scope (io.connect("http://server:9090/scope)) and will send one event "userBindOk" (socket.emit("userBindOk", message)) to the socket.io server;
the server receives the connection and the "userBindOk". At this moment, the server should get the active connection to rabbitmq server and bind the queue to the respective user that just connected to the application thru socket.io. sample:
socket.on("connection", function(client){
//client id is 1234
// bind rabbitmq exchange, queue, and:
queue.subscribe(//receive callback);
})
So far, no problem - I can send/receive messages thru socket.io without problems.
BUT, If I refresh the page, all those steps will be done again. As consequence, the binding to the queue will occur, but this time related to another session of the socket.io client. This means that if I send a message to the queue which is related to the first socket.io session (before the page refresh), that bind should (I think) receive the message and send it to a invalid socket.io client (page refresh = new client.id on the socket.io context). I can prove this behaviour because every time I refresh the page I need to send x times more messages. For instance: I`ve connected for the first time: - so, 1 message - one screen update; refresh the page: I need to send 2 messages to the queue and only the second message will be received from the "actual" socket.io client session - this behaviour will occur as many as I refresh the page (20 page refreshs, 20 messages to be sent to a queue and the server socket.io "last" client will send the message to the client socket.io to render into the screen).
The solutions I believe are:
Find a way to "unbind" the queue when disconnecting from the socket.io server - I didn`t see this option at the node-amqp api yet (waiting for it :D)
find a way to reconnect the socket.io client using the same client.id. This way I can identify the client that is coming and apply some logic to cache the socket.
Any ideas? I tried to be very clear... But, as you know, it`s not so eaey to expose your problem when trying to clarify something that is very specific to some context...
tks
I solved it like this:
I used to declare the rabbitMq queue as durable=true,autoDelete=false,exclusive=false and in my app there was 1 queue/user and 1 exchange(type=direct) with the routing_key name=queueName, my app also used the queue for other client diffent to browsers like android app or iphone app as push fallback, so i use to crear 1 queue for earch user.
The solution to this problem was to change my rabbitMQ queue and exchange declaration. Now i declare the exchange/user as fanout and autoDelete=True, and the user is going to have N queues with durable=true, autoDelete=true, exclusive=true (No. queue = No. clients) and all the queues are bind to the user-exchange(multicast).
NOTE: my app is wirten in django, and i use node+socket+amqp to be able to comunicate with the browser using web.scokets, so i use node-restler to query my app api to get the user-queue info.
thats the rabbitMQ side, for the node+amqp+socket i did this:
server-side:
onConnect: the declaration of the user exchange as fanout, autoDelete, durable. then declaration of the queue as durable, autodelete and exclusive, then the queue.bind to the user-exchange and finaly the queue.subscribe and the socket.disconnect will destroy the queue so there are going to exist queue as client connected the app and this solve the problem of the refresh and allow the user to have more than 1 window-tab with the app:
Server-side:
/*
* unCaught exception handler
*/
process.on('uncaughtException', function (err) {
sys.p('Caught exception: ' + err);
global.connection.end();
});
/*
* Requiere libraries
*/
global.sys = require('sys');
global.amqp = require('amqp');
var rest = require('restler');
var io = require('socket.io').listen(8080);
/*
* Module global variables
*/
global.amqpReady = 0;
/*
* RabbitMQ connection
*/
global.connection = global.amqp.createConnection({
host: host,
login: adminuser,
password: adminpassword,
vhost: vhost
});
global.connection.addListener('ready',
function () {
sys.p("RabbitMQ connection stablished");
global.amqpReady = 1;
}
);
/*
* Web-Socket declaration
*/
io.sockets.on('connection', function (socket) {
socket.on('message', function (data) {
sys.p(data);
try{
var message = JSON.parse(data);
}catch(error){
socket.emit("message", JSON.stringify({"error": "invalid_params", "code": 400}));
var message = {};
}
var message = JSON.parse(data);
if(message.token != undefined) {
rest.get("http://dev.kinkajougames.com/api/push",
{headers:
{
"x-geochat-auth-token": message.token
}
}).on('complete',
function(data) {
a = data;
}).on('success',
function (data){
sys.p(data);
try{
sys.p("---- creating exchange");
socket.exchange = global.connection.exchange(data.data.bind, {type: 'fanout', durable: true, autoDelete: true});
sys.p("---- declarando queue");
socket.q = global.connection.queue(data.data.queue, {durable: true, autoDelete: true, exclusive: false},
function (){
sys.p("---- bind queue to exchange");
//socket.q.bind(socket.exchange, "*");
socket.q.bind(socket.exchange, "*");
sys.p("---- subscribing queue exchange");
socket.q.subscribe(function (message) {
socket.emit("message", message.data.toString());
});
}
);
}catch(err){
sys.p("Imposible to connection to rabbitMQ-server");
}
}).on('error', function (data){
a = {
data: data,
};
}).on('400', function() {
socket.emit("message", JSON.stringify({"error": "connection_error", "code": 400}));
}).on('401', function() {
socket.emit("message", JSON.stringify({"error": "invalid_token", "code": 401}));
});
}
else {
socket.emit("message", JSON.stringify({"error": "invalid_token", "code": 401}));
}
});
socket.on('disconnect', function () {
socket.q.destroy();
sys.p("closing socket");
});
});
client-side:
The socket intance with options 'force new connection'=true and 'sync disconnect on unload'= false.
The client side use the onbeforeunload and onunload windows object events to send socket.disconnect
The client on socket.connect event send the user token to node.
proces message from socket
var socket;
function webSocket(){
//var socket = new io.Socket();
socket = io.connect("ws.dev.kinkajougames.com", {'force new connection':true, 'sync disconnect on unload': false});
//socket.connect();
onSocketConnect = function(){
alert('Connected');
socket.send(JSON.stringify({
token: Get_Cookie('liveScoopToken')
}));
};
socket.on('connect', onSocketConnect);
socket.on('message', function(data){
message = JSON.parse(data);
if (message.action == "chat") {
if (idList[message.data.sender] != undefined) {
chatboxManager.dispatch(message.data.sender, {
first_name: message.data.sender
}, message.data.message);
}
else {
var username = message.data.sender;
Data.Collections.Chats.add({
id: username,
title: username,
user: username,
desc: "Chat",
first_name: username,
last_name: ""
});
idList[message.data.sender] = message.data.sender;
chatboxManager.addBox(message.data.sender, {
title: username,
user: username,
desc: "Chat",
first_name: username,
last_name: "",
boxClosed: function(id){
alert("closing");
}
});
chatboxManager.dispatch(message.data.sender, {
first_name: message.data.sender
}, message.data.message);
}
}
});
}
webSocket();
window.onbeforeunload = function() {
return "You have made unsaved changes. Would you still like to leave this page?";
}
window.onunload = function (){
socket.disconnect();
}
And that's it, so no more round-robing of the message.