I have a GRPC Service using protobuf-net.Grpc where I use Server Streaming to send messages back to the client. I'm using Azure App Services, so I thought I would use an Azure Service Bus and Mass Transit to wait for messages/events and send them back to the client.
public async IAsyncEnumerable<TestResult> SubscribeAsync([EnumeratorCancellation] CancellationToken cancel)
{
while (!cancel.IsCancellationRequested)
{
// wait for new mass transit message
yield return new TestResult { Result = MessageContent };
}
}
Is something like this even possible using mass transit, or would I need some kind of observable queue.
Related
I am trying to build a group chat message using WebSocket using spring-webflux and rector-netty. I am new to reactor-netty framework and even after reading multiple articles posts I couldn't figure out if it is possible to throttle a client from sending too many messages in reactor-netty framework.
public class ServerWebSocketHandler implements WebSocketHandler {
Map<String, WebSocketSession> sessions = new HashMap<>();
ConcurrentLinkedQueue<String> messages = new ConcurrentLinkedQueue<>();
public ServerWebSocketHandler() {
// logic to start a thread which will drain all the messages in the queue to all the sessions periodically
}
#Override
public Mono<Void> handle(WebSocketSession session) {
System.out.println("Client connected: " + session);
sessions.put(session.getId(), session);
Flux<String> stringFlux = session.receive()
.map(WebSocketMessage::getPayloadAsText)
.map(String::toLowerCase)
.doOnNext(m -> messages.offer(m))
.doFinally(m -> System.out.printf("Client %s Disconnected due to %s\n", session, m));
return stringFlux.then();
}
}
ReactorNettyWebSocketSession is the implementation which is used in this case and it doesn't seem to expose any methods to have any control over the inbound/outbound, like marking the inbound as not readable or something. Is it possible to throttle/block a client from sending too many messages. If it is not possible, I am thinking that creating a bounded queue for each session and receive and then ignore/drop the incoming message in application layer.
I have a websocket implementation using redis messaging operation on webflux. And what it does is it listens to topic and returns the values via websocket endpoint.
The problem I have is each time a user sends a message via websocket to the endpoint it seems a brand new redis subscription is made, resulting in the accumulation of subscribers on the redis message topic and the websocket responses are increased with the number of redis topic message subscribtions as well (example user sends 3 messages, redis topic subscriptions are increased to three, websocket connection responses three times).
Would like to know if there is a way to reuse the same subscription to the messaging topic so it would prevent multiple redis topic subscriptions.
The code I use is as follows:
Websocket Handler
public class SendingMessageHandler implements WebSocketHandler {
private final Gson gson = new Gson();
private final MessagingService messagingService;
public SendingMessageHandler(MessagingService messagingService) {
this.messagingService = messagingService;
}
#Override
public Mono<Void> handle(WebSocketSession session) {
Flux<WebSocketMessage> stringFlux = session.receive()
.map(WebSocketMessage::getPayloadAsText)
.flatMap(inputData ->
messagingService.playGame(inputData)
.map(data ->
session.textMessage(gson.toJson(data))
)
);
return session.send(stringFlux);
}
}
Message Handling service
public class MessagingService{
private final ReactiveRedisOperations<String, GamePubSub> reactiveRedisOperations;
public MessagingService(ReactiveRedisOperations<String, GamePubSub> reactiveRedisOperations) {
this.reactiveRedisOperations = reactiveRedisOperations;
}
public Flux<Object> playGame(UserInput userInput){
return reactiveRedisOperations.listenTo("TOPIC_NAME");
}
}
Thank you in advance.
Instead of using ReactiveRedisOperations, MessageListener is the way to go here. You can register a listener once, and use the following as the listener.
data -> session.textMessage(gson.toJson(data))
The registration should happen only once at the beginning of the connection. You can override void afterConnectionEstablished(WebSocketSession session) of SendingMessageHandler to accomplish this. That way a new subscription created per every new Websocket connection, per every message.
Also, don't forget to override afterConnectionClosed, and unsubscribe from the redis topic, and clean up the listener within it.
Instructions on how to use MessageListener.
I'm working with 2 .NET Core console applications in a producer/consumer scenario with MassTransit/RabbitMQ. I need to ensure that even if NO consumers are up-and-running, the messages from the producer are still queued up successfully. That didn't seem to work with Publish() - the messages just disappeared, so I'm using Send() instead. The messages at least get queued up, but without any consumers running the messages all end up in the "_skipped" queue.
So that's my first question: is this the right approach based on the requirement (even if NO consumers are up-and-running, the messages from the producer are still queued up successfully)?
With Send(), my consumer does indeed work, but still many messages are falling through the cracks and getting dumped into to the "_skipped" queue. The consumer's logic is minimal (just logging the message at the moment) so it's not a long-running process.
So that's my second question: why are so many messages still getting dumped into the "_skipped" queue?
And that leads into my third question: does this mean my consumer needs to listen to the "_skipped" queue as well?
I am unsure what code you need to see for this question, but here's a screenshot from the RabbitMQ management UI:
Producer configuration:
static IHostBuilder CreateHostBuilder(string[] args)
{
return Host.CreateDefaultBuilder()
.ConfigureServices((hostContext, services) =>
{
services.Configure<ApplicationConfiguration>(hostContext.Configuration.GetSection(nameof(ApplicationConfiguration)));
services.AddMassTransit(cfg =>
{
cfg.AddBus(ConfigureBus);
});
services.AddHostedService<CardMessageProducer>();
})
.UseConsoleLifetime()
.UseSerilog();
}
static IBusControl ConfigureBus(IServiceProvider provider)
{
var options = provider.GetRequiredService<IOptions<ApplicationConfiguration>>().Value;
return Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri(options.RabbitMQ_ConnectionString), h =>
{
h.Username(options.RabbitMQ_Username);
h.Password(options.RabbitMQ_Password);
});
cfg.ReceiveEndpoint(host, typeof(CardMessage).FullName, e =>
{
EndpointConvention.Map<CardMessage>(e.InputAddress);
});
});
}
Producer code:
Bus.Send(message);
Consumer configuration:
static IHostBuilder CreateHostBuilder(string[] args)
{
return Host.CreateDefaultBuilder()
.ConfigureServices((hostContext, services) =>
{
services.AddSingleton<CardMessageConsumer>();
services.Configure<ApplicationConfiguration>(hostContext.Configuration.GetSection(nameof(ApplicationConfiguration)));
services.AddMassTransit(cfg =>
{
cfg.AddBus(ConfigureBus);
});
services.AddHostedService<MassTransitHostedService>();
})
.UseConsoleLifetime()
.UseSerilog();
}
static IBusControl ConfigureBus(IServiceProvider provider)
{
var options = provider.GetRequiredService<IOptions<ApplicationConfiguration>>().Value;
return Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri(options.RabbitMQ_ConnectionString), h =>
{
h.Username(options.RabbitMQ_Username);
h.Password(options.RabbitMQ_Password);
});
cfg.ReceiveEndpoint(host, typeof(CardMessage).FullName, e =>
{
e.Consumer<CardMessageConsumer>(provider);
});
//cfg.ReceiveEndpoint(host, typeof(CardMessage).FullName + "_skipped", e =>
//{
// e.Consumer<CardMessageConsumer>(provider);
//});
});
}
Consumer code:
class CardMessageConsumer : IConsumer<CardMessage>
{
private readonly ILogger<CardMessageConsumer> logger;
private readonly ApplicationConfiguration configuration;
private long counter;
public CardMessageConsumer(ILogger<CardMessageConsumer> logger, IOptions<ApplicationConfiguration> options)
{
this.logger = logger;
this.configuration = options.Value;
}
public async Task Consume(ConsumeContext<CardMessage> context)
{
this.counter++;
this.logger.LogTrace($"Message #{this.counter} consumed: {context.Message}");
}
}
In MassTransit, the _skipped queue is the implementation of the dead letter queue concept. Messages get there because they don't get consumed.
MassTransit with RMQ always delivers a message to an exchange, not to a queue. By default, each MassTransit endpoint creates (if there's no existing queue) a queue with the endpoint name, an exchange with the same name and binds them together. When the application has a configured consumer (or handler), an exchange for that message type (using the message type as the exchange name) also gets created and the endpoint exchange gets bound to the message type exchange. So, when you use Publish, the message is published to the message type exchange and gets delivered accordingly, using the endpoint binding (or multiple bindings). When you use Send, the message type exchange is not being used, so the message gets directly to the destination exchange. And, as #maldworth correctly stated, every MassTransit endpoint only expects to get messages that it can consume. If it doesn't know how to consume the message - the message is moved to the dead letter queue. This, as well as the poison message queue, are fundamental patterns of messaging.
If you need messages to queue up to be consumed later, the best way is to have the wiring set up, but the endpoint itself (I mean the application) should not be running. As soon as the application starts, it will consume all queued messages.
When the consumer starts the bus bus.Start(), one of the things it does is create all exchanges and queues for the transport. If you have a requirement that publish/send happens before the consumer, your only option is to run DeployTopologyOnly. Unfortunately this feature is not documented in official docs, but the unit tests are here: https://github.com/MassTransit/MassTransit/blob/develop/src/MassTransit.RabbitMqTransport.Tests/BuildTopology_Specs.cs
The skipped queue happens when messages are sent to a consumer that doesn't know how to process.
For example if you have a consumer that can process IConsumer<MyMessageA> which is on receive endpoint name "my-queue-a". But then your message producer does Send<MyMessageB>(Uri("my-queue-a")...), Well this is a problem. The consumer only understands the A, it doesn't know how to process B. And so it just moves it to a skipped queue and continues on.
In my case, the same queue listens to multiple consumers at the same time
I use Masstransit in C# project.
I have a publisher and consumer services, and when both of them are up, then there are no problems. But if the consumer goes offline, published messages don't go to the queue. They just disappear.
The expected behavior is to keep messages in the queue until the consumer is started, and then send them to it. I've found several topics in google groups with same questions, but it wasn't clear for me how to solve that problem.
It seems strange to me that this functionality isn't provided out of the box because, in my understanding, it is the main purpose of RabbitMQ and MT.
The way I create publisher bus:
public static IBusControl CreateBus()
{
return Bus.Factory.CreateUsingRabbitMq(sbc =>
{
var host = sbc.Host(new Uri("rabbitmq://RMQ-TEST"), h =>
{
h.Username("test");
h.Password("test");
});
sbc.ReceiveEndpoint(host, "test_queue", ep =>
{
ep.Handler<IProductDescriptionChangedEvent>(
content => content.CompleteTask);
});
});
}
And the consumer:
public static void StartRmqBus()
{
var bus = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri("rabbitmq://RMQ-TEST"), h =>
{
h.Username("test");
h.Password("test");
});
cfg.ReceiveEndpoint(host, "test_queue", ep =>
{
ep.Consumer<ProductChangedConsumer>();
});
});
bus.Start();
}
EDIT:
Here is one more interesting feature: if I stop both services and manually put a message to the queue via admin interface of MT, the message is waiting in test_queue. But when I start publisher or consumer service, it falls to test_queue_error queue.
You use the same queue for published and consumer, plus publisher has a consumer for this message type, as you pointed out in your own answer.
If your publisher does not consume messages, it is better to remove the receiving endpoint from it at all and then your service will be send-only.
If you have several services, where each of them need to have their own consumers for the same message type - this is how pub-sub works and you must have different queues per service. This is described in the Common Gotchas section of the documentation. In such scenario, each service will get it's own copy of the published message.
If you have one queue - you get competing consumers and this scenario is only valid for horizontal scalability, where you run several instance of the same services to increase the number of processed messages if the processing is too slow. In such case all these instances will consume messages from the same queue. In this scenario only one instance will get a message.
It seems like my publisher was set up incorrectly. After removing this part:
sbc.ReceiveEndpoint(host, "test_queue", ep =>
{
ep.Handler<IProductDescriptionChangedEvent>(
content => content.CompleteTask);
});
it started to work as expected. Looks like it consumed its own messages, that's why I didn't see messages in the queue when the consumer was down.
I have a WCF Service Using MSMQ hosted on IIS. I want to create a windows application which can stop WCF Service from picking MSMQ message. Once I have seen the MSMQ message in the queue I need to click a button and Start the WCF service to pick the message in MSMQ. Code sample would be apperciated.
IIS is not an appropriate container to host a MSMQ client in. This is because when the app pool unloads during times of low traffic the queue client also unloads. This behaviour is automatic and you don't have any control over it.
It would be far better to host your client in a windows service. However, the kind of "consume-on-demand" functionality you require is not easy to achieve and certainly is not supported by the standard bindings.
The best I can suggest is consume the message as soon as it's received and persist it somewhere until the user clicks the button, upon which you do whatever you want as the data in the message is already available.
I was able to solve this problem by applying a workaround. I created another queue in a different machine. Changed the address of the WCF client endpoint address to this queue in config. I created another external application which moved the message from the alternate queue to the actual queue. Thus the behavior of stopping IIS hosted WCF service with MSMQ binding was achieved
Stopping the "Net.Msmq Listener Adapter" Windows service and the "Windows Process Activation Service" will stop the messages from being pulled out of the queue. Starting the services back up will causes the messages to be pulled from the queue again. I'm doing this manually, rather than through another application, but I'd assume you could do it through another application as well. I haven't tested this completely, but something like this would probably work:
Dictionary<string,List<string>> runningDependentServices = new Dictionary<string,List<string>>();
private void StartMsmqBinding()
{
StartService("WAS");
StartService("NetMsmqActivator");
}
private void StopMsmqBinding()
{
StopService("NetMsmqActivator");
StopService("WAS");
}
private void StartService(string serviceName)
{
List<string> previouslyRunningServices = null;
var sc = new ServiceController();
sc.ServiceName = serviceName;
if (runningDependentServices.ContainsKey(serviceName))
{
previouslyRunningServices = runningDependentServices[serviceName];
}
try
{
sc.Start();
sc.WaitForStatus(ServiceControllerStatus.Running);
if(previouslyRunningServices != null)
{
previouslyRunningServices.ForEach(a =>
{
var serviceController = new System.ServiceProcess.ServiceController() { ServiceName = a };
serviceController.Start();
serviceController.WaitForStatus(ServiceControllerStatus.Running);
});
}
}
catch (InvalidOperationException)
{
}
}
private void StopService(string serviceName)
{
var sc = new System.ServiceProcess.ServiceController() { ServiceName = serviceName };
runningDependentServices[serviceName] = sc.DependentServices.Where(a => a.Status == System.ServiceProcess.ServiceControllerStatus.Running).Select(a => a.ServiceName).ToList();
if (sc.CanStop)
{
try
{
sc.Stop();
sc.WaitForStatus(ServiceControllerStatus.Stopped);
}
catch (InvalidOperationException)
{
}
}
}
I'd think a similar approach would work for Net.Tcp binding. You'd probably have to stop the "Net.Tcp Listener Adapter" Windows service (ServiceName: "NetTcpActivator") and the "Windows Process Activation Service" in that case.