I'm using RabbitMQ plugin of Logstash to pull from RabbitMQ and push to Elasticsearch using this pipeline:
input {
rabbitmq {
queue => "Elasticsearch_Queue"
host => "rabbitmq"
exchange => "my_event_bus"
key => "SomeIntegrationEvent"
}
}
output {
elasticsearch {
hosts => [ "elasticsearch:9200" ]
}
stdout { codec => rubydebug }
}
It works fine and creates a queue named Elasticsearch_Queue and binds to my_event_bus exchange using routing_key SomeIntegrationEvent.
I need to subscribe to multiple events but there is no clear solution on the plugin docs.
We have to add multiple entries with same queue and exchange but different key, as below:
input {
rabbitmq {
queue => "Elasticsearch_Queue"
host => "rabbitmq"
exchange => "my_event_bus"
key => "SomeIntegrationEvent1"
}
rabbitmq {
queue => "Elasticsearch_Queue"
host => "rabbitmq"
exchange => "my_event_bus"
key => "SomeIntegrationEvent2"
}
}
I got the answer from elastic forums.
Since the key must be a string, you can't explicitly bind the queue to the exchange using multiple keys.
I suggest to make my_event_bus a fanout exchange. This will route every message delivered into this exchange to the bound queue.
Then define a second direct exchange that you manually bind to the first exchange multiple times using the routing keys (= events) you are interested in.
Related
Using MassTransit with RabbitMQ trying to use default exchange.
With RabbitMQ library is was fine as it was default behaviour, but with MassTransit seems to make this is difficult?
I found out how to configure custom exchange name:
...
serviceCollection.AddMassTransit(x =>
{
x.UsingRabbitMq((rabbitContext, rabbitConfig) =>
{
rabbitConfig.Host(config.GetConnectionString("RabbitMq"));
Bus.Factory.CreateUsingRabbitMq(cfg =>
{
cfg.Message<TestMessage>(x => x.SetEntityName(""));
});
rabbitConfig.ConfigureEndpoints(rabbitContext);
rabbitConfig.Durable = true;
});
});
Then I get:
RabbitMQ.Client.Exceptions.OperationInterruptedException: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=403, text='ACCESS_REFUSED - operation not permitted on the default exchange', classId=40, methodId=10
Operation is not permitted? I'm a little confused here, why is this so toublesum, am I doing something wrong? I only want one consumer to ever pick up one message from the queue, so the default exchange Masstransit created which is fanout doesn't work for me here.
In StartUp of the project, I make the following settings for MassTransit.ActiveMQ. But when I run, it creates two queues for me, one is event-listener and the other is called Generation.
When I publish information, the information goes into the queues generated by the system.
But I want the information to be published inside queue event-listener that I set.
Please guide me
services.AddMassTransit(x =>
{
x.AddConsumer<EventConsumer>();
x.UsingActiveMq((context, cfg) =>
{
cfg.Host("localhost", h =>
{
h.Username("admin");
h.Password("admin");
});
cfg.ReceiveEndpoint("event-listener", e =>
{
e.ConfigureConsumer<EventConsumer>(context);
});
});
});
MassTransit will only create queues for configured consumers, or explicitly configured receive endpoints. In the code above, the only queue created would be called event-listener. For each message type consumed by the consumer, a topic is created and a virtual topic consumer is created so that the receive endpoint can consume messages of each type.
When messages are published, a topic is created for each published message type.
If you want to send a message directly to a queue, instead of publishing:
var provider = serviceProvider.GetRequiredService<ISendEndpointProvider>();
var endpoint = await provider.GetSendEndpoint(new Uri("queue:event-listener"));
await endpoint.Send(...);
I would like all the messaging for a particular saga to all take place on the same topic.
I set my saga up like so:
public void ConfigureServices(IServiceCollection services)
{
services.AddMassTransit(x =>
{
var machine = new MyStateMachine();
var repository = new InMemorySagaRepository<MyState>();
x.UsingAzureServiceBus((ctx,cfg) => {
cfg.Host(config.ServiceBusConnectionString);
cfg.SubscriptionEndpoint("mySub", "myTopic", e =>
{
e.StateMachineSaga(machine, repository);
});
});
});
}
In the state machine I do this:
Initially(
When(FirstEvent)
.PublishAsync(context => context.Init<SecondMessage>(new { TestParam = "test"}))
.TransitionTo(FirstState));
I would like the SecondMessage to be published on the topic 'myTopic' but instead MassTransit creates a new topic for this message.
MassTransit creates a topic per message type, and publishes messages to their corresponding topics. Assigning the same topic name to multiple message types is not recommended.
If you want to use a subscription endpoint for a saga, you would need to configure the saga on the corresponding topic for each event. By default, MassTransit will forward the message topics for each saga event to the receive endpoint queue.
An example topology is shown below:
I created a microservice application that microservices using MassTransit and RabbitMQ for communication.
Each microservice developed using clean architecture, so we have MediatR inside each microservice.
Is it possible to use MassTransit for inside communication as well? so I can use the same signature for all services and when I want to expose a service to be used inter-microservice, it will be doable with ease.
So MediatR used for intra-communication and RabbitMQ used for inter-communication, and whole universe is on MassTransit system.
[Update] My question is how we can configure consumers so some can be used for inside communication (via MediatR) and some can be used for external communication (via RabbitMQ) and easily change them from inside to outside.
[Update2] for example here is my MassTransit registration:
services.AddMassTransit(x =>
{
x.AddConsumers(Assembly.GetExecutingAssembly());
x.AddBus(provider =>
Bus.Factory.CreateUsingRabbitMq(cfg =>
{
cfg.Host(new Uri(config.RabbitMQ.Address), h =>
{
h.Username(config.RabbitMQ.Username);
h.Password(config.RabbitMQ.Password);
});
cfg.ReceiveEndpoint("my-queue", ep => { ep.ConfigureConsumers(provider); });
}));
x.AddMediator((provider, cfg) => { cfg.ConfigureConsumers(provider); });
});
How can I differ in internal communication and external communication? in other words, how can I register some consumers to MediatR and some to RabbitMQ?
They can be used together, and MassTransit has its own Mediator implementation as well so you can write your handlers once and use them either via the mediator or via a durable transport such as RabbitMQ.
There are videos available that take you through the capabilities, starting with mediator and moving to RabbitMQ.
I found that I should create a separate bus for each. then external services inherit from an interface like IExternalConsumer, so I can separate them form internal ones and add them to related bus:
UPDATED for version 7
// find consumers
var types = AssemblyTypeCache.FindTypes(new[]{Assembly.GetExecutingAssembly()},TypeMetadataCache.IsConsumerOrDefinition).GetAwaiter().GetResult();
var consumers = types.FindTypes(TypeClassification.Concrete | TypeClassification.Closed).ToArray();
var internals = new List<Type>();
var externals = new List<Type>();
foreach (Type type in consumers)
{
if (type.HasInterface<IExternalConsumer>())
externals.Add(type);
else
internals.Add(type);
}
services.AddMediator(x =>
{
x.AddConsumers(internals.ToArray());
x.ConfigureMediator((provider, cfg) => cfg.UseFluentValidation());
});
services.AddMassTransit<IExternalBus>(x =>
{
x.AddConsumers(externals.ToArray());
x.AddBus(provider =>
Bus.Factory.CreateUsingRabbitMq(cfg =>
{
cfg.Host(new Uri(config.RabbitMQ.Address), h =>
{
h.Username(config.RabbitMQ.Username);
h.Password(config.RabbitMQ.Password);
});
cfg.ReceiveEndpoint(apiProviderName, ep => { ep.ConfigureConsumers(provider); });
}));
});
services.AddMassTransitHostedService();
I'm using massTransit with RabbitMQ. Publishing messages with massTransit will create an exchange for my message type. And a masstransit consumer will create queues and bindings to an exchange. Great, make things easy.
Before looking at massTransit I used rabbitMQ's api to create queues, exchanges and binding. I would get both publisher and consumers to run the same setup code. So no matter who ran first all queues, exchanges and binding will be created no matter which part of the application ran first. This was great when running in a development environment.
I was wondering if something similar could be achieved with massTransit?
With MassTransit should be the same: consumers will create queues bound to the exchanges of the messages they consume (with names equal to the messages types).
Publishers will create the exchanges with same names of the types of the messages they publish.
Remember that if the messages published or consumed have super classes or implement interfaces, MassTransit will create the same hierarchy, creating and binding as many exchanges as your message class hierarchy has.
You could use HareDu 2 to achieve this with the below code. This works with both Autofac and .NET Core DI. Check the docs here: https://github.com/ahives/HareDu2
// Create a queue
var result = _container.Resolve<IBrokerObjectFactory>()
.Object<Queue>()
.Create(x =>
{
x.Queue("fake_queue");
x.Configure(c =>
{
c.IsDurable();
c.AutoDeleteWhenNotInUse();
c.HasArguments(arg =>
{
arg.SetQueueExpiration(1000);
arg.SetPerQueuedMessageExpiration(2000);
});
});
x.Targeting(t =>
{
t.VirtualHost("fake_vhost");
t.Node("fake_node");
});
});
// Create an exchange
var result = _container.Resolve<IBrokerObjectFactory>()
.Object<Exchange>()
.Create(x =>
{
x.Exchange("fake_exchange");
x.Configure(c =>
{
c.IsDurable();
c.IsForInternalUse();
c.HasRoutingType(ExchangeRoutingType.Fanout);
c.HasArguments(arg =>
{
arg.Set("fake_arg", "fake_arg_value");
});
});
x.Targeting(t => t.VirtualHost("fake_vhost"));
});
// Create a binding
var result = _container.Resolve<IBrokerObjectFactory>()
.Object<Binding>()
.Create(x =>
{
x.Binding(b =>
{
b.Source("fake_exchange");
b.Destination("fake_queue");
b.Type(BindingType.Exchange);
});
x.Configure(c =>
{
c.HasRoutingKey("your_routing_key");
c.HasArguments(arg =>
{
arg.Set("your_arg", "your_arg_value");
});
});
x.Targeting(t => t.VirtualHost("fake_vhost"));
});