We have RabbitMQ producing compressed (gziped) messages, and we are trying to consume them via PyFlink RMQConnectionConfig, RMQSource, Deserialization and related libraries. However, we were not able to consume and suspect that the Deserialization schema being used is unable to deserialize given that the messages are compressed.
Error that we get is -
org.apache.flink.api.python.shaded.py4j.py4Exception: Constructor org.apache.flink.streaming.connectors.rabbitmq.RMQSource([class org.apache.flink.streaming.connectors.rabbitmq.common.RMQConnectionConfig, class java.lang.String, class java.lang.Boolean, class org.apache.flink.formats.json.JsonRowDeserializationSchema]) does not exist
Given this, could anyone help me in understanding how one can decompress the messages from RabbitMQ in PyFlink?
Related
Is it a good idea to send files with size about 1Mb through RabbitMQ? I want to send message in json format with binary fields corresponding to the files.
And how to do it properly using spring-amqp? Just by publishing object with next class?
class Message {
String field1;
byte[] fileField1;
byte[] fileField2;
}
I would suggest not only reading those links that were posted but also, doing some of your own experimentation. The thing I would be concerned about is performance at the service level and at the client level.
You might want to consider having a server host the files/data and allow rabbitmq just send the message to the consumer with the id of the message in it. So when your consumer gets the message, it sends an HTTP GET request to a service that requests the actual message payload. That way RabbitMQ stays lightweight. You can always add consumers and servers if you need.
That's my opinion without experimenting. You might find that it's still lighting fast with 1MB payloads. That's why I would say to experiment and find out for yourself.
Hope you find this helpful!
I have a service that is configured to handle a command object. In the Handle method I am throwing an exception in specific business cases. When this occurs it send the error to the Error queue and I think the Particular Management service is processing the Error queue. It is my understanding is that the Particular Management service will monitor the Error queue and process the messages, that is to say, mine the metadata and persist it to RavenDB. Lastly, it should forward the message to the Error.Log queue. Regrettably, every message seems to get routed to the Service.Management.Errors queue. Presumably, because the Particular Management service handler is failing.
If I look the Extension Headers of the messages in the Service.Management.Errors queue, I see the following information:
<HeaderInfo>
<Key>NServiceBus.ExceptionInfo.ExceptionType</Key>
<Value>System.NullReferenceException</Value>
</HeaderInfo>
<HeaderInfo>
<Key>NServiceBus.ExceptionInfo.Message</Key>
<Value>Object reference not set to an instance of an object.</Value>
</HeaderInfo>
<HeaderInfo>
<Key>NServiceBus.ExceptionInfo.Source</Key>
<Value>NServiceBus.Core</Value>
</HeaderInfo>
<HeaderInfo>
<Key>NServiceBus.ExceptionInfo.StackTrace</Key>
<Value>System.NullReferenceException: Object reference not set to an instance of an object.
at NServiceBus.Unicast.Transport.TransportReceiver.ProcessMessage (TransportMessage message) in c:\TeamCity\buildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 357
at NServiceBus.Unicast.Transport.TransportReceiver.TryProcess(TransportMessage message) in c:\TeamCity\buildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 235
at NServiceBus.Transports.Msmq.MsmqDequeueStrategy.Action() in c:\TeamCity\buildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Transports\Msmq\MsmqDequeueStrategy.cs:line 170
Since the ServiceInsight Particular Management service is presumably supposed to be generic in nature, I would assume that it does not need to know about the actual types of messages. Meaning, I don't have to throw my Message dll's into the directory of the service (correct?). Thus implying that it simply mines all of it's data to persist to RavenDB by simply reading the Extension Headers.
So with that ground work in place. Can someone please validate or invalidate my understanding of how ServiceInsight works and is configured. Secondly, my error looks like it is an error in NServiceBus.Core. Am I supposed to add some DLL's or something. Of is there another likely candidate for causing this issue that I am simply overlooking.
We were having a similar problem, and although our problem was probably pretty specific to us, I'll share it just in case.
Most of our services use NServiceBus 4.0.3 with XML serialization, but we have one older service that is using NSB 3.3.1 and binary serialization. That service was logging to the audit queue, and when the Particular Management service tried to read those messages, it would fail. I'm not sure what the source of the error was, due to NSB re-throwing the exception and losing the original stack trace. I suspect that it is trying to read a header that isn't there. It may also be due to using binary serialization.
Our solution was to turn off auditing for that one older service.
What is the benefit of building on top of MassTransit compared to building directly on top of RabbitMQ?
I believe one benefit provided by MassTransit is 'type' exchange (publish subscribe by interface / type) so the content of the message is structured, compared to plain RabbitMQ exchanges where the content of the message is unstructured text / blob.
What other benefits provided by MassTransit?
Things that MT adds on top of just using RabbitMQ:
Optimized, asynchronous multithreaded, concurrent consumers
Message serialization, with support for interfaces, classes, and records, including guidance on versioning message contracts
Automatic exchange bindings, publish conventions
Saga state machines, including persistent state via Entity Framework Core, MongoDB, Redis, etc.
Built-in metrics, Open Telemetry, Prometheus
Message headers
Fault handling, message retry, message redelivery
Those are just a few, some more significant than others. The fact that the bus hosts your consumers, handlers, sagas, and manages all of the threading is probably the biggest advantage, and the fact that you can host multiple buses in the same process.
Serialization is the next biggest benefit, since that can be painful to figure out, and getting an interface-based message contract with automatic deserialized into types (including dynamically-backed interface types) is huge. Publishing a single class that implements multiple interfaces, and seeing all interested consumers pick up their piece of the message asynchronously is just awesome in production as new interfaces can be added to producers and down-level consumers are unaffected.
Those are a few, you can check out the documentation for more information, or give the really old .NET Rocks! podcast a listen for some related content by yours truly.
UPDATE: There is an entire series on YouTube covering MassTransit now.
I'm using the web console against my AMQ 5.2 instance successfully, except for I cannot see the content of all of my messages.
If I send a test message using the web console, I can see the sample text content, but I believe the vendor app I am working with has binary or byte array message content.
Is there something I need to do to be able to view this raw data?
Thanks,
To my knowledge, it is not possible to inspect messages in the Admin Console. You can get some statistics (like how many messages have been sent etc.).
ActiveMQ does not unmarshal messages when receiving them (for performance reasons, unmarshalling is rather expensive).
Thus, if you want to have some way to inspect messages for their content, you can basically do 2 things:
Write a consumer which registers for all topics/queues, through which you can see messages' content. Drawback: if you're using queue-based interaction, your "real" consumers will not get all messages
Write an activeMQ plugin which looks at the messages. Have a look at ActiveMQ's Logger Plugin. Then write your own (you'll need the sources to compile it) and load it with ActiveMQ (see the documentation on how to configure ActiveMQ to load plugins). You want to override the send() method which is called whenever someone sends a message to the broker. There you get a reference to the message and can access its content.
Neither of the two messages provides a convenient viewing-mechanism though. You'll have to resort to standard out, or write your own web-based access.
hawtio now shows first 256 chars of messages. Don't know if that is enough for you. Use browse() method.
I have a requirement to build a NSB subscriber which will subscribe to messages being published by a service which is already live. This service was built with a messages assembly containing the NSB IMessage implementations which I want to subscribe to. So to build my subscriber I need a hard dependency on this assembly.
When my subscriber starts up it sends some messages to the publisher input queue which results in the publisher recording my subscriptions in the database. One of my subscriptions looks like this:
MyNamespace.MyMessageType, MyNamespace, Version=1.0.0.0, Culture=neutral, PublicKeyToken=MyPublicKeyToken
Unfortunately the publisher is publishing messages from an assembly which is not strong named. So when publishing, the subscription evaluation process fails to evaluate successfully against my subscription, because the public key token of the message being published (value = null) does not match my subscription.
My question is: Can I not subscribe to messages by Type and version only? Even better - can I not subscribe to messages based on some external contract (like an XSD) and break this dependency altogether?
Many thanks in advance.
UPDATE: The NSB docs hint at something like this here:
http://www.nservicebus.com/MessagesAsInterfaces.aspx
For regular messages you create you can use the XsdGenerator tool in the tools directory of NSB to generate a schema that you can pass along to other endpoints. You will want to use this tool instead of the .NET Framework tool as it does not support interfaces. From there you can use the schema to handle messages.
For subscription messages, if you don't want to use the assembly you could tell NSB to DoNotAutoSubscribe() and subscribe manually(Bus.Subscribe(Type)) passing along the Type of the message however you wish. This could be off the schema or some other configuration.