I'm trying to add some custom logic to messages in ServiceStack and RabbitMQ.
It seems that the queues created by ServiceStack have some illegible characters prepended to the queue name and that makes it hard to reference them by name. For example (link from the RabbitMQ admin tool):
http://localhost:15672/#/queues/%2F/%E2%80%8E%E2%80%8Emq%3ATestRequest.inq
Note the %E2%80%8E%E2%80%8E prepended to the queue name. Although the queue looks like mq:TestRequest.inq it seems to have a different name. I also checked on another machine and the behaviour is consistent. I also suspect routing keys are affected in the same manner.
However, if I manually create a queue like this (and as far as I can see, ServiceStack does it in a similar way):
RabbitMqServer mqServer = new RabbitMqServer(connectionString: hostName, username: userName, password: password);
RabbitMqMessageFactory factory = (RabbitMqMessageFactory)MqServer.MessageFactory;
using (var mqClient = new RabbitMqProducer(factory))
{
var channel = mqClient.Channel;
string qName = new QueueNames(typeof(TestRequest)).In;
channel.QueueDeclare(qName, true, false, false, null);
}
The creted queue has a "normal" name without extra characters.
http://localhost:15672/#/queues/%2F/mq%3ATestRequest.inq
Also, it seems that the exchanges are created with names as expected.
My questions:
How to force ServiceStack to create queues without appending these characters?
OR
How to construct queue names containing these characters?
EDIT:
It seems that the inserted character is Left-to-right mark ( or \u200e). Prepending these characters to the queue name / routing key seems to get the job done. However, this looks rather hacky so I'd like to avoid doing this.
This might be inside the internals of RabbitMQ and may depend if you are using AMQP or STOMP. Here is an except from the full page:
If /, % or non-ascii bytes are in the queuename, exchange_name or routing_key, they are each replaced with the sequence %dd, where dd is the hexadecimal code for the byte.
RabbitMQ - Stomp - Destinations - AMQP Semantics
Related
We're looking for the best way to ingest data in warp10. We are on a Microservices architecture that use Kafka mainly.
Two solutions:
Use Ingress endpoint as defined here: https://www.warp10.io/content/03_Documentation/03_Interacting_with_Warp_10/03_Ingesting_data/01_Ingress (This is the solution we use for now)
Use the warp10 Kafka plugin as defined here: https://blog.senx.io/introducing-the-warp-10-kafka-plugin/
As described here, we use Ingress solution as of now, based on an aggregation of data for x seconds, and call the Ingress API to send data per packet. (Instead of calling the API each time we need to insert something).
For few days, we are experimenting with the Kafka Plugin. We successfully set up the plugin and create an .mc2 responsible to consume data from a given topic and then insert them using UPDATE into warp10.
Questions:
Using the Kafka plugin, would it be better to apply the same buffer mechanism as the one applied when we use the Ingress endpoint? Or, is there any specific implementation in warp10 Kafka plugin that allows to consume message per message in the topic and call the UPDATE function for each ?
Today, as both solutions are working, we're trying to find differences to get the best performance results during ingestion of data. And if possible, without having to apply any buffer mechanism because we are trying to be in real-time as much as possible.
MC2 file:
{
'topics' [ 'our_topic_name' ] // List of Kafka topics to subscribe to
'parallelism' 1 // Number of threads to start for processing the incoming messages. Each thread will handle a certain number of partitions.
'config' { // Map of Kafka consumer parameters
'bootstrap.servers' 'kafka-headless:9092'
'group.id' 'senx-consumer'
'enable.auto.commit' 'true'
}
'macro' <%
// macro executed each time a kafka record is consumed
/*
// received record format :
{
'timestamp' 123 // The record timestamp
'timestampType' 'type' // The type of timestamp, can be one of 'NoTimestampType', 'CreateTime', 'LogAppendTime'
'topic' 'topic_name' // Name of the topic which received the message
'offset' 123 // Offset of the message in 'topic'
'partition' 123 // Id of the partition which received the message
'key' ... // Byte array of the message key
'value' ... // Byte array of the message value
'headers' { } // Map of message headers
}
*/
"recordArray" STORE
"preprod.write" "token" STORE
// macro can be called on timeout with an empty entry map
$recordArray SIZE 0 !=
<%
$recordArray 'value' GET // kafka record value is retrieved in bytes
'UTF-8' BYTES-> // convert bytes to string (WARP10 INGRESS format)
JSON->
"value" STORE
"Records received through Kafka" LOGMSG
$value LOGMSG
$value
<%
DROP
PARSE
// PARSE outputs a gtsList, including only one gts
0 GET
// GTS rename is required to use UPDATE function
"gts" STORE
$gts $gts NAME RENAME
%>
LMAP
// Store GTS in Warp10
$token
UPDATE
%>
IFT
%> // end macro
'timeout' 10000 // Polling timeout (in ms), if no message is received within this delay, the macro will be called with an empty map as input
}
If you want to cache something in Warp 10 to avoid lots of UPDATE per second, you can use SHM (SHared Memory). This is a built-in extension you need to activate.
Once activated, use it with SHMSTORE and SHMLOAD to keep objects in RAM between two WarpScript executions.
In you example, you can push all the incoming GTS in a list, or a list of list of GTS, using +! to append elements to an existing list.
The MERGE of all the GTS in the cache (by name + labels) and UPDATE in the database can then be done in a runner (don't forget to use a MUTEX)
Don't forget the total operation cost:
The ingress format can be optimized for ingestion speed, if you do not repeat classname and labels, and if you gather lines per gts. See here.
PARSE deserialize data from the Warp 10 ingress format.
UPDATE serialize data to the Warp 10 optimized ingress format (and push it to the update endpoint).
the update endpoint deserialize again.
It makes sense to do these deserialize/serialize/deserialize operation if your input data is far from the optimal ingress format. It also make sense if you want to RANGECOMPACT your data to save disk space, or do any preprocessing.
I'm trying to write something for my iOS app, using RMQClient that scans for existing exchanges on a rabbitmq server. I came up with this so far.
class AMQPExchangeScanner {
static func scan() {
let connection:RMQConnection = RMQConnection(uri: "amqp://user:user#abc.def.com:5672", delegate: RMQConnectionDelegateLogger())
connection.start()
for exchangeName in Foo.pastExchanges() {
let channel = connection.createChannel()
let exchange = channel.fanout(exchangeName, options: .passive)
"scan \(exchangeName) \(exchange)".print()
channel.close()
}
}
}
I'm not sure how to determine if the exchange actually exists though. The print() statement prints exchange objects. I get a whole bunch of output in the console. I had hoped that I would get back an optional so I could do something like
if let exchange... {
}
But that doesn't appear to be the case. How do I programmatically check if the exchange is real or not? Or get at those errors? Do I need my own connection delegate and have to parse a bunch of text?
The best way to scan for existing exchanges is using rabbitmq management HTTP API - it would be under /api/exchanges.
For a single exchange you could call the declare method, with passive parameter set accordingly (quoting from here):
for the declare method
This method creates an exchange if it does not already exist, and if
the exchange exists, verifies that it is of the correct and expected
class.
for the passive bit
If set, the server will reply with Declare-Ok if the exchange already
exists with the same name, and raise an error if not.
...
If not set and the exchange exists, the server MUST check that the existing exchange has the same values for type, durable, and arguments fields. The server MUST respond with Declare-Ok if the requested exchange matches these fields, and MUST raise a channel exception if not.
I'm working on some changes to:
https://github.com/puppetlabs/puppetlabs-rabbitmq/
I haven't worked much with rspec or with puppet types / providers, so it's been slow going. Haven't had much input via their ticket system or in Github, so just wanted to get some advice in terms of the design changes.
Basically, the module doesn't currently support multiple bindings with the same source / destination / vhost combo, but different routing keys.
I had a (mostly) working fix that was backwards compatible involving making routing_key work as either a string or array (https://tickets.puppetlabs.com/browse/MODULES-3679)
However, this doesn't work too well, because the existing provider expects the resource to be unique, and seems to assume a predictable mapping between the title of the resource and the
I had thought about making the title and name attributes different from each other, or even completely abstracting the name / title from the attributes that must be unique:
rabbitmq_binding { 'exchange1#queue1#host3-1':
name => 'exchange1#queue1#host3',
destination_type => 'queue',
routing_key => 'routingkey1',
ensure => present,
}
rabbitmq_binding { 'exchange1#queue1#host3-2':
name => 'exchange1#queue1#host3',
destination_type => 'queue',
routing_key => 'routingkey2',
ensure => present,
}
The actual rabbitmqctl output that self.instances is collecting for each vhost (in an example where there are two routing keys for same source / dest / vhost combo (no vhost in this example) looks like:
foo.bar.exchange exchange axs.bar.baz.queue queue axs.bar.baz.key []
foo.bar.exchange exchange axs.bar.baz.queue queue axs.bar.baz.published.key []
I'd rather not try to encode the routing key in the title / name attribute of the resource, but is it possible to have routing_key also tied to the resource (presumably by updating the exists? method as well as self.instances)? What would be the best things to check in unit tests for this (I already have an acceptance test), and how can I modify lib/puppet/provider/rabbitmq_binding/rabbitmqadmin.rb to support this?
Or do I just need to encode the routing key in the name as well, which just seems really ugly, and would also make backwards compatibility more difficult?
rabbitmq_binding { 'exchange1#queue1#host3#routingkey':
or
rabbitmq_binding { 'exchange1#queue1#/#routingkey':
I have a camel route which processes a message from a process queue and sends it to upload queue.
from("activemq:queue:process" ).routeId("activemq_processqueue")
.process(exchange -> {
SomeImpl impl = new SomeImpl();
impl.process(exchange);
})
.to(ExchangePattern.InOnly, "activemq:queue:upload");
In impl.process I am populating an Id and destination server path. Now I need to define a new route which consumes messages from upload queue ,and copy a local folder (based on Id generated in previous route) and upload it to destination folder which is an ftp server (this is also populated in previous route)
So how to design a new route where both from and to endpoints are dynamic which would look something like below ?
from("activemq:queue:upload" )
.from("file:basePath/"+{idFromExchangeObject})
.to("ftp:"+{serverIpFromExchangeObject}+"/"+{pathFromExchangeObject});
I think there is a better alternative for your case, taking as granted that you are using a Camel version newer than 2.16.(alternatives for a previous version exist but the are more complicated and don't look elegant - ( e.g consumerTemplate & recipientList).
You can replace the first "dynamic from" with pollEnrich which enriches the message using a polling consumer and simple expression to build the dynamic file endpoint. For the second part, as already mentioned, a dynamic uri .toD will do the job. So your route would look like this:
from("activemq:queue:upload" )
.pollEnrich().simple("file:basePath/${header.idFromExchangeObject})
.aggregationStrategy(new ExampleAggregationStrategy()) // * see explanation
.timeout(2000) // the timeout is optional but recommended
.toD("ftp:${header.serverIpFromExchangeObject}/${header.pathFromExchangeObject}")
See content enricher section "Using dynamic uris"
http://camel.apache.org/content-enricher.html .
You will need an aggregation strategy, to combine the original exchange with the resource exchange in order to make sure that the headers serverIpFromExchangeObject, pathFromExchangeObject will be included in the aggregated exchange after the enrichment. If you don't include the custom strategy then Camel will by default use the body obtained from the resource. Have a look at the ExampleAggregationStrategy example in content-enricher.html to see how this works.
For the .toD() have a look at http://camel.apache.org/how-to-use-a-dynamic-uri-in-to.html
Adding a dynamic to endpoint in Camel (as noted in the comment) can be done with the .toD() which is described on this page on the Camel site.
I don't know of any fromD() equivalent. However, you could add a dynamic route by calling the addRoutes method on the CamelContext. This is described on this page on the Camel site.
Expanding slightly on the example from the Camel site here is something that should get you heading in the right direction.
public void process(Exchange exchange) throws Exception {
String idFromExchangeObject = ...
String serverIpFromExchangeObject = ...
String pathFromExchangeObject = ...
exchange.getContext().addRoutes(new RouteBuilder() {
public void configure() {
from("file:basePath/"+ idFromExchangeObject)
.to("ftp:"+ serverIpFromExchangeObject +"/"+pathFromExchangeObject);
}
});
}
There may be other options in Camel as well since this framework has an amazing number of EIP and capabilities.
I'm using Akka 2.4.7 to read a web resource that is essentially a stream of JSON objects, delimited with newlines. The stream is practically unlimited in size.
When around 8MB has been consumed, I get an exception:
[error] (run-main-0) EntityStreamSizeException: actual entity size (None) exceeded content length limit (8388608 bytes)! You can configure this by setting `akka.http.[server|client].parsing.max-content-length` or calling `HttpEntity.withSizeLimit` before materializing the dataBytes stream.
The "actual entity size (None)" seems a bit funny, but my real question is, how to use the HttpEntity.withSizeLimit (or in my case, rather .withoutSizeLimit that should be there, as well).
My request code is like this:
val chunks_src: Source[ByteString,_] = Source.single(req)
.via(connection)
.flatMapConcat( _.entity.dataBytes )
I tried adding a .map( (x: HttpResponse) => x.withoutSizeLimit ), but it does not compile. What's the role of the HttpEntity when doing client side programming, anyways?
I can change the global config, but that's kind of missing the point. I'd like to flag "no limits" only for a particular request.
As a further question, I understand the need for a max-content-length on the server side, but why affect the client?
References:
Akka 2.4.7: Limiting message entity length
Akka 2.4.7: HttpEntity
I'm far from an expert on this topic, but it would seem you need to add the .withoutSizeLimit() to the entity like:
Source.single(req)
.via(connection)
.flatMapConcat( _.entity.withoutSizeLimit().dataBytes )