I'm using rabbitmq 3.6.6 using the docker image "rabbitmq:3"
Whenever I add a new consumer to my RabbitMQ queue it hangs from anywhere to 10 seconds 10 hours.
Below is an example of code used to get the error. I also get this error in Go. So it's not library dependant.
<?php
include(__DIR__."/vendor/autoload.php");
print "Start" . PHP_EOL;
$connection = new \PhpAmqpLib\Connection\AMQPStreamConnection('xxxx', 5697, 'guest', 'guest');
$channel = $connection->channel();
$callback = function($msg) {
echo " [x] Received ", $msg->body, "\n";
};
$channel->basic_consume('repositories', '', false, false, false, false, $callback);
while(count($channel->callbacks)) {
$channel->wait();
}
When I look at the logs I see
=INFO REPORT==== 31-Jan-2017::21:14:33 ===
accepting AMQP connection <0.891.0> (10.32.0.1:54216 -> 10.44.0.3:5672)
=INFO REPORT==== 31-Jan-2017::21:14:34 ===
accepting AMQP connection <0.902.0> (10.32.0.1:54247 -> 10.44.0.3:5672)
When I do list_consumer during via rabbitmqctl I see the consumer in the list, yet no messages are processed by it.
It turns out I needed to set the Qos setting.
Some more information can be found at:
http://www.rabbitmq.com/consumer-prefetch.html
https://github.com/streadway/amqp/blob/master/channel.go#L576
Related
I'm new to RabbitMQ but I have now installed onto a Windows server and have a couple of demo console apps (C#) that happily write to a read from a queue.
The following code works to pull messages from a queue called "RabbitPoCQueue_2" on the local server:
string queueName = "RabbitPoCQueue_2";
var factory = new ConnectionFactory();
bool keepGoing = true;
factory.HostName = "127.0.0.1";
try
{
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
const bool durable = false;
channel.QueueDeclare(queueName, durable, false, false, null);
System.Console.WriteLine(" [*] Waiting for messages.");
while (keepGoing)
{
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
System.Console.WriteLine(" [x] Received {0}", message);
};
channel.BasicConsume(queue: queueName,
autoAck: true,
consumer: consumer);
channel.BasicGet(queue: queueName, autoAck: true);
System.Console.WriteLine("Press Y to continue or any other key to exit");
keepGoing = System.Console.ReadKey().Key == ConsoleKey.Y;
}
}
}
I now need to configure a BizTalk (2016 FP3 CU5) receive location to do the same. I have ensured I've stopped the console receiver and that I have messages sat on the queue for BizTalk to collect.
I followed the article https://social.technet.microsoft.com/wiki/contents/articles/7401.biztalk-server-and-rabbitmq.aspx
Problem is, when I start the receive location, I get no errors but nothing is received.
The config for the WCF receive location can be seen below:
and here:
and here's a pic from the RabbitMQ management console showing messages sat on the queue:
When I look in the RabbitMQ log file, I see 2 rows on starting the receive location. I see 3 rows when starting the .Net console app (using RabbitMQ API), as shown below - first 2 rows are from BizTalk, last 3 from the console app:
2019-08-28 16:17:45.693 [info] <0.13361.2> connection <0.13361.2> ([::1]:16807 -> [::1]:5672): user 'guest' authenticated and granted access to vhost '/' ** Start of Receive location
2019-08-28 16:19:57.958 [info] <0.13452.2> accepting AMQP connection <0.13452.2> (127.0.0.1:17173 -> 127.0.0.1:5672)
2019-08-28 16:19:58.026 [info] <0.13452.2> connection <0.13452.2> (127.0.0.1:17173 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/' ** Receive from command line
2019-08-28 18:56:26.267 [info] <0.13452.2> closing AMQP connection <0.13452.2> (127.0.0.1:17173 -> 127.0.0.1:5672, vhost: '/', user: 'guest')
2019-08-28 18:56:39.815 [info] <0.17923.2> accepting AMQP connection <0.17923.2> (127.0.0.1:41103 -> 127.0.0.1:5672)
Can anyone spot where I went wrong?
I have a queue to which a lot of messages are published (~10K). Connected to this queue are multiple consumers with the following code in codeigniter using the php-amqplib library
public function processQueue()
{
// Make connection
$connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest');
// Make channel
$channel = $connection->channel();
// Declare queue
$channel->queue_declare(QUEUE_NAME, false, false, false, false);
// PHP callable
$callback = function ($msg) {
//DO MESSAGE PROCESSING HERE
$msg->delivery_info['channel']->basic_ack($msg->delivery_info['delivery_tag']);
};
$channel->basic_consume(AGENTS_QUEUE_PROCESSING, '', false, true, false, false, $callback);
// While queue is empty, wait
while (count($channel->callbacks)) {
// Wait
$channel->wait();
}
// Close channel and connection
$channel->close();
$connection->close();
}
The messages gets filled up and are simultaneously consumed by multiple such consumers. I've observed that with some 5-6k messages remaining (i.e. after consuming around 4-5k) messages, the queue suddenly gets empty with the consumers idling and waiting for more messages. Also, there is a sudden drop at this point in time in the total number of messages on the RabbitMQ Management web panel.
I've tried making the queue with the durable parameter but the problem seems to be the same. What could be the issue and its resolution?
I have been using ELK for about six months now, and it's been great so far. I'm on logstash version 6.2.3.
RabbitMQ makes up the heart of my distributed system (RabbitMQ is itself distributed), and as such it is very important that I track the logs of RabbitMQ.
Most other conversations on this forum seem to use RabbitMQ as an input/output stage, but I just want to monitor the logs.
The only problem I'm finding is that RabbitMQ has multiline logging, like so:
=WARNING REPORT==== 19-Nov-2017::06:53:14 ===
closing AMQP connection <0.27161.0> (...:32799 -> ...:5672, vhost: '/', user: 'worker'):
client unexpectedly closed TCP connection
=WARNING REPORT==== 19-Nov-2017::06:53:18 ===
closing AMQP connection <0.22410.0> (...:36656 -> ...:5672, vhost: '/', user: 'worker'):
client unexpectedly closed TCP connection
=WARNING REPORT==== 19-Nov-2017::06:53:19 ===
closing AMQP connection <0.26045.0> (...:55427 -> ...:5672, vhost: '/', user: 'worker'):
client unexpectedly closed TCP connection
=WARNING REPORT==== 19-Nov-2017::06:53:20 ===
closing AMQP connection <0.5484.0> (...:47740 -> ...:5672, vhost: '/', user: 'worker'):
client unexpectedly closed TCP connection
I have found a brilliant code example here, which I have stripped just to the filter stage, such that it looks like this:
filter {
if [type] == "rabbitmq" {
codec => multiline {
pattern => "^="
negate => true
what => "previous"
}
grok {
type => "rabbit"
patterns_dir => "patterns"
pattern => "^=%{WORD:report_type} REPORT=+ %{RABBIT_TIME:time_text} ===.*$"
}
date {
type => "rabbit"
time_text => "dd-MMM-yyyy::HH:mm:ss"
}
mutate {
type => "rabbit"
add_field => [
"message",
"%{#message}"
]
}
mutate {
gsub => [
"message", "^=[A-Za-z0-9: =-]+=\n", "",
# interpret message header text as "severity"
"report_type", "INFO", "1",
"report_type", "WARNING", "3",
"report_type", "ERROR", "4",
"report_type", "CRASH", "5",
"report_type", "SUPERVISOR", "5"
]
}
}
}
But when I save this to a conf file and restart logstash I get the following error:
[2018-04-04T07:01:57,308][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-04-04T07:01:57,316][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-04-04T07:01:57,841][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.3"}
[2018-04-04T07:01:57,973][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-04-04T07:01:58,037][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 3, column 15 (byte 54) after filter {\n if [type] == \"rabbitmq\" {\n codec ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in `block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in `converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
Any ideas what the issue could be?
Thanks,
In case you are sending your logs from the rabbitMQ server to logstash with filebeat, you should configure the multiline there.
The answer is multiline indeed. The goal is to merge the lines starting with something else than a date with the previous line that started with a date. This is how :
multiline.pattern: '^\d{4}-\d{2}-\d{2}'
multiline.negate: true
multiline.match: after
Note: I previously tried to merge any lines started with space characters ^\s+ but that did not work because not all warning or error messages started with a space.
Complete filebeat input (7.5.2 format)
filebeat:
inputs:
- exclude_lines:
- 'Failed to publish events caused by: EOF'
fields:
type: rabbitmq
fields_under_root: true
paths:
- /var/log/rabbitmq/*.log
tail_files: false
timeout: 60s
type: log
multiline.pattern: '^\d{4}-\d{2}-\d{2}'
multiline.negate: true
multiline.match: after
Logstash patterns:
# RabbitMQ
RABBITMQDATE %{MONTHDAY}-%{MONTH}-%{YEAR}::%{HOUR}:%{MINUTE}:%{SECOND}
RABBITMQLINE (?m)=%{DATA:severity} %{DATA}==== %{RABBITMQDATE:timestamp} ===\n%{GREEDYDATA:message}
I am sure they had good reasons to log in this odd way in RMQ 3.7.x but without knowing them, it really makes our life hard.
You can't use a codec as a filter plugin. Codecs can only be used in input or output plugins (see the doc), with the codec configuration option.
You'll have to put your multiline codec in the input plugin that's producing your rabbitmq logs.
I'm following the following tutorial to the letter:
https://www.rabbitmq.com/tutorials/tutorial-two-java.html.
I start the RabbitMQ server as such:
docker pull rabbitmq
docker run -d --hostname my-rabbit-host --name my-rabbit -p 5672:5672 rabbitmq:3
From the tutorial:
Using this code we can be sure that even if you kill a worker using
CTRL+C while it was processing a message, nothing will be lost. Soon
after the worker dies all unacknowledged messages will be redelivered.
I spawn two consumers, and when I CTRL+C one of them, the other running one does not receive the messages that were originally destined to the former consumer. How do I get the messages to be redelivered after CTRL+C'ing out of one of the consumers?
Edit: I'm now installing RabbitMQ via 'brew', but I'm still seeing the same issue.
brew update
brew install rabbitmq
/usr/local/sbin/rabbitmq-server &
There is no need to put sleep or anything like that in the consumer code. On the link you provided, search for paragraph starting with Manual message acknowledgments and look at the code there. The key thing is not to acknowledge the message. If you have autoACK flag set to true, then you can call anything you want, the message is acknowledged as soon as it is received. So simply don't set that flag, and also for testing you could comment out the line channel.basicAck(envelope.getDeliveryTag(), false); in order not to do manual ACK either. So when the consumer exits, the message will still be in the queue.
Strange, RabbitMQ works for me out of the box.
Step 1, I started RabbitMQ:
$ docker run -d --hostname my-rabbit-host --name my-rabbit -p 5672:5672 rabbitmq:3
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e0c3257b8b49 rabbitmq:3 "docker-entrypoint.s…" 18 minutes ago Up 14 minutes 4369/tcp, 5671/tcp, 25672/tcp, 0.0.0.0:5672->5672/tcp my-rabbit
Step 2, I published a message (By the way, I tried with Node.js. See Appendix below for source code):
$ node src/producer.js
Publisher: TODO 1st
Step 3, I started two consumers one after another (my consumer is designed, for the testing purpose, not to acknowledge, so RabbitMQ will never dequeue a message).
Consumer 1 will receive the message, while consumer 2 won't.
Consumer 1:
$ node src/consumer.js
Consumer: TODO 1st
Consumer 2:
$ node src/consumer.js
Step 4, when I stop consumer 1 by 'Ctrl + c', Consumer 2 will immediately receive the message from RabbitMQ:
Consumer 2:
$ node src/consumer.js
Consumer: TODO 1st
Conclusion: Basically, when setting up a consumer, we need to tell RabbitMQ not to dequeue the message until its acknowledgement has been received from the consumer. As a result, if consumer 1 is stopped for any reason before it has a chance to acknowledge the message, RabbitMQ will redeliver the message to consumer 2.
Appendix
src/producer.js
var q = 'tasks'
function bail (err) {
console.error(err)
process.exit(1)
}
// Publisher
function publisher (conn) {
conn.createChannel(onOpen)
function onOpen (err, ch) {
if (err != null) bail(err)
ch.assertQueue(q)
const msg = 'TODO 1st'
ch.sendToQueue(q, Buffer.from(msg), { persistent: true })
console.log('Publisher: ', msg)
}
}
require('amqplib/callback_api')
.connect('amqp://guest:guest#localhost', function (err, conn) {
if (err != null) bail(err)
publisher(conn)
})
src/consumer.js
var q = 'tasks'
function bail (err) {
console.error(err)
process.exit(1)
}
// Consumer
function consumer (conn) {
conn.createChannel(onOpen)
function onOpen (err, ch) {
if (err != null) bail(err)
ch.assertQueue(q)
ch.consume(q, function (msg) {
if (msg !== null) {
console.log('Consumer: ', msg.content.toString())
// Commented out the line below, so RabbitMQ never dequeues a message
// ch.ack(msg)
}
}, { noAck: false })
}
}
require('amqplib/callback_api')
.connect('amqp://guest:guest#localhost', function (err, conn) {
if (err != null) bail(err)
consumer(conn)
})
I am working on a fun project which requires me to learn message queues and websockets. I am trying to connect browsers via websockets to an instance of rabbitmq using sockjs rather than pure websockets. On rabbit I have activated the plugins for stomp and web_stomp (web_stomp is required when using sockjs).
The problem I am running into is that while the call from the browser seems to be working properly because a very brief connection to Rabbit is made through the webstomp/stomp connection but after 2 or 3 seconds the connection is dropped by Rabbit.
This is confirmed by the rabbitmq logs:
=INFO REPORT==== 11-Jul-2016::23:01:54 ===
accepting STOMP connection (192.168.1.10:49746 -> 192.168.1.100:55674)
=INFO REPORT==== 11-Jul-2016::23:02:02 ===
closing STOMP connection (192.168.1.10:49746 -> 192.168.1.100:55674)
This is the browser code that connects to RabbitMQ via the webstomp plugin:
var url = "http://192.168.1.100:55674/stomp";
var ws = new SockJS(url);
var client = Stomp.over(ws);
var header = {
login: 'test',
passcode: 'test'
};
client.connect(header,
function(){
console.log('Hooray! Connected');
},
function(error){
console.log('Error connecting to WS via stomp:' + JSON.stringify(error));
}
);
Here is the Rabbit config:
[
{rabbitmq_stomp, [{default_user, [{login, "test"},
{passcode, "test"}
]
},
{tcp_listeners, [{"192.168.1.100", 55674}]},
{heartbeat, 0}
]
}
]
I have been over the Rabbit docs a million times but this feels like something simple that I am overlooking.
Resolved. After combing through the logs I realized that web_stomp was listening on port 15674 so I changed the config file to reflect that. I swear I had made that change at some point but it did not seem to make a difference.
One of the late changes I made before sending out my request was to turn off heartbeat. Everything I have read states that sockjs does not support heartbeat and that there were suggestions to turn it off rather than use the default. In addition to turning off heartbeat in the config file I also added this to the browser code:
client.heartbeat.outgoing=0;
client.heartbeat.incoming=0;