I've got logstash running, and successfully reading in a file
rabbitmq is running, I'm watching the log, and I can see the web interface
I've configured logstash to output to a rabbitmq exchange... I think!
Here's the problem: nothing ever gets posted to the exchange, as seen in the web interface.
Any ideas?
My output config:
output {
rabbitmq {
codec => plain
host => localhost
exchange => yomtvraps
exchange_type => direct
}
file { path => "/tmp/heartbeat-from-logstash.log" }
}
UPDATE: I'm watching the rabbit log with
tail -F /usr/local/var/log/rabbitmq/rabbit\#localhost.log
As it turns out, the problem was that there was no routing key set for the exchange and queue.
A working config is:
output {
rabbitmq {
codec => plain
host => localhost
exchange => yomtvraps
exchange_type => direct
key => yomtvraps
# these are defaults but you never know...
durable => true
port => 5672
user => "guest"
password => "guest"
}
}
Here's a sample receiver code (using ruby "Bunny")
require "bunny"
conn = Bunny.new(:automatically_recover => false)
conn.start
ch = conn.create_channel
q = ch.queue("yomtvraps")
exchange = ch.direct("yomtvraps", :durable => true)
begin
puts " [*] Waiting for messages. To exit press CTRL+C"
q.bind(exchange, :routing_key => "yomtvraps").subscribe(:block => true) do |delivery_info, properties, body|
puts " [x] Received #{body}"
end
rescue Interrupt => _
conn.close
exit(0)
end
you rabbitmq's parameter seems not enough, username,password and port have not been configured.
You can configure two outputs, one is to rabbitmq, the other is to file for vertifying the log's creation and log stash is ok.
pay attention to the logstash's version(log stash, rabbitmq plugin), it gave me lots of trouble in my trial before (log stash to another redis server etc).
You could debug rabbitmq's log.
ps -ef|grep erl you could find the log file's path in the arguments.
Be sure that rabbitmq's web manager plugin is enabled, and firewall is rightly configured, then open rabbitmq's web manager, ipaddress:15672
check the exchange's type is ok (in this case 'direct' may be a correct choice), your message consumer is configured ok, and your consumer's queue has been been bound to the exchange correctly.
try to post the message to your consumer through web manager and ensure consumer work well.
Monitor your queue when log stash push log into your consumer.
Related
I have registered receive endpoint in SingleActiveConsumer mode. However I can't find a way to send a message directly to queue by using sendEndpoint. I receive following error:
The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=406, text='PRECONDITION_FAILED - inequivalent arg 'x-single-active-consumer' for queue 'test' in vhost '/': received none but current is the value 'true' of type 'bool'',
I tried setting header "x-single-active-consumer"=true by using bus configurer:
var bus = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
cfg.Host("localhost", "/", h =>
{
h.Username("guest");
h.Password("guest");
});
cfg.ConfigureSend(a => a.UseSendExecute(c => c.Headers.Set("x-single-active-consumer", true)));
});
and directly on sendEndpoint:
await sendEndpoint.Send(msg, context => {
context.Headers.Set("x-single-active-consumer", true);
});
If you want to send directly to a receive endpoint in MassTransit, you can use the short address exchange:test instead, which will send to the exchange without trying to create/bind the queue to the exchange with the same name. That way, you decouple the queue configuration from the message producer.
Or, you could just use Publish, and let the exchange bindings route the message to the receive endpoint queue.
I'm following the following tutorial to the letter:
https://www.rabbitmq.com/tutorials/tutorial-two-java.html.
I start the RabbitMQ server as such:
docker pull rabbitmq
docker run -d --hostname my-rabbit-host --name my-rabbit -p 5672:5672 rabbitmq:3
From the tutorial:
Using this code we can be sure that even if you kill a worker using
CTRL+C while it was processing a message, nothing will be lost. Soon
after the worker dies all unacknowledged messages will be redelivered.
I spawn two consumers, and when I CTRL+C one of them, the other running one does not receive the messages that were originally destined to the former consumer. How do I get the messages to be redelivered after CTRL+C'ing out of one of the consumers?
Edit: I'm now installing RabbitMQ via 'brew', but I'm still seeing the same issue.
brew update
brew install rabbitmq
/usr/local/sbin/rabbitmq-server &
There is no need to put sleep or anything like that in the consumer code. On the link you provided, search for paragraph starting with Manual message acknowledgments and look at the code there. The key thing is not to acknowledge the message. If you have autoACK flag set to true, then you can call anything you want, the message is acknowledged as soon as it is received. So simply don't set that flag, and also for testing you could comment out the line channel.basicAck(envelope.getDeliveryTag(), false); in order not to do manual ACK either. So when the consumer exits, the message will still be in the queue.
Strange, RabbitMQ works for me out of the box.
Step 1, I started RabbitMQ:
$ docker run -d --hostname my-rabbit-host --name my-rabbit -p 5672:5672 rabbitmq:3
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e0c3257b8b49 rabbitmq:3 "docker-entrypoint.s…" 18 minutes ago Up 14 minutes 4369/tcp, 5671/tcp, 25672/tcp, 0.0.0.0:5672->5672/tcp my-rabbit
Step 2, I published a message (By the way, I tried with Node.js. See Appendix below for source code):
$ node src/producer.js
Publisher: TODO 1st
Step 3, I started two consumers one after another (my consumer is designed, for the testing purpose, not to acknowledge, so RabbitMQ will never dequeue a message).
Consumer 1 will receive the message, while consumer 2 won't.
Consumer 1:
$ node src/consumer.js
Consumer: TODO 1st
Consumer 2:
$ node src/consumer.js
Step 4, when I stop consumer 1 by 'Ctrl + c', Consumer 2 will immediately receive the message from RabbitMQ:
Consumer 2:
$ node src/consumer.js
Consumer: TODO 1st
Conclusion: Basically, when setting up a consumer, we need to tell RabbitMQ not to dequeue the message until its acknowledgement has been received from the consumer. As a result, if consumer 1 is stopped for any reason before it has a chance to acknowledge the message, RabbitMQ will redeliver the message to consumer 2.
Appendix
src/producer.js
var q = 'tasks'
function bail (err) {
console.error(err)
process.exit(1)
}
// Publisher
function publisher (conn) {
conn.createChannel(onOpen)
function onOpen (err, ch) {
if (err != null) bail(err)
ch.assertQueue(q)
const msg = 'TODO 1st'
ch.sendToQueue(q, Buffer.from(msg), { persistent: true })
console.log('Publisher: ', msg)
}
}
require('amqplib/callback_api')
.connect('amqp://guest:guest#localhost', function (err, conn) {
if (err != null) bail(err)
publisher(conn)
})
src/consumer.js
var q = 'tasks'
function bail (err) {
console.error(err)
process.exit(1)
}
// Consumer
function consumer (conn) {
conn.createChannel(onOpen)
function onOpen (err, ch) {
if (err != null) bail(err)
ch.assertQueue(q)
ch.consume(q, function (msg) {
if (msg !== null) {
console.log('Consumer: ', msg.content.toString())
// Commented out the line below, so RabbitMQ never dequeues a message
// ch.ack(msg)
}
}, { noAck: false })
}
}
require('amqplib/callback_api')
.connect('amqp://guest:guest#localhost', function (err, conn) {
if (err != null) bail(err)
consumer(conn)
})
ES version - 2.3.5 , Logstash - 2.4
'Attempted to send bulk request to Elasticsearch, configured at ["xxxx.com:9200"] ,
An error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided ?
Error:
"SSL peer shut down incorrectly", Manticore::ClientProtocolException
logstash"'
My logstash Output section:
output
{
stdout { codec => rubydebug }
stdout { codec => json }
elasticsearch
{
user => "xxxx"
password => "xxx"
index => "wrike_jan"
document_type => "data"
hosts => ["xxxx.com:9200"]
ssl => true
ssl_certificate_verification => false
truststore => "elasticsearch-2.3.5/config/truststore.jks"
truststore_password => "83dfcdddxxxxx"
}
}
Logstash file is executed , but it is failing to send the data to ES.
Could you please suggest, thank you.
Be particular about http or https in the url, in the above case i am sending data to https but my ES is using http.
Later, upgrade of logstash version solved to send data to ES.
I am working on a fun project which requires me to learn message queues and websockets. I am trying to connect browsers via websockets to an instance of rabbitmq using sockjs rather than pure websockets. On rabbit I have activated the plugins for stomp and web_stomp (web_stomp is required when using sockjs).
The problem I am running into is that while the call from the browser seems to be working properly because a very brief connection to Rabbit is made through the webstomp/stomp connection but after 2 or 3 seconds the connection is dropped by Rabbit.
This is confirmed by the rabbitmq logs:
=INFO REPORT==== 11-Jul-2016::23:01:54 ===
accepting STOMP connection (192.168.1.10:49746 -> 192.168.1.100:55674)
=INFO REPORT==== 11-Jul-2016::23:02:02 ===
closing STOMP connection (192.168.1.10:49746 -> 192.168.1.100:55674)
This is the browser code that connects to RabbitMQ via the webstomp plugin:
var url = "http://192.168.1.100:55674/stomp";
var ws = new SockJS(url);
var client = Stomp.over(ws);
var header = {
login: 'test',
passcode: 'test'
};
client.connect(header,
function(){
console.log('Hooray! Connected');
},
function(error){
console.log('Error connecting to WS via stomp:' + JSON.stringify(error));
}
);
Here is the Rabbit config:
[
{rabbitmq_stomp, [{default_user, [{login, "test"},
{passcode, "test"}
]
},
{tcp_listeners, [{"192.168.1.100", 55674}]},
{heartbeat, 0}
]
}
]
I have been over the Rabbit docs a million times but this feels like something simple that I am overlooking.
Resolved. After combing through the logs I realized that web_stomp was listening on port 15674 so I changed the config file to reflect that. I swear I had made that change at some point but it did not seem to make a difference.
One of the late changes I made before sending out my request was to turn off heartbeat. Everything I have read states that sockjs does not support heartbeat and that there were suggestions to turn it off rather than use the default. In addition to turning off heartbeat in the config file I also added this to the browser code:
client.heartbeat.outgoing=0;
client.heartbeat.incoming=0;
can activeMQ work with logstash?
I was switching from rabbitMQ to activeMQ, and trying to make logstash to work with activeMQ..
In my previous rabbitMQ, I have something like:
input {
rabbitmq {
host => "hostname"
queue => "queue1"
key => "key1"
exchange => "ex1"
type => "all"
durable => true
auto_delete => false
exclusive => false
format => "json_event"
debug => false
}
}
filter {....}
on logstash webpage -> doc, it does not show activeMQ supported as input...
http://logstash.net/docs/1.4.1/
any suggestions?
You can probably use (not tried it myself) the STOMP input. ActiveMQ supports stomp.