kafka{
topic_id => "myTopic"
bootstrap_servers => "127.0.0.1:9092"
value_serializer => "io.confluent.kafka.serializers.KafkaAvroSerializer"
}
}
[[main]-pipeline-manager] kafka - Unable to create Kafka producer from given configuration {:kafka_error_message=>org.apache.kafka.common.config.ConfigException: Invalid value io.confluent.kafka.serializers.KafkaAvroSerializer for configuration value.serializer: Class io.confluent.kafka.serializers.KafkaAvroSerializer could not be found., :cause=>nil}
Has anyone made logstash work with io.confluent.kafka.serializers.KafkaAvroSerializer ?
You'll need to use the ByteArraySerializer and install this codec
https://github.com/revpoint/logstash-codec-avro_schema_registry
Related
ES version - 2.3.5 , Logstash - 2.4
'Attempted to send bulk request to Elasticsearch, configured at ["xxxx.com:9200"] ,
An error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided ?
Error:
"SSL peer shut down incorrectly", Manticore::ClientProtocolException
logstash"'
My logstash Output section:
output
{
stdout { codec => rubydebug }
stdout { codec => json }
elasticsearch
{
user => "xxxx"
password => "xxx"
index => "wrike_jan"
document_type => "data"
hosts => ["xxxx.com:9200"]
ssl => true
ssl_certificate_verification => false
truststore => "elasticsearch-2.3.5/config/truststore.jks"
truststore_password => "83dfcdddxxxxx"
}
}
Logstash file is executed , but it is failing to send the data to ES.
Could you please suggest, thank you.
Be particular about http or https in the url, in the above case i am sending data to https but my ES is using http.
Later, upgrade of logstash version solved to send data to ES.
can activeMQ work with logstash?
I was switching from rabbitMQ to activeMQ, and trying to make logstash to work with activeMQ..
In my previous rabbitMQ, I have something like:
input {
rabbitmq {
host => "hostname"
queue => "queue1"
key => "key1"
exchange => "ex1"
type => "all"
durable => true
auto_delete => false
exclusive => false
format => "json_event"
debug => false
}
}
filter {....}
on logstash webpage -> doc, it does not show activeMQ supported as input...
http://logstash.net/docs/1.4.1/
any suggestions?
You can probably use (not tried it myself) the STOMP input. ActiveMQ supports stomp.
all.
I am using logstash-1.4.2 to consume messages stored in my activeMQ(with stomp plubgin).
in my acitveMQ.xml config file, I have the line:
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
when I run my logstash, I have this error:
C:\logstash\logstash-1.4.2\bin>logstash agent -f logstashconfig.conf
+---------------------------------------------------------+
| An unexpected error occurred. This is probably a bug. |
| You can find help with this problem in a few places: |
| |
| * chat: #logstash IRC channel on freenode irc. |
| IRC via the web: http://goo.gl/TI4Ro |
| * email: logstash-users#googlegroups.com |
| * bug system: https://logstash.jira.com/ |
| |
+---------------------------------------------------------+
The error reported is:
Couldn't find any input plugin named 'stomp'. Are you sure this is correct? Trying to load the stomp input plugin resulted in this error: no such file to load -- logstash/inputs/stomp
in my logstashconfig.conf, I have :
input {
stomp {
password => "admin"
user => "admin"
}
}
output {
file {
path => "C:\logstash\logstash-1.4.2\cosumedfromstomp.txt"
}
}
If I consume from rabbitMQ, with the following logstashconfig.conf, it would be correct (here is my rabbitMQ version of config):
input {
rabbitmq {
host => "amqp"
queue => "elasticsearch"
key => "elasticsearch"
exchange => "elasticsearch"
type => "all"
durable => true
auto_delete => false
exclusive => false
format => "json_event"
debug => false
}
}
output {
file {
path => "C:\logstash\logstash-1.4.2\cosumedfromstomp.txt"
}
}
I don't have trouble with my rabbitMQ version of logstash, the text file created looks correct.
My question is :
1, do I configure my stomp input wrong? since I don't have the "queue" name in my config, it might be wrong?
2, or if the problem is I didn't create the stomp plugin correctly, if that is the reason, it would not be about logstash...
Thanks
You need to install the Contributed Plugins. These have been removed from the core download for Logstash. The Stomp plugin is located in the contributed plugins:
Stomp
Milestone: 2
This is a community-contributed plugin! It does not ship with logstash
by default, but it is easy to install! To use this, you must have
installed the contrib plugins package.
Directions here:
http://logstash.net/docs/1.4.2/contrib-plugins
Hosted on GitHub here:
https://github.com/elasticsearch/logstash-contrib
I am using Logstash, Redis DB, ElasticSearch and Kibana 3 for my centalize log server. It's working fine and I am able to see the logs in Kibana. Now I want to keep only 30 days log in ElasticSearch and Redis Server. Is it possible to purge data from Redis?
I am using the below configuration
indexer.conf
input {
redis {
host => "127.0.0.1"
port => 6379
type => "redis-input"
data_type => "list"
key => "logstash"
format => "json_event"
}
}
output {
stdout { debug => true debug_format => "json"}
elasticsearch {
host => "127.0.0.1"
}
}
shipper.conf
input {
file {
type => "nginx_access"
path => ["/var/log/nginx/**"]
exclude => ["*.gz", "error.*"]
discover_interval => 10
}
}
filter {
grok {
type => nginx_access
pattern => "%{COMBINEDAPACHELOG}"
}
}
output {
stdout { debug => true debug_format => "json"}
redis { host => "127.0.0.1" data_type => "list" key => "logstash" }
}
As per this configuration the shipper file is sending data to Redis DB with the key "logstash". From the redis db documents I came to know that we can set TTL for any key with expire command to purge them. But when I am searching for the key "logstash" in redis db keys logstash or keys *I am not getting any result. Please let me know if my question is not understandable. Thanks in advance.
Redis is a key:value store. Keys are unique by definition. So if you want to store several logs, you need to add a new entry, with a new key and associated value, for each log.
So it seems to me you have a fundamental flaw here, as you're always using the same key for all your logs. Try with a different key for each log (not sure how to do that).
Then set TTL to 30 days.
We're receiving logs using Logstash with the following configuration:
input {
udp {
type => "logs"
port => 12203
}
}
filter {
grok {
type => "tracker"
pattern => '%{GREEDYDATA:message}'
}
date {
type => "tracker"
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss,SSS" ]
}
}
output{
tcp{
type => "logs"
host => "host"
port => 12203
}
}
We're then picking the logs up on the machine "host" with the following settings:
input {
tcp {
type => "logs"
port => 12203
}
}
output {
pipe {
command => "python /usr/lib/piperedis.py"
}
}
From here, we're doing parsing of the lines and putting them into a Redis database. However, we've discovered an interesting problem.
Logstash 'wraps' the log message in a JSON style package i.e.:
{\"#source\":\"source/\",\"#tags\":[],\"#fields\":{\"timestamp\":[\"2013-09-16 15:50:47,440\"],\"thread\":[\"ajp-8009-7\"],\"level\":[\"INFO\"],\"classname\":[\"classname\"],\"message\":[\"message"\]}}
We then, on receiving it and passing it on on the next machine, take that as the message and put it in another wrapper! We're only interested in the actual log message and none of the other stuff (source path, source, tags, fields, timestamp e.t.c.)
Is there a way we can use filters or something to do this? We've looked through the documentation but can't find any way to just pass the raw log lines between instances of Logstash.
Thanks,
Matt
The logstash documentation is wrong - it indicates that the default "codec" is plain but in fact it doesn't use a codec - it uses an output format.
To get a simpler output, change your output to something like
output {
pipe {
command => "python /usr/lib/piperedis.py"
message_format => "%{message}"
}
}
Why not just extract those messages from stdout?
line = sys.stdin.readline()
line_json = json.loads(line)
line_json['message'] # will be your #message