rabbitmq / logstash lost message - rabbitmq

Question

I have a rabbitmq which stores message successfully but my logstash reading the queue ignore most of my messages.
RabbitMQ is OK, I have a small python script to display all messages
import pika
i=0
def on_message(channel, method_frame, header_frame, body):
global i
print i
print("Message body", body)
channel.basic_ack(delivery_tag=method_frame.delivery_tag)
i+=1
credentials = pika.PlainCredentials('***', '***')
parameters = pika.ConnectionParameters('***',5672,'logstash', credentials=credentials)
connection = pika.BlockingConnection(parameters)
channel = connection.channel()
channel.exchange_declare(exchange="logstash", exchange_type="topic", passive=False, durable=True, auto_delete=False)
channel.queue_declare(queue="hbbtv", auto_delete=False, durable=True)
channel.queue_bind(queue="hbbtv", exchange="logstash", routing_key="hbbtv")
channel.basic_qos(prefetch_count=1)
channel.basic_consume(on_message, 'hbbtv')
try:
channel.start_consuming()
except KeyboardInterrupt:
channel.stop_consuming()
connection.close()
I can see all my messages
12 ('Message body', '{"message":"212.95.70.118 - -
[25/Feb/2016:11:19:53 +0100] \"GET
/services/web/index.php/OPA/categories/ARTEPLUS7/fr HTTP/1.1\" 200
348
\"http://www.arte.tv/hbbtvv2/notv/cehtml/index.cehtml?lang=de_DE&page=PLUS7&tv=false\"
\"Opera/9.80 (Linux armv7l; HbbTV/1.1.1 (; Philips; ; ; PhilipsTV; )
CE-HTML/1.0 NETTV/4.3.1 PhilipsTV/2.1.1 Firmware/003.015.000.001
(PhilipsTV, 2.1.1,) en) Presto/2.12.362 Version/12.11 \"
hbbtvdyn.arte.tv","#version":"1","#timestamp":"2016-02-25T10:19:53.000Z","path":"/data/logs/access","host":"arte-hbbtvdyn-web1.sdv.fr","type":"apache-access","application":"hbbtv","clientip":"212.95.70.118","ident":"-","auth":"-","timestamp":"25/Feb/2016:11:19:53 +0100","verb":"GET","request":"/services/web/index.php/OPA/categories/ARTEPLUS7/fr","httpversion":"1.1","response":"200","bytes":"348","referrer":"\"http://www.arte.tv/hbbtvv2/notv/cehtml/index.cehtml?lang=de_DE&page=PLUS7&tv=false\"","agent":"\"Opera/9.80
(Linux armv7l; HbbTV/1.1.1 (; Philips; ; ; PhilipsTV; ) CE-HTML/1.0
NETTV/4.3.1 PhilipsTV/2.1.1 Firmware/003.015.000.001 (PhilipsTV,
2.1.1,) en) Presto/2.12.362 Version/12.11 \"","targethost":"hbbtvdyn.arte.tv","geoip":{"ip":"212.95.70.118","country_code2":"FR","country_code3":"FRA","country_name":"France","continent_code":"EU","region_name":"C1","city_name":"Strasbourg","latitude":48.60040000000001,"longitude":7.787399999999991,"timezone":"Europe/Paris","real_region_name":"Alsace","location":[7.787399999999991,48.60040000000001]}}')
13 ('Message body', '{"message":"212.95.70.118 - -
[25/Feb/2016:11:19:53 +0100] \"GET
/services/web/index.php/OPA/videos/highlights/6/ARTEPLUS7/de/GE
HTTP/1.1\" 500 4519
\"http://www.arte.tv/hbbtvv2/notv/cehtml/index.cehtml?lang=de_DE&page=PLUS7&tv=false\"
\"Opera/9.80 (Linux armv7l; HbbTV/1.1.1 (; Philips; ; ; PhilipsTV; )
CE-HTML/1.0 NETTV/4.3.1 PhilipsTV/2.1.1 Firmware/003.015.000.001
(PhilipsTV, 2.1.1,) en) Presto/2.12.362 Version/12.11 \"
hbbtvdyn.arte.tv","#version":"1","#timestamp":"2016-02-25T10:19:53.000Z","path":"/data/logs/access","host":"arte-hbbtvdyn-web1.sdv.fr","type":"apache-access","application":"hbbtv","clientip":"212.95.70.118","ident":"-","auth":"-","timestamp":"25/Feb/2016:11:19:53 +0100","verb":"GET","request":"/services/web/index.php/OPA/videos/highlights/6/ARTEPLUS7/de/GE","httpversion":"1.1","response":"500","bytes":"4519","referrer":"\"http://www.arte.tv/hbbtvv2/notv/cehtml/index.cehtml?lang=de_DE&page=PLUS7&tv=false\"","agent":"\"Opera/9.80
(Linux armv7l; HbbTV/1.1.1 (; Philips; ; ; PhilipsTV; ) CE-HTML/1.0
NETTV/4.3.1 PhilipsTV/2.1.1 Firmware/003.015.000.001 (PhilipsTV,
2.1.1,) en) Presto/2.12.362 Version/12.11 \"","targethost":"hbbtvdyn.arte.tv","geoip":{"ip":"212.95.70.118","country_code2":"FR","country_code3":"FRA","country_name":"France","continent_code":"EU","region_name":"C1","city_name":"Strasbourg","latitude":48.60040000000001,"longitude":7.787399999999991,"timezone":"Europe/Paris","real_region_name":"Alsace","location":[7.787399999999991,48.60040000000001]}}')
14 ('Message body', '{"message":"212.95.70.119 - -
[25/Feb/2016:11:19:53 +0100] \"GET
/OPA/getOPAData.php?url=videoStreams%3Flanguage%3Dfr%26protocol%3DHTTP%26mediaType%3Dmp4%26quality%3DEQ%2CSQ%2CHQ%26profileAmm%3D%24nin%3AAMM-YTFR-HAB%2CAMM-YTFR%2CAMM-DT%26kind%3DSHOW%26availableScreens%3Dtv%26fields%3DprogramId%2Curl%2Cquality%2CaudioSlot%2CaudioCode%2CaudioLabel%2CaudioShortLabel%2Cchannel%26programId%3D048353-033-A%26platform%3DARTEPLUS7&filename=PLUS7_stream_048353-033-A_fr_FR.json
HTTP/1.1\" 200 5508 \"-\" \"Mozilla/5.0 (Linux; Tizen 2.3;
SmartHub; SMART-TV; SmartTV; U; Maple2012) AppleWebKit/538.1+ (KHTML,
like Gecko) TV Safari/538.1+ \"
hbbtvdyn.arte.tv","#version":"1","#timestamp":"2016-02-25T10:19:53.000Z","path":"/data/logs/access","host":"arte-hbbtvdyn-web1.sdv.fr","type":"apache-access","application":"hbbtv","clientip":"212.95.70.119","ident":"-","auth":"-","timestamp":"25/Feb/2016:11:19:53 +0100","verb":"GET","request":"/OPA/getOPAData.php?url=videoStreams%3Flanguage%3Dfr%26protocol%3DHTTP%26mediaType%3Dmp4%26quality%3DEQ%2CSQ%2CHQ%26profileAmm%3D%24nin%3AAMM-YTFR-HAB%2CAMM-YTFR%2CAMM-DT%26kind%3DSHOW%26availableScreens%3Dtv%26fields%3DprogramId%2Curl%2Cquality%2CaudioSlot%2CaudioCode%2CaudioLabel%2CaudioShortLabel%2Cchannel%26programId%3D048353-033-A%26platform%3DARTEPLUS7&filename=PLUS7_stream_048353-033-A_fr_FR.json","httpversion":"1.1","response":"200","bytes":"5508","referrer":"\"-\"","agent":"\"Mozilla/5.0
(Linux; Tizen 2.3; SmartHub; SMART-TV; SmartTV; U; Maple2012)
AppleWebKit/538.1+ (KHTML, like Gecko) TV Safari/538.1+
\"","targethost":"hbbtvdyn.arte.tv","geoip":{"ip":"212.95.70.119","country_code2":"FR","country_code3":"FRA","country_name":"France","continent_code":"EU","region_name":"C1","city_name":"Strasbourg","latitude":48.60040000000001,"longitude":7.787399999999991,"timezone":"Europe/Paris","real_region_name":"Alsace","location":[7.787399999999991,48.60040000000001]}}')
with the good rate message (several per seconds) and I have absolutely not grok parse failure.
So the issue happen while logstash read message. Problems are
a lot of message missing
all message have _grokparsefailure even if
it's complete
The input part of logstash is
rabbitmq {
host=>"arte-elasticlog.sdv.fr"
user=>"***"
password=>"***"
queue=>"hbbtv"
vhost=>"logstash"
port=>5672
auto_delete=>false
durable=>true
type => "rabbit_hbbtv"
}

Answer 1

_grokparsefailure indicates that it is unable to parse the message. meaning the message successfully read from the queue, but your grok expression has problem or could not be applied on your message content.
One more thing is the default codec of the rabbitmq input is "json", if your rabbitmq message content is not json, you should set the codec of your input to for instance:
codec => plain {}

Answer 2

The problem was due to my logstash filter,
I had two apache access but with different pattern so when logstash tried to parsee a message, sometimes he had the good pattern --> in ES and sometimes not --> no message.
Now, for all my different log, I add (add-field)
application-->"my application name"
in my input and all my grok filter depends on the application.
All is good now, thanks for your help.