Filebeat not read all logs from directory - filebeat

Question

I am configuring filebeat to send to elastic logs located in /var/log/myapp/batch_*
Here my filebeat configuration:
# Version
filebeat version 7.11.0 (amd64), libbeat 7.11.0 [84c4d4c4034fcb49c1a318ccdc7311d70adee15b built 2021-02-08 22:42:11 +0000 UTC]
# Filebeat config
logging.metrics.period: 1h
logging.to_files: true
logging.files:
rotateeverybytes: 16777216
keepfiles: 7
permissions: 0600
filebeat.inputs:
- type: log
enabled: true
scan_frequency: 5m
paths:
- "/var/log/myapp/batch_*"
output.elasticsearch:
hosts: ["server:9200"]
index: "log_test_app-%{+yyyy.MM.dd}"
setup.ilm.enabled: false
setup.template.name: "log_test_app"
setup.template.pattern: "log_test_app-*"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 3
index.number_of_replicas: 1
I only see that two logs are being sent and within the established directory there are a total of eight logs:
2022-05-24T19:39:55.904Z INFO log/input.go:157 Configured paths: [/var/log/myapp/batch_*]
2022-05-24T19:39:55.904Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 3328309751929357009)
2022-05-24T19:39:55.904Z INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1
2022-05-24T19:44:55.905Z INFO log/harvester.go:302 Harvester started for file: /var/log/myapp/batch_emails.log
2022-05-24T19:44:55.905Z INFO log/harvester.go:302 Harvester started for file: /var/log/loyalty/batch_import.log
I show you a list of directory files:
ls -l /var/log/loyalty/batch_*
-rw-r--r-- 1 batch batch 154112 May 24 03:20 /var/log/myapp/batch_gifts.log
-rw-r--r-- 1 batch batch 112319 May 24 02:30 /var/log/myapp/batch_http.log
-rw-r--r-- 1 batch batch 7575342 May 24 02:30 /var/log/myapp/batch_vouchers.log
-rw-r--r-- 1 batch batch 4847849 May 24 19:30 /var/log/myapp/batch_ftp.log
-rw-r--r-- 1 batch batch 99413 May 24 03:40 /var/log/myapp/batch_category.log
-rw-r--r-- 1 root root 367207 May 24 19:50 /var/log/myapp/batch_emails.log
-rw-r--r-- 1 batch batch 479 Jan 1 23:00 /var/log/myapp/batch_history.log
-rw-r--r-- 1 batch batch 2420916 Jan 1 23:00 /var/log/myapp/batch_lists.php
-rw-r--r-- 1 batch batch 25779499 May 24 19:50 /var/log/myapp/batch_import.log
Is there something wrong with my setup? I tried using the ignore_older parameter: 36h but only two log files are processed.
Thanks for the help.

Answer 1

Welcome to stack overflow, Emanuel :)
I believe you are reading only the log files (*.log) thus you can mention the same.
filebeat.inputs:
- type: log
enabled: true
scan_frequency: 5m
paths:
- "/var/log/myapp/batch_*.log"
Keep Posted!!! Thanks!!!