Filebeat autodiscover doesn't work with nomad - filebeat

Question

I'm having some trouble to configure autodiscover to run with nomad, to get allocation logs from the nomad agent and send to logstash. Everytime i got this message in filebeat logs.
"message":"Non-zero metrics in the last 30s"
but in nomad a have an job that is generating logs every 60 seconds.
This is my filebeat config:
#### debug mode
loggin.info: debug
#### logstash output
output.logstash:
hosts: [ "xxx.xxx.xxx.xxx:5055"]
slow_start: true
### autodicover
filebeat.autodiscover:
providers:
- type: nomad
scope: cluster
hints.enabled: true
allow_stale: true
templates:
config:
- type: log
paths:
- /opt/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/*
This is the filebeat logs:
{"log.level":"info","#timestamp":"2022-11-10T19:35:15.049Z","log.origin":{"file.name":"instance/beat.go","file.line":293},"message":"Setup Beat: filebeat; Version: 8.5.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","#timestamp":"2022-11-10T19:35:15.054Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: ip-10-202-9-107.ec2.internal","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","#timestamp":"2022-11-10T19:35:15.055Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","#timestamp":"2022-11-10T19:35:15.056Z","log.origin":{"file.name":"beater/filebeat.go","file.line":162},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","#timestamp":"2022-11-10T19:35:15.057Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","#timestamp":"2022-11-10T19:35:15.058Z","log.origin":{"file.name":"instance/beat.go","file.line":470},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","#timestamp":"2022-11-10T19:35:15.059Z","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","#timestamp":"2022-11-10T19:35:15.059Z","log.origin":{"file.name":"beater/filebeat.go","file.line":288},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","#timestamp":"2022-11-10T19:35:15.060Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","#timestamp":"2022-11-10T19:35:15.061Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","#timestamp":"2022-11-10T19:35:15.061Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","#timestamp":"2022-11-10T19:35:15.062Z","log.logger":"cfgwarn","log.origin":{"file.name":"nomad/nomad.go","file.line":58},"message":"EXPERIMENTAL: The nomad autodiscover provider is experimental.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","#timestamp":"2022-11-10T19:35:15.063Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","#timestamp":"2022-11-10T19:35:15.064Z","log.logger":"autodiscover","log.origin":{"file.name":"autodiscover/autodiscover.go","file.line":118},"message":"Starting autodiscover manager","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","#timestamp":"2022-11-10T19:35:15.696Z","log.logger":"nomad","log.origin":{"file.name":"nomad/watcher.go","file.line":192},"message":"Watching API for resource events","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","#timestamp":"2022-11-10T19:35:45.066Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":186},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"filebeat.service"},"cpuacct":{"id":"filebeat.service","total":{"ns":650388214}},"memory":{"id":"filebeat.service","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":33452032}}}},"cpu":{"system":{"ticks":70,"time":{"ms":70}},"total":{"ticks":640,"time":{"ms":640},"value":640},"user":{"ticks":570,"time":{"ms":570}}},"handles":{"limit":{"hard":65535,"soft":65535},"open":10},"info":{"ephemeral_id":"6aa6ed52-c270-4b8e-8f35-48feb7d77ebd","name":"filebeat","uptime":{"ms":30155},"version":"8.5.0"},"memstats":{"gc_next":21512168,"memory_alloc":15031152,"memory_sys":28918792,"memory_total":69080704,"rss":114196480},"runtime":{"goroutines":23}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"load":{"1":0.09,"15":0.67,"5":0.38,"norm":{"1":0.045,"15":0.335,"5":0.19}}}},"ecs.version":"1.6.0"}}
This is my nomad job:
job "template" {
datacenters = ["us-east-1"]
type = "batch"
periodic {
cron = "* * * * *"
}
group "group" {
count = 1
task "command" {
driver = "exec"
config {
command = "bash"
args = ["-c", "cat local/template.out"]
}
template {
data = <<EOH
node.unique.id: {{ env "node.unique.id" }}
node.datacenter: {{ env "node.datacenter" }}
node.unique.name: {{ env "node.unique.name" }}
node.class: {{ env "node.class" }}
attr.cpu.arch: {{ env "attr.cpu.arch" }}
attr.cpu.numcores: {{ env "attr.cpu.numcores" }}
attr.cpu.totalcompute: {{ env "attr.cpu.totalcompute" }}
attr.consul.datacenter: {{ env "attr.consul.datacenter" }}
attr.unique.hostname: {{ env "attr.unique.hostname" }}
attr.unique.network.ip-address: {{ env "attr.unique.network.ip-address" }}
attr.kernel.name: {{ env "attr.kernel.name" }}
attr.kernel.version: {{ env "attr.kernel.version" }}
attr.platform.aws.ami-id: {{ env "attr.platform.aws.ami-id" }}
attr.platform.aws.instance-type: {{ env "attr.platform.aws.instance-type" }}
attr.os.name: {{ env "attr.os.name" }}
attr.os.version: {{ env "attr.os.version" }}
NOMAD_ALLOC_DIR: {{env "NOMAD_ALLOC_DIR"}}
NOMAD_TASK_DIR: {{env "NOMAD_TASK_DIR"}}
NOMAD_SECRETS_DIR: {{env "NOMAD_SECRETS_DIR"}}
NOMAD_MEMORY_LIMIT: {{env "NOMAD_MEMORY_LIMIT"}}
NOMAD_CPU_LIMIT: {{env "NOMAD_CPU_LIMIT"}}
NOMAD_ALLOC_ID: {{env "NOMAD_ALLOC_ID"}}
NOMAD_ALLOC_NAME: {{env "NOMAD_ALLOC_NAME"}}
NOMAD_ALLOC_INDEX: {{env "NOMAD_ALLOC_INDEX"}}
NOMAD_TASK_NAME: {{env "NOMAD_TASK_NAME"}}
NOMAD_GROUP_NAME: {{env "NOMAD_GROUP_NAME"}}
NOMAD_JOB_NAME: {{env "NOMAD_JOB_NAME"}}
NOMAD_DC: {{env "NOMAD_DC"}}
NOMAD_REGION: {{env "NOMAD_REGION"}}
VAULT_TOKEN: {{env "VAULT_TOKEN"}}
GOMAXPROCS: {{env "GOMAXPROCS"}}
HOME: {{env "HOME"}}
LANG: {{env "LANG"}}
LOGNAME: {{env "LOGNAME"}}
NOMAD_ADDR_export: {{env "NOMAD_ADDR_export"}}
NOMAD_ADDR_exstat: {{env "NOMAD_ADDR_exstat"}}
NOMAD_ALLOC_DIR: {{env "NOMAD_ALLOC_DIR"}}
NOMAD_ALLOC_ID: {{env "NOMAD_ALLOC_ID"}}
NOMAD_ALLOC_INDEX: {{env "NOMAD_ALLOC_INDEX"}}
NOMAD_ALLOC_NAME: {{env "NOMAD_ALLOC_NAME"}}
NOMAD_CPU_LIMIT: {{env "NOMAD_CPU_LIMIT"}}
NOMAD_DC: {{env "NOMAD_DC"}}
NOMAD_GROUP_NAME: {{env "NOMAD_GROUP_NAME"}}
NOMAD_HOST_PORT_export: {{env "NOMAD_HOST_PORT_export"}}
NOMAD_HOST_PORT_exstat: {{env "NOMAD_HOST_PORT_exstat"}}
NOMAD_IP_export: {{env "NOMAD_IP_export"}}
NOMAD_IP_exstat: {{env "NOMAD_IP_exstat"}}
NOMAD_JOB_NAME: {{env "NOMAD_JOB_NAME"}}
NOMAD_MEMORY_LIMIT: {{env "NOMAD_MEMORY_LIMIT"}}
NOMAD_PORT_export: {{env "NOMAD_PORT_export"}}
NOMAD_PORT_exstat: {{env "NOMAD_PORT_exstat"}}
NOMAD_REGION: {{env "NOMAD_REGION"}}
NOMAD_SECRETS_DIR: {{env "NOMAD_SECRETS_DIR"}}
NOMAD_TASK_DIR: {{env "NOMAD_TASK_DIR"}}
NOMAD_TASK_NAME: {{env "NOMAD_TASK_NAME"}}
PATH: {{env "PATH"}}
PWD: {{env "PWD"}}
SHELL: {{env "SHELL"}}
SHLVL: {{env "SHLVL"}}
USER: {{env "USER"}}
VAULT_TOKEN: {{env "VAULT_TOKEN"}}
concat key: service/fabio/{{ env "NOMAD_JOB_NAME" }}/listeners
key: {{ keyOrDefault ( printf "service/fabio/%s/listeners" ( env "NOMAD_JOB_NAME" ) ) ":9999" }} {{ define "custom" }}service/fabio/{{env "NOMAD_JOB_NAME" }}/listeners{{ end }}
key: {{ keyOrDefault (executeTemplate "custom") ":9999" }} math - alloc_id + 1: {{env "NOMAD_ALLOC_INDEX" | parseInt | add 1}}
EOH
destination = "local/template.out"
}
}
}
}
Can you help me understand why autodiscover doesn't work with nomad?