I set filebeat to read logs from rotating logs (rotated when 5 mg is reached) and below is my config :
- type: log
fields:
source: 'filebeat2'
logID: logbackup
fields_under_root: true
enabled: true
paths:
- /home/logbackup/a.log
-/home/logbackup/backup/a.log-*
output.logstash:
# The Logstash hosts
hosts: ["ip:5044"]
worker: 4
bulk_max_size: 4096
queue:
mem:
events: 16384
and logstash.yml :
pipeline.workers: 4
pipeline.batch.size: 4096
and close-inactive is default(5min).we have 100 transaction per second .I stop filebeat manually for specific time(for crash test) and when start it manually (with 2 million docs stored in second directory path )and some logs been duplicated.
what is the solution ,is it possible solution to increase close-inactive time ?
Related
I am trying to send data from filebeat to wazuh-indexer directly but I get connection errors between filebeat and elasticsearch. Following is my filebeat configuration:
filebeat.inputs:
- input_type: log
paths:
- /home/siem/first4.log
enable: true
output.elasticsearch:
hosts: ["192.168.0.123:9200"]
protocol: https
index: "test"
username: admin
password: admin
ssl.certificate_authorities:
- /etc/filebeat/certs/root-ca.pem
ssl.certificate: "/etc/filebeat/certs/filebeat-1.pem"
ssl.key: "/etc/filebeat/certs/filebeat-1-key.pem"
setup.template.json.enabled: false
setup.ilm.overwrite: true
setup.ilm.enabled: false
setup.template.name: false
setup.template.pattern: false
#setup.template.json.path: '/etc/filebeat/wazuh-template.json'
#setup.template.json.name: 'wazuh'
#filebeat.modules:
# - module: wazuh
# alerts:
# enabled: true
# archives:
# enabled: false
Following is the error:
2023-01-30T09:29:18.634Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://192.168.0.123:9200)): Get "https://192.168.0.123:9200": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2023-01-30T09:29:18.635Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://192.168.0.123:9200)) with 1 reconnect attempt(s)
2023-01-30T09:29:18.635Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2023-01-30T09:29:18.635Z INFO [publisher] pipeline/retry.go:223 done
2023-01-30T09:29:46.177Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s
Can anyone tell what mistake am I doing?
Yes, you could send logs directly using Filebeat without a Wazuh agent but that way you won't benefit from the Wazuh analysis engine.
With your current configuration, the logs will be ingested under filebeat-<version>-<date>. Make sure to create an index pattern for these events.
As your logs indicate, there's a connectivity issue between Filebeat and the Wazuh indexer. To diagnose the problem:
Try running the following call to make sure you can reach the Wazuh indexer:
curl -k -u admin:admin https://192.168.0.123:9200
Run a Filebeat test output:
filebeat test output
Despite looking at many posts on SO and Ansible's doc, I'm still failing at understanding what Ansible is doing.
My scenario is following: I need to rename the network interface Ansible is connected over to control the remote and restore connection.
My first attempts revolved around something like this:
- name: Hot Rename Main Iface
become: true
shell:
cmd: |
ip link set oldiface down
ip link set oldiface name newiface
ip link set newiface up
async: 0
poll: 0
register: asynchotrename
- name: Wait For Reconnection
wait_for_connection:
delay: 15
timeout: 180
But whatever the values I would set for async or poll, Ansible would hang indefinitely. On the remote, I could see that the interface was brought down and then nothing. So obviously, nothing was done asynchronously, and as soon as the interface was down, the script could not continue. Probably, the process was killed by the termination of the ssh session.
Then I read that when doing this, Ansible had no time to properly spawn the process and disconnect. It needed the process to wait a bit before cutting the connection short. So I modified the playbook:
- name: Hot Rename Main Iface
become: true
shell:
cmd: |
sleep 5 # <-- Wait for Ansible disconnection
ip link set oldiface down
ip link set oldiface name newiface
ip link set newiface up
async: 0
poll: 0
register: asynchotrename
- name: Wait For Reconnection
wait_for_connection:
delay: 15
timeout: 180
But this did nothing. Ansible still hangs indefinitely, while nothing happens on the remote after the ip link down statement.
Then, I figured out that maybe I had to force send the subprocess to the background, even if this would mean not making use of Ansible's asynchronous feature and so not being able to possibly come back later to check if everything went fine (although of course if that's the case, chances are that the remote is unreachable anyway). I still kept the async and poll values, just to ensure that Ansible would disconnect properly, even if obviously it would do this only once the script had returned. At least, this would prevent some errors that I would have to mask with ignore_errors: true.
I may try without someday, to see if I can just remove these async and poll entirely. (Edit: Done, and it works. No errors to mask.)
The complete playbooks steps ended being (for those interrested, although I'm not going to explain in this post why I had to order the statements this way):
- name: Hot Rename Main Iface
become: true
shell:
cmd: |
(
sleep 5 && \
ip link set oldiface down && \
ip link set oldiface name newiface && \
ip link set newiface up && \
nmcli networking off && \
sleep 1 && \
nmcli networking on && \
sleep 5 && \
systemctl restart sshd
)&
async: 90
poll: 0
register: asynchotrename
- name: Wait For Reconnection
wait_for_connection:
delay: 15
timeout: 180
But then I read that if I use poll: 0, I have to manually cleanup the async job cache. So I added this task:
- name: Cleanup Leftover Async Files
async_status:
jid: "{{ asynchotrename.ansible_job_id }}"
mode: cleanup
result: FAILED! => {"ansible_job_id": "603790343886.29503", "changed": false, "finished": 1, "msg": "could not find job", "started": 1}
I'm totally puzzled. Ansible doesn't even seem to consider the task as an async job.
How to spawn an asynchronous task in Ansible??
During research regarding Ansible doesn't return job_id for async task I've setup a small test on a RHEL 7.9.9 system with Ansible 2.9.25 and Python 2.7.5 which seems to be working so far.
- name: Start async job
systemd:
name: network
state: restarted
async: 60 # 1min
poll: 0
register: network_restart
- name: Wait shortly before check
pause:
seconds: 5
- name: Check async status
async_status:
jid: "{{ network_restart.ansible_job_id }}"
changed_when: false
register: job_result
until: job_result.finished
retries: 6
delay: 10
Because of your comment
Ansible had no time to properly spawn the process and disconnect. It needed the process to wait a bit before cutting the connection short.
and the documentation of Run tasks concurrently: poll = 0
If you want to run multiple tasks in a playbook concurrently, use async with poll set to 0. When you set poll: 0, Ansible starts the task and immediately moves on to the next task without waiting for a result.
I've included the
- name: Wait shortly before check
pause:
seconds: 5
resulting into an execution of
TASK [Start async job] *****************************************************************************************************************************************
changed: [test1.example.com]
Saturday 06 November 2021 17:20:43 +0100 (0:00:02.287) 0:00:10.228 *****
Pausing for 5 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
TASK [Wait shortly] ********************************************************************************************************************************************
ok: [test1.example.com]
Saturday 06 November 2021 17:20:48 +0100 (0:00:05.057) 0:00:15.285 *****
TASK [Check async status] **************************************************************************************************************************************
ok: [test1.example.com]
As you can see the pausing message came almost instantly and seconds before task name message.
On the test host it is seen that the network interface restarted
sudo systemctl status network
● network.service - LSB: Bring up/down networking
Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled)
Active: active (exited) since Sat 2021-11-06 17:20:46 CET; 874ms ago
Regarding
... as soon as the interface was down, the script could not continue. Probably, the process was killed by the termination of the ssh session.
I am too renaming interfaces, frequently during baseline setups like
- name: Make sure main network interface is named correctly
shell:
cmd: nmcli conn mod "ens192" connection.id "eth0"
- name: Gather current interface configuration
shell:
cmd: nmcli conn show eth0
register: nmcli_conn
- name: STDOUT nmcli_conn
debug:
msg: "{{ nmcli_conn.stdout_lines }}"
I have only to make sure before that the interfaces can be managed by NetworkManager. An asynchronous task isn't necessary in my setups to have a reliable restart of the network interfaces, also not for restarting sshd.
By using NetworkManager more advanced task are possilbe later like
- name: Configure DNS resolver
nmcli:
conn_name: eth0
type: ethernet
dns4_search:
- dns.example.com
state: present
In a AIX Server I’ve been trying to save my IBM Broker (version 8.0) logs, but they’re not getting logged at all. I’ve configured the syslog file using either tabs on the first space or simply separated the path/file and the facility “user” by spaces characters.
syslog.conf
user.info /var/mqsi/info.log rotate size 4m files 4
user.err /var/mqsi/err.log rotate size 4m files 4
Also tried:
user.info /var/mqsi/info.log rotate size 4m files 4
user.err /var/mqsi/err.log rotate size 4m files 4
Then run:
refresh -s syslogd
After that I waited a couple of hours an the files were still without any logs.
The info.log and err.log files have system & root permissions, as well as chmod 640 (write and read) configured.
I set up ELK stack and filebeat with my ELK node as a RedHat server following the digitalocean tutorial. Kibana is up and running, but I dont see any logstash indexes when I go to configure an index pattern as logstash-*:
Unable to fetch mapping. Do you have any indices matching the pattern?
When I do a curl to see the indexes I have, they are only filebeat indexes. Filebeat should be pushing data to logstash which is listening on 5044
$curl 'localhost:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open filebeat-2017.01.10 5 1 3864 0 1.7mb 1.7mb
yellow open filebeat-2017.06.17 5 1 1848 0 740.1kb 740.1kb
yellow open filebeat-2017.01.18 5 1 77062 0 33mb 33mb
yellow open filebeat-2017.09.14 5 1 1932 0 1.1mb 1.1mb
yellow open filebeat-2017.01.11 5 1 19094 0 3.6mb 3.6mb
yellow open .kibana
You can see I only have filebeat indexes. I checked my ports are open, and My config files are correct according to the tutorial. What could be wrong? Filebeat should be sending logs from /var/log/*.log to logstash, to elasticsearch.
When I
tail /var/log/logstash/logstash.log
there is nothing in my logstash log. I've checked and logstash, filebeat, kibana, and elasticsearch are all running. Ive also done the config file test and it said it was OK:
$sudo service logstash status
logstash is running
On my ELK node, I can clearly see the port 5044 is listening:
$ netstat -tulpn | grep -i listen | grep -v tcp6
tcp 0 0 :::5044 :::* LISTEN -
Filebeat creates daily indices using a pattern of filebeat-YYYY.MM.dd so you should not expect to see logstash indices in Elasticsearch.
The Logstash configuration recommended in the Filebeat documentation writes the data to an index based on "%{[#metadata][beat]}-%{+YYYY.MM.dd}" where [#metadata][beat] defaults to the name of the beat (filebeat) unless output.logstash.index is configured in the Filebeat config. Here's the base configuration for Logstash.
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
}
You can inspect the data in those indices to see if it's what you are expected to get from filebeat with a command like:
curl http://localhost:9200/filebeat-*/_search?pretty&size=100
Did you define your index in your Kibana, from Management > Index Patterns > Add New?
It's obvious that you won't be able to find the index which you've created using logstash in Kibana, unless you're manually creating it there within the Managemen section of Kibana.
Make sure, that you have the same name of the indice which you created using logstash. Have a look at the doc, which conveys:
When you define an index pattern, indices that match that pattern must
exist in Elasticsearch. Those indices must contain data.
which pretty much says that the indice should exist for you to create the index in Kibana. What logstash does is, to only create the indices in Elasticsearch itself, where as you have to manually create them in Kibana in order to access the and visualize the data.
Hope it helps!
I'm currently facing a problem when trying to upload a file after running the Jmeter using the zk-plugin. It works fine when uploading without running the Jmeter.
It displays a pop-up message in ZK:
Upload Aborted : (contentId is required)
Inside the Jmeter:
Thread Name: Thread Group 1-1
Sample Start: 2015-04-16 17:35:15 SGT
Load time: 2
Connect Time: 0
Latency: 0
Size in bytes: 2549
Headers size in bytes: 0
Body size in bytes: 2549
Sample Count: 1
Error Count: 1
Response code: Non HTTP response code: java.io.FileNotFoundException
Response message: Non HTTP response message: 13 4 2015.txt (The system cannot find the file specified)
Response headers: HTTPSampleResult fields: ContentType: DataEncoding: null
How to fix this problem?
Basically ZK could return not very meaningful messages so it can be different route causes of this issues.
Look below for possible points in deployment components configuration and check they one by one:
First of all - check that directory pointed to by java.io.tmpdir exists.
In case you use Tomcat java.io.tmpdir will be set to $CATALINA_BASE/temp by default.
Look into catalina.sh and check that directory pointed to by $CATALINA_TMPDIR exists and has corresponding permissions applied:
if [ -z "$CATALINA_TMPDIR" ] ; then
# Define the java.io.tmpdir to use for Catalina
CATALINA_TMPDIR="$CATALINA_BASE"/temp
fi
. . .
. . .
-Dcatalina.base=\"$CATALINA_BASE\" \
-Dcatalina.home=\"$CATALINA_HOME\" \
-Djava.io.tmpdir=\"$CATALINA_TMPDIR\" \
org.apache.catalina.startup.Bootstrap "$#" start
WEB-INF/zk.xml: max-upload-size value in ZK configuration descriptor (5120 Kb by default, should be enough).
WEB-INF/web.xml: max-file-size and max-request-size values in deployment descriptor:
<multipart-config>
<!-- 52MB max -->
<max-file-size>52428800</max-file-size>
<max-request-size>52428800</max-request-size>
<file-size-threshold>0</file-size-threshold>
</multipart-config>
conf/server.xml: maxPostSize value in Connector section (the maximum size in bytes of the POST which will be handled by the container FORM URL parameter parsing):
<Connector port="80" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443"
maxPostSize="67589953" />
It seems like we can only upload file that is inside our jmeter/bin. I upload using some files inside the jmeter/bin and the message is gone.
During recording you need to put the file you want to upload in jmeter/bin folder. This is due to some limitations of browsers which do not transmit the full path.
Reference : File upload fails during recording using JMeter , the first answer by pmpm