Icinga2 event plugin command starting a rundeck job via api - api

Question

i made myself a test environment in icinga2 with a tomcat server. I would like to combine the two softwares rundeck and icinga. My idea is to start a rundeck job, when icinga detects a problem. In my case I have a tomcat server, where i fill up the swap memory, which should start the rundeck job to clear the swap.
I am using the Icinga2 Director for managing. I created an event plugin command, which should execute the rundeck api command as a script, called "rundeckapi". It looks like this:
#/usr/lib64/nagios/plugins/rundeckapi
#!/bin/bash
curl --location --request POST 'rundeck-server:4440/api/38/job/9f04657a-eaab-4e79-a5f3-00d3053f6cb0/run' \
--header 'X-Rundeck-Auth-Token: GuaoD6PtH5BhobhE3bAPo4mGyfByjNya' \
--header 'Content-Type: application/json' \
--header 'Cookie: JSESSIONID=node01tz8yvp4gjkly8kpj18h8u5x42.node0' \
--data-raw '{
"options": {
"IP":"192.168.4.13"
}
}'
(I also tried to just paste the command in the command field in the director, but this didn't work either.)
I placed it in the /usr/lib64/nagios/plugins/ directory and set the configuration in icinga for the command as following:
#zones.d/director-global/command.conf
object EventCommand "SWAP clear" {
import "plugin-event-command"
command = [ PluginDir + "/rundeckapi" ]
}
The service template looks like this:
#zones.d/master/service_templates.conf
template Service "SWAP" {
check_command = "swap"
max_check_attempts= "5"
check_interval = 1m
retry_interval = 15s
check_timeout = 10s
enable_notifications = true
enable_active_checks = true
enable_passive_checks = true
enable_event_handler = true
enable_flapping = true
enable_perfdata = true
event_command = "SWAP clear"
command_endpoint = host_name
}
Then I added the service to the host.
I enabled the debug mode and started to fill the SWAP and watched at the debug.log, with tail -f /var/log/icinga2/debug.log | grep 'event handler' and found this:
notice/Checkable: Executing event handler 'SWAP clear' for checkable 'centos_tomcat_3!SWAP'
The centos_tomcat_3 is the host for testing. IT seems like the event handler is executing the the script, but when I look at the rundeck server, i can't find a running job. When i start the rundeckapi script manually it works and i can see the job on rundeck.
I already read the documentation from icinga, but i didn't help.
I would be very thankful if someone could help me.
Thanks in advance.

Answer 1

Define the plugin as an event handler and assign it to the host.
I tested using this docker environment modified with Rundeck official image + a NGINX host:
version: '2'
services:
icinga2:
#image: jordan/icinga2
build:
context: ./
dockerfile: Dockerfile
restart: on-failure:5
# Set your hostname to the FQDN under which your
# sattelites will reach this container
hostname: icinga2
env_file:
- secrets_sql.env
environment:
- ICINGA2_FEATURE_GRAPHITE=1
# Important:
# keep the hostname graphite the same as
# the name of the graphite docker-container
- ICINGA2_FEATURE_GRAPHITE_HOST=graphite
- ICINGA2_FEATURE_GRAPHITE_PORT=2003
- ICINGA2_FEATURE_GRAPHITE_URL=http://graphite
# - ICINGA2_FEATURE_GRAPHITE_SEND_THRESHOLDS=true
# - ICINGA2_FEATURE_GRAPHITE_SEND_METADATA=false
- ICINGAWEB2_ADMIN_USER=admin
- ICINGAWEB2_ADMIN_PASS=admin
#- ICINGA2_USER_FULLNAME=Icinga2 Docker Monitoring Instance
- DEFAULT_MYSQL_HOST=mysql
volumes:
- ./data/icinga/cache:/var/cache/icinga2
- ./data/icinga/certs:/etc/apache2/ssl
- ./data/icinga/etc/icinga2:/etc/icinga2
- ./data/icinga/etc/icingaweb2:/etc/icingaweb2
- ./data/icinga/lib/icinga:/var/lib/icinga2
- ./data/icinga/lib/php/sessions:/var/lib/php/sessions
- ./data/icinga/log/apache2:/var/log/apache2
- ./data/icinga/log/icinga2:/var/log/icinga2
- ./data/icinga/log/icingaweb2:/var/log/icingaweb2
- ./data/icinga/log/mysql:/var/log/mysql
- ./data/icinga/spool:/var/spool/icinga2
# Sending e-mail
# See: https://github.com/jjethwa/icinga2#sending-notification-mails
# If you want to enable outbound e-mail, edit the file mstmp/msmtprc
# and configure to your corresponding mail setup. The default is a
# Gmail example but msmtp can be used for any MTA configuration.
# Change the aliases in msmtp/aliases to your recipients.
# Then uncomment the rows below
# - ./msmtp/msmtprc:/etc/msmtprc:ro
# - ./msmtp/aliases:/etc/aliases:ro
ports:
- "80:80"
- "443:443"
- "5665:5665"
graphite:
image: graphiteapp/graphite-statsd:latest
container_name: graphite
restart: on-failure:5
hostname: graphite
volumes:
- ./data/graphite/conf:/opt/graphite/conf
- ./data/graphite/storage:/opt/graphite/storage
- ./data/graphite/log/graphite:/var/log/graphite
- ./data/graphite/log/carbon:/var/log/carbon
mysql:
image: mariadb
container_name: mysql
env_file:
- secrets_sql.env
volumes:
- ./data/mysql/data:/var/lib/mysql
# If you have previously used the container's internal DB use:
#- ./data/icinga/lib/mysql:/var/lib/mysql
rundeck:
image: rundeck/rundeck:3.3.12
hostname: rundeck
ports:
- '4440:4440'
nginx:
image: nginx:alpine
hostname: nginx
ports:
- '81:80'
Rundeck side:
To access Rundeck open a new tab in your browser using the http://localhost:4440 web address. You can access with user: admin and password: admin.
Create a new project and create a new job, I created the following one, you can import it to your instance:
- defaultTab: nodes
description: ''
executionEnabled: true
id: c3e0860c-8f69-42f9-94b9-197d0706a915
loglevel: INFO
name: RestoreNGINX
nodeFilterEditable: false
options:
- name: opt1
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- exec: echo "hello ${option.opt1}"
keepgoing: false
strategy: node-first
uuid: c3e0860c-8f69-42f9-94b9-197d0706a915
Now go to the User Icon (up to right) > Profile, now click on the + icon ("User API Tokens" section) and save the API key string, useful to create the API call script from the Icinga2 container.
Go to the Activity page (left menu) and click on the "Auto Refresh" checkbox.
Incinga2 side:
You can enter Icinga 2 by opening a new tab in your browser using the http://localhost URL, I defined username: admin and password: admin in the docker-compose file.
Add the following script as a command at /usr/lib/nagios/plugins path with the following content (it's a curl api call like your scenario, the API key is the same generated in the third step from "Rundeck side" section of this step-by-step):
#!/bin/bash
curl --location --request POST 'rundeck:4440/api/38/job/c3e0860c-8f69-42f9-94b9-197d0706a915/run' --header 'X-Rundeck-Auth-Token: Zf41wIybwzYhbKD6PrXn01ZMsV2aT8BR' --header 'Content-Type: application/json' --data-raw '{ "options": { "opt1": "world" } }'
Also make the script executable: chmod +x /usr/lib/nagios/plugin/restorenginx
In the Icinga2 browser tab, go to the Icinga Director (Left Menu) > Commands. On the "Command Type" list select "Event Plugin Command", on the "Command Name" textbox type "restorenginx" and on the "Command" textbox put the full path of the script (/usr/lib/nagios/plugins/restorenginx). Then click on the "Store" button (bottom) and now click on Deploy (up).
Check how looks.
This is the config preview (at zones.d/director-global/commands.conf):
object EventCommand "restorenginx" {
import "plugin-event-command"
command = [ "/usr/lib/nagios/plugins/restorenginx" ]
}
Now, create the host template (In my example I'm using an Nginx container to monitoring), go to Icinga Director (Left Menu) > Hosts, and select "Host Templates". Then click on the + Add link (up). On the Name type the host template name, I used "nginxSERVICE", on the "check command" textbox put the command to check the host alive (I used "ping"). Now in the Event command textbox select the Command created in the previous step.
Check how looks.
Now it's time to create the host (based on the previous steps template). Go to Icinga Direcrector (Left Menu) > Hosts and select "Host". Then click on the + Add link (up). On the hostname type the server hostname (nginx, defined on the docker-compose file), In "Imports" select the template is created in the previous step ("nginxSERVICE"), type anything on the "Display name" textbox, and in the "Host address" add the Nginx container IP. Click on the "Store" button" and then on the "Deploy" link at the top.
Check how looks.
To enable the Event Hander on the host, go to the "Overview" (Left menu) > Hosts, select "NGINX", scroll down on the right section and enable "Event Handler" on the "Feature Commands" section.
Check how looks.
Nginx side (it's time to test the script):
Stop the container and go to the Rundeck Activity page browser tab, you'll see the job launched by the Icinga2 monitoring tool.