nxlog process logs files based on filename - nxlog

I have lots of log files in C:\Logs.
I have IIS logs in each individual site folder (W3SVCXX, where XX is site id from IIS) but inside this same C:\Logs\ folder I have other log files from those same websites but logged via Log4Net as part of the application running at the website.
My current NXLog Config:
<Input i_IISSITES>
Module im_file
File "C:\Logs\u_ex*"
SavePos TRUE
Recursive TRUE
InputType LineBased
Exec $Hostname = 'stg-01-iom';
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$SourceName = "IIS"; \
$EventTime = parsedate($date + " " + $time); \
$Message = to_json(); \
}
</Input>
<Output o_PLAINTEXT>
Module om_udp
Host 10.50.108.32
Port 5555
</Output>
<Route r_IIS>
Path i_IISSITES => o_PLAINTEXT
</Route>
This works perfectly currently for IIS logs which match the filename U_ex*.
Folder structure:
c:\Logs\
- L009
- Dir1
- Dir2
- FileName.Log
- L008
- Dir3
- RandomFileName.Log
- L008
- W3SVC1
- u_ex1232.log
- W3SVC2
- W3SVC3
etc etc..
Now I'm able to be specific for IIS logs because the filename is u_ex* but with my other log files they could be called anything.
So for my other input I need to be able to target all .log but not u_ex.log.
Any ideas?
Thanks,
Michael

If you need to process all logs then instead of defining two input instances you could just use one and parse differently depending on the file name:
<Exec>
if file_name() =~ /u_ex\d+\.log$/
{
# parse as IIS
}
else
{
# parse as other
}
</Exec>

Related

Get the original filename of symlinks in nginx

From another script i got some generated symlinks.
2QGPCKVNG1R -> /anotherdir/movie1.mp4
HJS7J9ND2L5 -> /anotherdir/movie2.mp4
LKA6A9LA7SK -> /anotherdir/movie3.mp4
Displaying these files in NGINX works fine, but I'd like to rename the files at download via content disposition.
Question is how do i get the original filename in nginx variable?
I'm not sure it is possible at all. Is that another script yours or under your control? You can generate an additional nginx config file with a map block with the same script where you can describe a ruleset for mapping an URI value to the Content-Disposition header value (or you can write an additional script to do it with readlink -f <symlink> command:
map $uri $content_disposition {
~/2QGPCKVNG1R$ movie1.mp4;
~/HJS7J9ND2L5$ movie2.mp4;
~/LKA6A9LA7SK$ movie3.mp4;
}
And then include that file to the main nginx config:
include /path/to/content-disposition-map.conf;
server {
...
add_header Content-Disposition $content_disposition;
Another way I see is to use lua-nginx-module and a LUA script like
map $symlink_target $content_disposition {
~/([^/]*)$ $1;
}
server {
...
set_by_lua_block $symlink_target {
local result = io.popen("/bin/readlink -n -f " .. ngx.var.request_filename)
return result:read()
}
add_header Content-Disposition $content_disposition;

Azure automation runbook Completed before running the code

I have a situation where in the Azure automation runbook results in 'Completed' state and does not run the 'actual' code. I have pasted the code below. It creates a Event Hub inside a Namespace. The code works perfectly executing in local machine but it does not execute in Runbook.
I have written a 'write-output "Declaring local variables for use in script"' --> to check if the printing is working. However, the code is not going beyond that. I am sure, I am missing some thing. Kindly help me.
Param(
[Parameter(Mandatory=$true)]
[string] $NameSpaceNameName,
[Parameter(Mandatory=$true)]
[string[]] $EventhubNames,
[Parameter(Mandatory=$true)]
[string] $ProjectId,
[Parameter(Mandatory=$true)]
[int] $PartitionCount,
[Parameter(Mandatory=$true)]
[string]$Requested_for,
[Parameter(Mandatory=$true)]
[string]$Comments
)
## Login to Azure using RunAsAccount
$servicePrincipalConnection = Get-AutomationConnection -Name 'AzureRunAsConnection'
Write-Output ("Logging in to Az Account...")
Login-AzAccount `
-ServicePrincipal `
-TenantId $servicePrincipalConnection.TenantId `
-ApplicationId $servicePrincipalConnection.ApplicationId `
-CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint
write-output "Declaring local variables for use in script"
## Declaring local variables for use in script
$Creation_date = [System.Collections.ArrayList]#()
$ResourceGroups = Get-AzResourceGroup
$provided_name_space_exists = $false
## Change context to Platform subscription
select-azsubscription -subscription "GC302_Sub-platform_Dev"
## Create Event Hub
foreach($Resourcegroup in $ResourceGroups){
Write-Host("Processing the Resource Group: {0} " -f $Resourcegroup.ResourceGroupName)
$EventhubNameSpaces = Get-AzEventHubNamespace -ResourceGroupName $ResourceGroup.ResourceGroupName
# Iterate over each Namespace. Fetch the Resource Group that contains the provided Namespace
foreach($EHNameSpace in $EventhubNameSpaces){
if($EHNameSpace.Name -eq $NameSpaceName){
$provided_name_space_exists = $true
Write-Host ("Found the provided Namespace in resource group: {0}" -f $Resourcegroup.ResourceGroupName)
Write-Output ("Found the provided Namespace in resource group: {0}" -f $Resourcegroup.ResourceGroupName)
$nameSpace_resource_group = $ResourceGroup.ResourceGroupName
# Fetch the existing Event Hubs in the Namespace
$existing_event_hubs_list = Get-AzEventHub -Namespace $EHNameSpace.Name -ResourceGroupName $nameSpace_resource_group
## Check provided EH for uniqueness
if($existing_event_hubs_list.Count -le 1000){
for($i = 0;$i -lt $EventhubNames.Count;$i++){
if($existing_event_hubs_list.name -notcontains $EventhubNames[$i]){
$EventHub = New-AzEventHub -ResourceGroupName $nameSpace_resource_group -Namespace $EHNameSpace.Name -Name $EventhubNames[$i] -PartitionCount $PartitionCount
$date = $EventHub.CreatedAt
$Creation_date+= $date.GetDateTimeFormats()[46]
}else{
Write-Host ("Event hub: '{0}' already exists in the NameSpace: {1}. skipping this Event hub creation" -f $EventhubNames[$i], $EHNameSpace.Name)
}
}
}else{
Write-Host ("The Namespace - {0} has Event Hubs count greater or equal to 1000." -f $EHNameSpace.Name)
Write-Host ("Please refer the Link for Eevent Hubs quota/limit: 'https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas#event-hubs-dedicated---quotas-and-limits'")
exit
}
}
}
}
# Print a message that Namespace does not exist
if($provided_name_space_exists -eq $false){
Write-Host ("Provided NameSpace: {0} does not exist." -f $NameSpaceName)
exit
}
Screenshot:
You have $NameSpaceNameName in the parameters section of the runbook but later in the runbook at 50th line you have $NameSpaceName which is not the same as mentioned in parameters section. Correct it and then it should work as expected. One suggestion is to always have an else block wherever you have if block to overcome such issues in future.

IntelliJ Find Action on Mac opens terminal app by error

When I press Cmd + Shift + A in IntelliJ it should open the "Find Action..." dialog. Instead it opens Terminal.app with the apropos command.
How can I resolve this problem?
Seems in MacOs 10.14.4 a new default shortcut was enabled - Services -> Search man Page Index in Terminal. It uses the same shortcut as Find Action - Cmd+Shift+A. As a result using Find Action in the IDE could sometimes open terminal window with apropos <smth> command output.
Explicit Location: System Preferences > Keyboard > Shortcuts > Services (on left hand side) > (uncheck) Search man Page Index in Terminal
Here’s a script (hosted in gist) that will do this via the command line, which might be useful for an organisation that wants to roll out this change for a number of users.
TEMP_SETTINGS_FILE=$(mktemp -t 'man-shortcuts-off.json')
cat > $TEMP_SETTINGS_FILE <<EOF
{
"NSServicesStatus": {
"com.apple.Terminal - Open man Page in Terminal - openManPage": {
"presentation_modes": {
"ContextMenu": false,
"ServicesMenu": false
},
"enabled_context_menu": false,
"enabled_services_menu": false
},
"com.apple.Terminal - Search man Page Index in Terminal - searchManPages": {
"presentation_modes": {
"ContextMenu": false,
"ServicesMenu": false
},
"enabled_context_menu": false,
"enabled_services_menu": false
}
}
}
EOF
# Settings are stored in the pbs domain. Other settings in this domain will not be overwritten. I’ve included the settings to change in JSON for brevity. They are converted to XML (which `defaults import` expects) before being imported.
plutil -convert xml1 -o - ${TEMP_SETTINGS_FILE} | defaults import pbs -
rm ${TEMP_SETTINGS_FILE}

icinga2 notifications to cachet

I would like to share with you a way to send notifications from icinga2 to cachet via the API.
Icinga2 version : 2.4.10-1
Cachet version : 2.3.9
First of all, you have to know which component ID you want to use (in my case, because you can update component by name)
To get the component ID, you can use the curl command :
curl --insecure --request GET --url https://URL/api/v1/components -H "X-Cachet-Token: TOKEN"
URL : The URL of your cachet installation
TOKEN : The Token of the member in Cachet
Create command in /etc/icinga2/conf.d/commands.conf
object NotificationCommand "cachet-incident-notification-v2" {
import "plugin-notification-command"
command = [ PluginDir + "/cachet-notification-v2.sh" ]
env = {
"SERVICESTATE" = "$service.state$"
}
}
Create notification template in /etc/icinga2/conf.d/templates.conf
template Notification "cachet-incident-notification-v2" {
command = "cachet-incident-notification-v2"
states = [ OK, Warning, Critical, Unknown ]
types = [ Problem, Acknowledgement, Recovery, Custom,
FlappingStart, FlappingEnd,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]
/*
period = "24x7"
*/
interval = 0
}
Create notification in /etc/icinga2/conf.d/notifications.conf
apply Notification "cachet-incident-notification-v2" to Service {
import "cachet-incident-notification-v2"
user_groups = host.vars.notification.pager.groups
assign where service.vars.cachetv2 == "1" && host.vars.cachetv2 == "1"
interval = 0 # Disable Re-notification
}
Add variable in your check service in /etc/icinga2/conf.d/service/your/service.conf
[...]
vars.cachetv2 = "1"
[...]
Add variable in your host config file in /etc/icinga2/conf.d/hosts/your/host
[...]
vars.cachetv2 = "1"
[...]
Create the script in /usr/lib/nagios/plugins/cachet-notification-v2.sh
#!/bin/bash
# Some Constants
NOW="$(date +'%d/%m/%Y')"
CACHETAPI_URL="https://URL/api/v1/components/<ID DU COMPOSANT>"
CACHETAPI_TOKEN="TOKEN><"
# Map Notification states for icinga2
# OK - 1 operational
# Warning - 3 Partial outage
# Critical - 4 Major outage
# Unknown - 2 Performance issues
case "$SERVICESTATE" in
'OK')
COMPONENT_STATUS=1
;;
'WARNING')
COMPONENT_STATUS=3
;;
'CRITICAL')
COMPONENT_STATUS=4
;;
'UNKNOWN')
COMPONENT_STATUS=2
;;
esac
curl -X PUT -H "Content-Type: application/json;" -H "X-Cachet-Token: ${CACHETAPI_TOKEN}" -d '{"status": "'"${COMPONENT_STATUS}"'"}' ${CACHETAPI_URL} -k
PS : Give the execution permission to the script
Check the syntax and reload
/etc/init.d/icinga2 checkconfig && /etc/init.d/icinga2 reload
The result :
When your check results in "CRITICAL", the status in Cachet will be MAJOR ISSUE
When your check results in "WARNING", the status in Cachet will be PARTIAL ISSUE
When your check results in "OK", the status in Cachet will be OPERATIONAL
When your check results in "UNKNOWN", the status in Cachet will be PERFORMANCE DELAY
I hope it will help.
Nicolas B.

Mod_security av_scanning

I installed mod_security with OWASP rules on a Debian Jessie server, and experience the problem that it does not run the "runav.pl" script when I try to upload a file.
I modified the script to create /tmp/filewrite.txt with content of "Test text" when it is run. If I run it by hand it creates the file, but when I upload a file it does not create the above mentioned test file.
Here is the modified runav.pl script:
#!/usr/bin/perl
#
# runav.pl
# Copyright (c) 2004-2011 Trustwave
#
# This script is an interface between ModSecurity and its
# ability to intercept files being uploaded through the
# web server, and ClamAV
my $filename = '/tmp/filewrite.txt';
open(my $fh, '>', $filename);
print $fh "Test text\n";
close $fh;
$CLAMSCAN = "clamdscan";
if ($#ARGV != 0) {
print "Usage: modsec-clamscan.pl <filename>\n";
exit;
}
my ($FILE) = shift #ARGV;
$cmd = "$CLAMSCAN --stdout --disable-summary $FILE";
$input = `$cmd`;
$input =~ m/^(.+)/;
$error_message = $1;
$output = "0 Unable to parse clamscan output [$1]";
if ($error_message =~ m/: Empty file\.?$/) {
$output = "1 empty file";
}
elsif ($error_message =~ m/: (.+) ERROR$/) {
$output = "0 clamscan: $1";
}
elsif ($error_message =~ m/: (.+) FOUND$/) {
$output = "0 clamscan: $1";
}
elsif ($error_message =~ m/: OK$/) {
$output = "1 clamscan: OK";
}
print "$output\n";
And here is the related lines from modsecurity.conf:
SecRuleEngine DetectionOnly
SecServerSignature FreeOSHTTP
SecRequestBodyAccess On
SecRequestBodyLimit 20971520
SecRequestBodyNoFilesLimit 131072
SecRequestBodyInMemoryLimit 20971520
SecRequestBodyLimitAction Reject
SecPcreMatchLimit 1000
SecPcreMatchLimitRecursion 1000
SecResponseBodyAccess On
SecResponseBodyMimeType text/plain text/html text/xml
SecResponseBodyLimit 524288
SecResponseBodyLimitAction ProcessPartial
SecTmpDir /tmp/
SecDataDir /tmp/
SecUploadDir /opt/modsecuritytmp/
SecUploadFileMode 0640
SecDebugLog /var/log/apache2/debug.log
SecDebugLogLevel 3
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogParts ABIJDEFHZ
SecAuditLogType Serial
SecAuditLog /var/log/apache2/modsec_audit.log
SecArgumentSeparator &
SecCookieFormat 0
SecUnicodeMapFile unicode.mapping 20127
SecStatusEngine On
Activated rules are under /etc/modsecurity/activated_rules, and all the other rules work well, but "modsecurity_crs_46_av_scanning.conf".
Does anyone have an idea why it does not do anything with uploaded file?