Using Apache 2.4, we had logging set to LogLevel=info and got lots of lines. In there were AH00128: File does not exist lines. Since then we reset to LogLevel=warn and have minimal entries.
However, I've been approached by people wanting to see the AH00128 lines, to keep abreast of hacking attempts.
Is there a way to configure Apache to always log certain errors, even when LogLevel is set to minimal reporting?
Thanks,
Jerome.
It's not currently possible to filter or promote messages based on ID or text and it requires some non trivial work to add it.
The best you could do is use the more liberal LogLevel (at least for the core module (LogLevel warn core:info) and then create a small filtering program to discard core:info w/o the message ID you care about.
Piped loggers can be as simple a shell or perl script that reads log entries from stdin and does whatever you want with them.
Related
So I have typical run of the mill logs from Nginx and tomcat servers which are just single line text files with typical log format. I have changed the tomcat access logs to output pipe delimited fields so I can easily process them using some unix scripts. I'd like to get rid of my unix scripts and move to using cloudwatch to process my logs in a similar manner, however I found out that cloudwatch really doesn't understand anything beyond timestamp, message, and logstream by default.
It will add fields using JSON, but JSON is verbose when it comes to log files. I'd like to just let it process a CSV file which seems like an obvious alternative to JSON. I'm willing to change my log format to meet a requirement like that, but I can't find any information about how I could do that.
Is my only option to translate my logs into JSON in order to add fields to cloudwatch? I am aware of the parse command, but I find that cumbersome to reconstitute my fields every time I want to build a query. Especially since these will mostly be access logs which will have numerous fields. I have aws cloudwatch log agent setup on my systems and I'm currently sending these logs to cloudwatch.
The closest thing there is to handling space delimited log files is to use Metric Filters. Or at least that's how the authors of CloudWatch designed it.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html
The best examples of this is here:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CountOccurrencesExample.html
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/ExtractBytesExample.html
Not sure if this is going to work for what I'm trying to do with logs, but it's a start. And it's the closest thing to a proper answer. If you want it done right, you gotta do it yo'self.
I want to quickly and efficiently analyze my apache log files.
Is there a software that would read in apache log files and display visually (no parser) with a menu statistics such as distinct IPs, requests type, ...?
Since you have (odd) requirement to not have a parser, you'll need to output your logs in a descriptive way (e.g. json). So, update your apache config to write json, then use a shipper like filebeat to send them to a store like elasticsearch where you can visualize them with a tool like kibana.
The parser (logstash, in ELK's case) will allow you to add more value to your log data, so I wouldn't dismiss it so quickly.
We have a legacy COBOL application based on OpenVMS for-which we do not have a clear idea of configuration. In this context, by "configuration" I am talking about:
Which executable files comprise the application;
Which pristine source files correspond to which executable files.
It may seem odd that 1 above is something that is not known, but over time what has happened is that executables have "come and gone" (and many still remain used). The knowledge of which executable files constitute the application as it exists today is not known since knowledge of which executables are no longer required has been lost in time. In practical terms, the team faithfully compiles all source code files and deploy the resultant executables despite the fact that there are obviously programs that are no longer used.
It goes without saying that there is no formal configuration management process and the source code is not kept in a version control system. Since the application runs on OpenVMS, the corresponding Files-11-based file system keeps older versions of files (including source files) and this has long been the excuse for not putting the application source into a version control system (despite the reasons for using a VCS extending far beyond merely having a record of previous versions).
There are a number of ways in which the configuration can be determined, of course, but I'd like to start with a first "small step", that is: determine the set of executables that comprise the application. At this point I should mention that the executable components of the application are not limited to OpenVMS images, but also DCL command files. I would like to:
Log all invocations of images that reside in a certain directory or set of directories;
Log all invocations of command files that reside in a certain directory or set of directories.
If we run this logging on our production system over an extended period of time, say two months, we can get a pretty good idea of what the application comprises. Together with user consultation, we'll be able to confirm the need for the executable files that aren't being called.
I think I have an idea of how to do 1 above, although I'm not sure of the specifics, that is, use SET/AUDIT. The second part, at this stage, I have no idea of how to do.
So, the main criterion for this effort is that as little of the existing system be affected in order to gain the above information. Due to the question mark around the configuration (and the complete lack of automated tests), changing anything is a nerve-wracking undertaking.
Using operating-system-level services like SET/AUDIT would allow one to get to know what's being run without the need to change source and/or recompile anything. So, my question is a multi-parter:
Is this the optimal way to do this on OpenVMS?
What would I need to do to restrict SET/AUDIT to only monitor images in a particular directory?
How would I log command file invocation without changing the .COM source files?
What should I expect in terms of performance degradation as a result of logging such information?
Ad 2., 3.
I would try security auditing with ACLs. From a a privileged account, something like ...
Make sure ACL auditing is enabled:
$ show audit
should show
System security audits currently enabled for:
...
ACL
...
If it doesn't, enable it with
$ set audit/audit/enable=acl
and then you may want to disable it when you are done with
$ set audit/audit/disable=acl
Set audit ACLs on all the wanted files:
$ set sec/acl=(audit=security,access=success+execute) [.app]*.com
$ set sec/acl=(audit=security,access=success+execute) [.app]*.exe
and you may want to delete the ACLs when you are done with
$ set security/acl=(audit=security,access=success+execute)/delete [.app]*.com
$ set security/acl=(audit=security,access=success+execute)/delete [.app]*.exe
You can check what ACLs are set with:
$ show security [.app]*.*
Run you application ...
Get the results from the audit file
$ analyze/audit [vms$common.sysmgr]security.audit$journal/sel=access=execute/full/since=17:00/out=app.log
Check your report for your files:
$ pipe type app.log |search sys$pipe "File name", ,"Access requested"
File name: _EMUVAX$DUA0:[USER.APP]NOW.COM;1
Access requested: READ,EXECUTE
Auditable event: Object access
File name: _EMUVAX$DUA0:[USER.APP]ECHO.EXE;1
Access requested: READ,EXECUTE
$
Sorry, I have no answer for 1. and 4.
It would help to know the OpenVMS Version (e.g. 6.2, 7.3-2, 8.4...) and the architecture (Vax, Alpha,Itanium).
Recent OpenVMS versions have great sda extensions
http://h71000.www7.hp.com/doc/84final/6549/6549pro_ext1.html
or
http://de.openvms.org/Spring2009/05-SDA_EXTENSIONS.pdf
such as LNM to check the logical names used by a process, PCS for PC sampling of a process, FLT to check the faulting behavior of applications, RMS for RMS data structures, PERF only for Itanium performance tracing, PROCIO for the reads and writes for all files opened by a process
Post a
dir sys$share:*sda.exe
so that we know which Sda extensions are available for you.
You can always check what a process with a pid of 204002B4 does with
$ ana/sys
set proc/id=204020b4
sh process /channel
exam #pc
and repeat while the process moves on.
I haven't worked on my Elasticsearch + Kibana project in a while, but when I go on my elasticsearch-head frontend now, I notice the addition of multiple indexes (image below)
If I scroll to the right, I see my Logstash related files, such as logstash-2015.06.26. So everything seems to be normal besides the fact that these elements appear (they are also numerous)
Surely I must have done something wrong for this to happen, it seems as if Logstash is parsing all the files of an unwanted directory. I cannot find anything unusual in my input or filter files.
Any ideas on how to figure this out?
Thank you.
You were scanned by a vulnerability scanner. Since you had no security on your cluster, a vulnerability scanner tried to do a POST to some URLs on your port 9200 which caused indexes to be created.
Where can I find a well-respected reference that details the proper handling of PID files on Unix?
On Unix operating systems, it is common practice to “lock” a program (often a daemon) by use of a special lock file: the PID file.
This is a file in a predictable location, often ‘/var/run/foo.pid’. The program is supposed to check when it starts up whether the PID file exists and, if the file does exist, exit with an error. So it's a kind of advisory, collaborative locking mechanism.
The file contains a single line of text, being the numeric process ID (hence the name “PID file”) of the process that currently holds the lock; this allows an easy way to automate sending a signal to the process that holds the lock.
What I can't find is a good reference on expected or “best practice” behaviour for handling PID files. There are various nuances: how to actually lock the file (don't bother? use the kernel? what about platform incompatibilities?), handling stale locks (silently delete them? when to check?), when exactly to acquire and release the lock, and so forth.
Where can I find a respected, most-authoritative reference (ideally on the level of W. Richard Stevens) for this small topic?
First off, on all modern UNIXes /var/run does not persist across reboots.
The general method of handling the PID file is to create it during initialization and delete it from any exit, either normal or signal handler.
There are two canonical ways to atomically create/check for the file. The main one these days is to open it with the O_EXCL flag: if the file already exists, the call fails. The old way (mandatory on systems without O_EXCL) is to create it with a random name and link to it. The link will fail if the target exists.
As far as I know, PID files are a convention rather than something that you can find a respected, mostly authoritative source for. The closest I could find is this section of the Filesystem Hierarchy Standard.
This Perl library might be helpful, since it looks like the author has at least given thought to some issues than can arise.
I believe that files under /var/run are often handled by the distro maintainers rather than daemons' authors, since it's the distro maintainers' responsibility to make sure that all of the init scripts play nice together. I checked Debian's and Fedora's developer documentation and couldn't find any detailed guidelines, but you might be able to get more info on their developers' mailing lists.
See Kerrisk's The Linux Programming Interface, section 55.6 "Running Just One Instance of a Program" which is based on the pidfile implementation in Stevens' Unix Network Programming, v2.
Note also that the location of the pidfile is usually something handled by the distro (via an init script), so a well written daemon will take a command line argument to specify the pidfile and not allow this to be accidentally overridden by a configuration file. It should also gracefully handle a stale pid file by itself (O_EXCL should not be used). fcntl() file locking should be used--you may assume that a daemon's pidfile is located on a local (non-NFS) filesystem.
Depending on the distribution, its actually the init script that handles the pidfile. It checks for existence at starting, removes when stopping, etc. I don't like doing it that way. I write my own init scripts and don't typically use the stanard init functions.
A well written program (daemon) will have some kind of configuration file saying where this pidfile (if any) should be written. It will also take care to establish signal handlers so that the PID file is cleaned up on normal, or abnormal exit, whenever a signal can be handled. The PID file then gives the init script the correct PID so it can be stopped.
Therefore, if the pidfile already exists when starting, its a very good indicator to the program that it previously crashed and should do some kind of recovery effort (if applicable). You kind of shoot that logic in the foot if you have the init script itself checking for the existence of the PID, or unlinking it.
As far as the name space, it should follow the program name. If you are starting 'foo-daemon', it would be foo-daemon.pid
You should also explore /var/lock/subsys, however that's used mostly on Red Hat flavors.
The systemd package on Red Hat 7 provides a man page daemon(7) with the header line "Writing and packaging system daemons."
This man page discusses both "old style" (SysV) and "new style" (systemd) daemonization. In new style, systemd itself handles the PID files for you (if so configured to do so). However, in old style, the man page has this to say:
In the daemon process, write the daemon PID (as returned by getpid())
to a PID file, for example /run/foobar.pid (for a hypothetical daemon
"foobar") to ensure that the daemon cannot be started more than once.
This must be implemented in race-free fashion so that the PID file is
only updated when it is verified at the same time that the PID
previously stored in the PID file no longer exists or belongs to a
foreign process.
You can also read this man page online.