I need to parse Apache log file to look for specific suspicious patterns (like SQL injections).
For example I'm looking for id='%20or%201=1;
I am using grep to check the log file for this pattern (and others) and because these logs are huge it takes a long amount of time
Here my command:
grep 'id=' Apache.log | egrep "' or|'%20"
Is there a better or a faster method or command I need use to make the search faster?
For starters, you don't need to pipe your grep output to egrep. egrep provides a superset of grep's regular expression parsing, so you can just do this:
egrep "id='( or|%20)'" apache.log
Calling egrep is identical to calling grep -E.
That may get you a little performance increase. If you can look for fixed strings rather than regular expressions, that might also help. You can tell grep to look for a fixed string with the -F option:
grep -F "id='%20or" apache.log
But using fixed strings you lose a lot of flexibility.
I assume most of your time is spent while getting the data from disk (CPU usage is not maxed out). Then you can't optimize a query. You could try to only log the interesting lines in a seperate file though....
Are you looking for grep -E "id=(' or|'%20)" apache.log ?
Related
In a Makefile, reading user input terminated by <ENTER> can be implemented by using (the shell function) read. On the shell (bash), reading a single character can be done with read -n 1. However, I was surprised that read -n 1 didn't seem to work with GNU Make 4.2.1. Do I miss some escaping here?
Bash:
$> echo x | read -n 1 mychar; echo 'you typed '$mychar
you typed x
Makefile:
all:
read -n 1 mychar; echo 'you typed '$$mychar
Make:
$> echo x | make
read -n 1 mychar; echo 'you typed '$mychar
/bin/sh: 1: read: Illegal option -n
Does make provide its own version of read ?
System: GNU Make 4.2.1, Ubuntu 20.04.3 LTS
PS: I am aware that user interaction is considered bad style, and it clearly should not be used in configuration workflows. However, it comes in quite handy for targets associated with pruning, resetting or deleting data (e.g., make clear). And asking for y/N confirmation gives you a chance to tell your users what it will take them to rebuild what they are going to remove.
PPS: I know that the regular read gives you almost the same functionality, except that you need to hit <ENTER>. This question about getting a better understanding for the relationship between make and its shell enviroment.
NB: This problem is different from Makefile - Why is the read command not reading the user input?. They just didn't properly escape the variable.
I am testing some PHP apps for injectable commands. I have to convert my commands to a URI/CGI encoded format. I am wondering if there is a better way to do it.
When I want to include a ping (to test if the app is, in fact, executing from an injection) I am converting it as follows.
hURL -X --esc ";ping localhost -c 1" | sed -e ‘s/\\x/\%/g’
Here is the output.
%3b%20%70%69%6e%67%20%6c%6f%63%61%6c%68%6f%73%74%20%2d%63%20%31
Works perfect. The code is injected and logs are showing it being handled as expected.
QUESTION: Is there a better way to convert to the above. I think I am over complicating things.
You could possibly use an out-of-the-box library for doing the escaping, may be a little easier on the eye ...
$ echo ';ping localhost -c 1' | perl -ne 'use URI::Escape; print(uri_escape($_) . "\n");'
%3Bping%20localhost%20-c%201%0A
Obviously this output does not escape legitimate url chars so not sure this entirely answers your question ...
I completed a nmap scan on a large-ish network and now I am trying to organize the data.
The report I have is the result of :
nmap -A -p 0-65535 -iL [filename] -oX [filename]
So what I am trying to do now is to extract the findings for each IP address that I scanned. I found another post here where the solution was to use awk :
awk 'BEGIN {RS="< host ";} /^starttime/ {print RS $0;}' [filename]
This didnt work for me because instead of stopping after the first block it ran right through the report. I realize of course that this is because '< host ' and 'starttime' are found in the output for all the IP addresses in the range.
Is there anyway for me to run through the nmap report and to extract the scan report for each IP address and save it in a separate file? A For loop will be required to do this of course... once the extraction and writing to file of one block is figured out then that can be expanded using the for loop (i think)...
Or does anyone, from experience or sheer inspiration, have a more refined solution/suggestion?
Any help in the matter will be greatly appreciated.
Don't use awk to parse XML data. Nmap's XML output format is well-documented and there are parsers for it in Python (Ndiff also installs as a Python 2 library and has a parser built-in), Ruby, Perl, or you can use a number of command-line XML parsers.
I'm trying to have a variable $totalLines that stores the total lines in a file (given as input $1).
I'm trying to do something like this:
totalLines= grep -c *.* $1
But Unix doesn't like that.
I've tried enclosing it in paranthesis, square brackets, and (), but that doesn't work either. This has got to be super simple but I'm searching for the answer around the web and not finding a page or forum that clearly states it.
Sorry to trouble you guys with such an easy one.
There are two ways to achieve it:
totalLines=$(grep -c *.* $1)
or
totalLines=`grep -c *.* $1`
Like:
totalLines=$(grep -c *.* $1)
I want to make certain programs only if the SSE instruction set is supported on the machine where make is run (native target). Assuming it runs linux,
grep sse /proc/cpu_info | wc -l
returns 0 if SSE instructions are not supported and >0 otherwise. But how can I use that in my makefile to facilitate conditional makes?
I currently use GNU Make 3.81, running on linux.
In traditional GNU usage, you wouldn't let make do this job, but rather configure, which writes its output into a Makefile. After all, you don't want to re-check your environment every time you run make - in the best case, you waste time rechecking an unchanged environment, and in the worst, your build gets inconsistent.
That said, a make-only solution would look like:
HAVE_SSE=$(filter-out 0,$(shell grep sse /proc/cpu_info | wc -l))
CFLAGS+=$(if $(HAVE_SSE),-msse)
ifneq ($(HAVE_SSE),)
sse-target
endif
The shell function expands to the shell command's output. The filter-out makes the non-SSE output an empty string instead of 0 because that's easier to conditionalize. Then, you can use the if function as indicated to have conditionals on the value.
So I figured out a (working!) solution by myself, similar to the suggestion by thiton:
SSE := $(shell grep sse /proc/cpuinfo)
INVALID := $(shell grep non-sense /proc/cpuinfo)
ifneq ($(INVALID),$(SSE))
sse_target
endif