In modsecurity default-script:
base_rules/modsecurity_crs_20_protocol_violations.conf
there is a rule, 960011:
SecRule REQUEST_METHOD "^(?:GET|HEAD)$" \
"msg:'GET or HEAD Request with Body Content.',\
severity:'2',\
id:'960011',\
ver:'OWASP_CRS/2.2.9',\
rev:'1',\
maturity:'9',\
accuracy:'9',\
phase:1,\
block,\
logdata:'%{matched_var}',\
t:none,\
tag:'OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ',\
tag:'CAPEC-272',\
chain"
SecRule REQUEST_HEADERS:Content-Length "!^0?$"\
"t:none,\
setvar:'tx.msg=%{rule.msg}',\
setvar:tx.anomaly_score=+%{tx.critical_anomaly_score},\
setvar:'tx.%{rule.id}-OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ-%{matched_var_name}=%{matched_var}'"
I only want to disable logging for this rule (it gives too many false positives),
and therefore add my own script
base_rules/z99_logging_suppress.conf
to remove the default-rule and create a new identical rule -- only without logging:
SecRuleRemoveById 960011
SecRule REQUEST_METHOD "^(?:GET|HEAD)$" \
"msg:'GET or HEAD Request with Body Content.',\
severity:'2',\
id:'9960011',\
ver:'OWASP_CRS/2.2.9',\
rev:'1',\
maturity:'9',\
accuracy:'9',\
phase:1,\
block,nolog,\
logdata:'%{matched_var}',\
t:none,\
tag:'OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ',\
tag:'CAPEC-272',\
chain"
SecRule REQUEST_HEADERS:Content-Length "!^0?$"\
"t:none,\
setvar:'tx.msg=%{rule.msg}',\
setvar:tx.anomaly_score=+%{tx.critical_anomaly_score},\
setvar:'tx.%{rule.id}-OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ-%{matched_var_name}=%{matched_var}'"
The only differences to the original rule are the new id 9960011, and the nolog additions:
...
id:'9960011',\
...
block,nolog,\
...
But when I restart httpd with this additional rule, I get error:
AH00526: Syntax error on line 18 of /path/base_rules/z99_logging_suppress.conf:
ModSecurity: Execution phases can only be specified by chain starter rules.
The same strategy --- SecRuleRemoveById + then re-create it with new id --- works for all other default-rules I tried, but not for this one.
Anyone can tell me why that is?
It basically says that the phase command can only be in the first rule in a chain and not in a subsequent rule which forms part of the chain.
There is nothing wrong with the rule as you have written it, phase is only specified in the first SecRule. In fact I've tried it on my instance and it works. So either one of two things has gone wrong:
You have copied and pasted it incorrectly into this question.
The rule above where you have defined this, has chain in it and so has left an open chain, that your rule 9960011 is then effectively trying to continue on from.
Or something else weird is happening! But I'm going with 1 or 2 for now :-)
Related
I'm trying to configure modsecurity for Apache to limit the number of times a given resource can be accessed.
I wrote this code, and it works (I'm getting a 429 rejection as I wanted), but I can't reinitiate ip.counter at a certain point of time (last line).
SecAction initcol:ip=%{REMOTE_ADDRESS},pass,nolog,id:132
SecAction "phase:2,setvar:ip.counter=+1,pass,nolog,id:332"
SecRule IP:COUNTER "#ge 1" "phase:3,id:'9000080007',pause:10,deny,status:429,setenv:RATELIMITED,skip:1,nolog,id:232"
SecRule TIME "^10:37:00$" "phase:2,id:'9000080008',setvar:!ip.counter"
However, if I switch the last line to use TIME_HOUR instead, the SecRule does apply correctly:
SecRule TIME_HOUR "#eq 10" "phase:2,id:'9000080008',setvar:!ip.counter"
Any help please for using TIME variable in SecRule to match the exact time?
Congratulations on getting a very advanced recipe to work properly. This is really cool.
Now your rule does not work, because the online reference is wrong about the format of the TIME variable (The Handbook is correct though).
Here is how to debug this on ModSec debug log level 9:
SecRule TIME "#unconditionalMatch" "id:1000,phase:2,pass,log,msg:'Key : Value : |%{MATCHED_VAR_NAME}| : |%{MATCHED_VAR}|'"
Leads to:
...4c20][/][5] Rule 562b28db5420: SecRule "TIME" "#unconditionalMatch " "phase:2,auditlog,id:1007,pass,log,msg:'Key : Value : |%{MATCHED_VAR_NAME}| : |%{MATCHED_VAR}|'"
...4c20][/][4] Transformation completed in 0 usec.
...4c20][/][4] Executing operator "unconditionalMatch" with param "" against TIME.
...4c20][/][9] Target value: "20220829070111"
...4c20][/][4] Operator completed in 0 usec.
...4c20][/][9] Resolved macro %{MATCHED_VAR_NAME} to: TIME
...4c20][/][9] Resolved macro %{MATCHED_VAR} to: 20220829070111
...4c20][/][2] Warning. Unconditional match in SecAction. [file "/apache/conf/httpd.conf_pod_2022-08-29_06:58"] [line "209"] [id "1007"] [msg "Key : Value : |TIME| : |20220829070111|"]
...4c20][/][4] Rule returned 1.
...4c20][/][9] Match -> mode NEXT_RULE.
So as the title says I can't bring my workflow to execute anything, except the all rule...
When Executing the all rule it correctly finds all the input files, so the configfile is okay, every path is correct.
when trying to run without additional tags I get
Building DAG of jobs...
Checking status of 0 jobs.
Nothing to be done
Things I tried:
-f rcorrector -> only all rule
filenameR1.fcor_val1.fq -> MissingRuleException (No Typos)
--forceall -> only all rule
Some more fiddling I can't formulate clearly
please Help
from os import path
configfile:"config.yaml"
RNA_DIR = config["RAW_RNA_DIR"]
RESULT_DIR = config["OUTPUT_DIR"]
FILES = glob_wildcards(path.join(RNA_DIR, '{sample}R1.fastq.gz')).sample
############################################################################
rule all:
input:
r1=expand(path.join(RNA_DIR, '{sample}R1.fastq.gz'), sample=FILES),
r2=expand(path.join(RNA_DIR, '{sample}R2.fastq.gz'), sample=FILES)
#############################################################################
rule rcorrector:
input:
r1=path.join(RNA_DIR, '{sample}R1.fastq.gz'),
r2=path.join(RNA_DIR, '{sample}R2.fastq.gz')
output:
o1=path.join(RESULT_DIR, 'trimmed_reads/corrected/{sample}R1.cor.fq'),
o2=path.join(RESULT_DIR, 'trimmed_reads/corrected/{sample}R2.cor.fq')
#group: "cleaning"
threads: 8
params: "-t {threads}"
envmodules:
"bio/Rcorrector/1.0.4-foss-2019a"
script:
"scripts/Rcorrector.py"
############################################################################
rule FilterUncorrectabledPEfastq:
input:
r1=path.join(RESULT_DIR, 'trimmed_reads/corrected/{sample}R1.cor.fq'),
r2=path.join(RESULT_DIR, 'trimmed_reads/corrected/{sample}R2.cor.fq')
output:
o1=path.join(RESULT_DIR, "trimmed_reads/filtered/{sample}R1.fcor.fq"),
o2=path.join(RESULT_DIR, "trimmed_reads/filtered/{sample}R2.fcor.fq")
#group: "cleaning"
envmodules:
"bio/Jellyfish/2.2.6-foss-2017a",
"lang/Python/2.7.13-foss-2017a"
#TODO: load as module
script:
"/scripts/filterUncorrectable.py"
#############################################################################
rule trim_galore:
input:
r1=path.join(RESULT_DIR, "trimmed_reads/filtered/{sample}R1.fcor.fq"),
r2=path.join(RESULT_DIR, "trimmed_reads/filtered/{sample}R2.fcor.fq")
output:
o1=path.join(RESULT_DIR, "trimmed_reads/{sample}.fcor_val1.fq"),
o2=path.join(RESULT_DIR, "trimmed_reads/{sample}.fcor_val2.fq")
threads: 8
#group: "cleaning"
envmodules:
"bio/Trim_Galore/0.6.5-foss-2019a-Python-3.7.4"
params:
"--paired --retain_unpaired --phred33 --length 36 -q 5 --stringency 1 -e 0.1 -j {threads}"
script:
"scripts/trim_galore.py"
In snakemake, you define final output files of the pipeline as target files and define them as inputs in first rule of the pipeline. This rule is traditionally named as all (more recently as targets in snakemake doc).
In your code, rule all specifies input files of the pipeline, which already exists, and therefore snakemake doesn't see anything to do. It just instead needs to specify output files of interest from the pipeline.
rule all:
input:
expand(path.join(RESULT_DIR, "trimmed_reads/{sample}.fcor_val{read}.fq"), sample=FILES, read=[1,2]),
Why your attempted methods didn't work?
-f not working:
As per doc:
--force, -f
Force the execution of the selected target or the first rule regardless of already created output.
Default: False
In your code, this means rule all, which doesn't have output defined, and therefore nothing happened.
filenameR1.fcor_val1.fq
This doesn't match output of any of the rules and therefore the error MissingRuleException.
--forceall
Same reasoning as that for -f flag in your case.
--forceall, -F
Force the execution of the selected (or the first) rule and all rules it is dependent on regardless of already created output.
Default: False
I have both Apache and Modsecurity working together. I'm trying to limit hit rate by request's header (like "facebookexternalhit"). And then return a friendly "429 Too Many Requests" and "Retry-After: 3".
I know I can read a file of headers like:
SecRule REQUEST_HEADERS:User-Agent "#pmFromFile ratelimit-bots.txt"
But I'm getting trouble building the rule.
Any help would be really appreciated. Thank you.
After 2 days of researching and understanding how Modsecurity works, I finally did it. FYI I'm using Apache 2.4.37 and Modsecurity 2.9.2 This is what I did:
In my custom file rules: /etc/modsecurity/modsecurity_custom.conf I've added the following rule:
# Limit client hits by user agent
SecRule REQUEST_HEADERS:User-Agent "#pm facebookexternalhit" \
"id:400009,phase:2,nolog,pass,setvar:global.ratelimit_facebookexternalhit=+1,expirevar:global.ratelimit_facebookexternalhit=3"
SecRule GLOBAL:RATELIMIT_FACEBOOKEXTERNALHIT "#gt 1" \
"chain,id:4000010,phase:2,pause:300,deny,status:429,setenv:RATELIMITED,log,msg:'RATELIMITED BOT'"
SecRule REQUEST_HEADERS:User-Agent "#pm facebookexternalhit"
Header always set Retry-After "3" env=RATELIMITED
ErrorDocument 429 "Too Many Requests"
Explanation:
Note: I want to limit to 1 request every 3 seconds.
The first rule matches the request header user agent against "facebookexternalhit". If the match was succesful, it creates the ratelimit_facebookexternalhit property in the global collection with the initial value of 1 (it will increment this value with every hit matching the user agent). Then, it sets the expiration time of this var in 3 seconds. If we receive a new hit matching "facebookexternalhit" it will sum 1 to ratelimit_facebookexternalhit. If we don't receive hits matching "facebookexternalhit" after 3 seconds, ratelimit_facebookexternalhit will be gone and this process will be restarted.
If global.ratelimit_clients > 1 (we received 2 or more hits within 3 seconds) AND user agent matches "facebookexternalhit" (this AND condition is important because otherwise all requests will be denied if a match is produced), we set RATELIMITED=1, stop the action with a 429 http error, and log a custom message in Apache error log: "RATELIMITED BOT".
RATELIMITED=1 is set just to add the custom header "Retry-After: 3". In this case, this var is interpreted by Facebook's crawler (facebookexternalhit) and will retry operation in the specified time.
We map a custom return message (in case we want) for the 429 error.
You could improve this rule by adding #pmf and a .data file, then initializing global collection like initcol:global=%{MATCHED_VAR}, so you are not limited just to a single match by rule. I didn't test this last step (this is what I needed right now). I'll update my answer in case I do.
UPDATE:
I've adapted the rule to be able to have a file with all user agents I want to rate limit, so a single rule can be used across multiple bots/crawlers:
# Limit client hits by user agent
SecRule REQUEST_HEADERS:User-Agent "#pmf data/ratelimit-clients.data" \
"id:100008,phase:2,nolog,pass,setuid:%{tx.ua_hash},setvar:user.ratelimit_client=+1,expirevar:user.ratelimit_client=3"
SecRule USER:RATELIMIT_CLIENT "#gt 1" \
"chain,id:1000009,phase:2,deny,status:429,setenv:RATELIMITED,log,msg:'RATELIMITED BOT'"
SecRule REQUEST_HEADERS:User-Agent "#pmf data/ratelimit-clients.data"
Header always set Retry-After "3" env=RATELIMITED
ErrorDocument 429 "Too Many Requests"
So, the file with user agents (one per line) is located inside a subdirectory under the same directory of this rule: /etc/modsecurity/data/ratelimit-clients.data. Then we use #pmf to read and parse the file (https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#pmfromfile). We initialize the USER collection with the user agent: setuid:%{tx.ua_hash} (tx.ua_hash is in the global scope in /usr/share/modsecurity-crs/modsecurity_crs_10_setup.conf). And we simply use user as collection instead of global. That's all!
Might be better to use "deprecatevar",
And you can allow a bit bigger burst leneanancy
# Limit client hits by user agent
SecRule REQUEST_HEADERS:User-Agent "#pmf data/ratelimit-clients.data" \
"id:100008,phase:2,nolog,pass,setuid:%{tx.ua_hash},setvar:user.ratelimit_client=+1,deprecatevar:user.ratelimit_client=3/1"
SecRule USER:RATELIMIT_CLIENT "#gt 1" \
"chain,id:100009,phase:2,deny,status:429,setenv:RATELIMITED,log,msg:'RATELIMITED BOT'"
SecRule REQUEST_HEADERS:User-Agent "#pmf data/ratelimit-clients.data"
Header always set Retry-After "6" env=RATELIMITED
ErrorDocument 429 "Too Many Requests"
I'm trying to set up a ModSecurity whitelist for arguments with an unknown name, but matching a value. For example, I want to whitelist any parameter that is a timestamp (e.g. timestamp=2016-01-01 00:00:00). Currently, this triggers rule 981173 (Restricted SQL Character Anomaly Detection Alert - Total # of special characters exceeded)
The following will work, but will skip checks on all parameters if at least one matches, so it doesn't catch the badvalue parameter in https://www.example.com/?timestamp=2016-01-01+00:00:00&badvalue=2016-01-01+00:00:00:00.
SecRule ARGS "#rx ^2[0-9]{3}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]$" \
"id:'99001', phase:1, nolog, pass, t:none, \
ctl:ruleRemoveTargetByTag=OWASP_CRS/WEB_ATTACK/SQL_INJECTION;ARGS"
The following works if I hardcode the parameter name.
SecRule ARGS:timestamp "#rx ^2[0-9]{3}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]$" \
"id:'99001', phase:1, nolog, pass, t:none, \
ctl:ruleRemoveTargetByTag=OWASP_CRS/WEB_ATTACK/SQL_INJECTION;ARGS:timestamp"
I've tried the following, but they haven't worked.
SecRule ARGS "#rx ^2[0-9]{3}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]$" \
"id:'99001', phase:1, nolog, pass, t:none, \
ctl:ruleRemoveTargetByTag=OWASP_CRS/WEB_ATTACK/SQL_INJECTION;/%{MATCHED_VAR_NAME}/"
SecRule ARGS "#rx ^2[0-9]{3}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]$" \
"id:'99001', phase:1, nolog, pass, t:none, \
ctl:ruleRemoveTargetByTag=OWASP_CRS/WEB_ATTACK/SQL_INJECTION;MATCHED_VAR_NAME"
Is this possible with ModSecurity? Is there a way to use MATCHED_VAR_NAME for this use case? I would rather not have to add a rule for every argument name that might contain a timestamp.
Unfortunately, it is currently not possible to use Macro Expansion within the ctl action argument.
As evidence consider the following examples:
SecRule ARGS "#contains bob" "id:1,t:none,pass,ctl:ruleRemoveTargetById=2;ARGS:x"
SecRule ARGS "#contains hello" "id:2,deny,status:403"
When providing the following request: 'http://localhost/?x=bobhello' we will see the following in the debug log when evaluating the second rule
[04/Aug/2016:00:44:07 --0400] [localhost/sid#55e47aa583e0][rid#55e47ad7cb10][/][4] Recipe: Invoking rule 55e47ab14638; [file "/etc/httpd/modsecurity.d/includeOWASP.conf"] [line "12"] [id "2"].
[04/Aug/2016:00:44:07 --0400] [localhost/sid#55e47aa583e0][rid#55e47ad7cb10][/][5] Rule 55e47ab14638: SecRule "ARGS" "#contains hello" "phase:2,log,auditlog,id:2,deny,status:403"
[04/Aug/2016:00:44:07 --0400] [localhost/sid#55e47aa583e0][rid#55e47ad7cb10][/][4] Transformation completed in 0 usec.
[04/Aug/2016:00:44:07 --0400] [localhost/sid#55e47aa583e0][rid#55e47ad7cb10][/][9] fetch_target_exception: Found exception target list [ARGS:x] for rule id 2
[04/Aug/2016:00:44:07 --0400] [localhost/sid#55e47aa583e0][rid#55e47ad7cb10][/][9] fetch_target_exception: Target ARGS:x will not be processed.
[04/Aug/2016:00:44:07 --0400] [localhost/sid#55e47aa583e0][rid#55e47ad7cb10][/][4] Executing operator "contains" with param "hello" against ARGS:x skipped.
[04/Aug/2016:00:44:07 --0400] [localhost/sid#55e47aa583e0][rid#55e47ad7cb10][/][4] Rule returned 0.
However, When we provide the same request ('http://localhost/?x=bobhello') While have Macro Expansion within our ctl action (as follows):
SecRule ARGS "#contains bob" "id:1,t:none,pass,ctl:ruleRemoveTargetById=2;%{MATCHED_VAR_NAME}"
SecRule ARGS "#contains hello" "id:2,deny,status:403"
Our Debug log will appear as follows:
[04/Aug/2016:00:44:41 --0400] [localhost/sid#559f82a0b3e0][rid#559f82d2fb50][/][5] Rule 559f82ac76e8: SecRule "ARGS" "#contains hello" "phase:2,log,auditlog,id:2,deny,status:403"
[04/Aug/2016:00:44:41 --0400] [localhost/sid#559f82a0b3e0][rid#559f82d2fb50][/][4] Transformation completed in 0 usec.
[04/Aug/2016:00:44:41 --0400] [localhost/sid#559f82a0b3e0][rid#559f82d2fb50][/][9] fetch_target_exception: Found exception target list [%{MATCHED_VAR_NAME}] for rule id 2
[04/Aug/2016:00:44:41 --0400] [localhost/sid#559f82a0b3e0][rid#559f82d2fb50][/][4] Executing operator "contains" with param "hello" against ARGS:x.
[04/Aug/2016:00:44:41 --0400] [localhost/sid#559f82a0b3e0][rid#559f82d2fb50][/][9] Target value: "bobhello"
[04/Aug/2016:00:44:41 --0400] [localhost/sid#559f82a0b3e0][rid#559f82d2fb50][/][4] Operator completed in 2 usec.
[04/Aug/2016:00:44:41 --0400] [localhost/sid#559f82a0b3e0][rid#559f82d2fb50][/][4] Rule returned 1.
I cannot think of a method of accomplishing this goal without excessive overhead. At this point the best solution would likely be to manually whitelist each offending argument.
I use procmail to forward certain 'From' to a Gmail account
/home/user/.procmailrc
:0c
* !^FROM_MAILER
* ^From: .*aaa | bbb | ccc.*
! ^X-Loop: user#gmail\.com
| formail -k -X "From:" -X "Subject:" \
-I "To: user#gmail.com" \
-I "X-Loop: user#gmail.com"
:0
* ^From: .*aaa | bbb | ccc.*
$DEFAULT
This works fine but on my server inbox I also get an 'undelivered' mail
The mail system <"^X-Loop:"#my-name-server.com> (expanded from
<"^X-Loop:">): unknown user:
"^x-loop:"
How can I avoid this?
I've tried to delete these mails.
This is not the best way.
Anyway It does not work.
:0B * <"\^X-Loop:"#my-name-server.com>
/dev/null
The recipe contains multiple syntax errors, but the bounce message comes because you lack an asterisk on one of the condition lines, which makes it an action line instead.
The general syntax of a Procmail recipe is
:0flags # "prelude", with optional flags
* condition # optional, can have zero conditions
* condition # ...
action
The action can be a mailbox name, or ! followed by a destination mailbox to forward the message to, or | followed by a shell pipeline.
So your first recipe is "If not from mailer and matching From: ..., forward to ^X-Loop:.
The | formail ... line after that is then simply a syntax error and ignored, because it needs to come after a prelude line :0 and (optionally) some condition lines.
Additionally, the ^From: regex is clearly wrong. It will match From: .*aaa or bbb (with spaces on both sides, in any header, not just the From: header) or ccc.
Finally, the intent is apparently to actually forward the resulting message somewhere.
:0c
* ! ^FROM_MAILER
* ^From:(.*\<)?(aaa|bbb|ccc)
* ! ^X-Loop: user#gmail\.com
| formail -I "X-Loop: user#gmail.com" | $SENDMAIL $SENDMAILFLAGS user#gmail.com
If you simply want to forward the incoming message, the other -X and -I and certainly -k options are superfluous or wrong. If they do accomplish something which is irrelevant for this question, maybe you need to add some or all of them back (and also remember to extract with -X any new headers you add with -I, as otherwise they will be suppressed; this sucks).
Your second recipe is also superfluous, unless you have more Procmail recipes later in the file which should specifically be bypassed for these messages. (If so, you will need to fix the From: regex there as well.)