I'm trying to set up a ModSecurity whitelist for arguments with an unknown name, but matching a value. For example, I want to whitelist any parameter that is a timestamp (e.g. timestamp=2016-01-01 00:00:00). Currently, this triggers rule 981173 (Restricted SQL Character Anomaly Detection Alert - Total # of special characters exceeded)
The following will work, but will skip checks on all parameters if at least one matches, so it doesn't catch the badvalue parameter in https://www.example.com/?timestamp=2016-01-01+00:00:00&badvalue=2016-01-01+00:00:00:00.
SecRule ARGS "#rx ^2[0-9]{3}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]$" \
"id:'99001', phase:1, nolog, pass, t:none, \
ctl:ruleRemoveTargetByTag=OWASP_CRS/WEB_ATTACK/SQL_INJECTION;ARGS"
The following works if I hardcode the parameter name.
SecRule ARGS:timestamp "#rx ^2[0-9]{3}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]$" \
"id:'99001', phase:1, nolog, pass, t:none, \
ctl:ruleRemoveTargetByTag=OWASP_CRS/WEB_ATTACK/SQL_INJECTION;ARGS:timestamp"
I've tried the following, but they haven't worked.
SecRule ARGS "#rx ^2[0-9]{3}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]$" \
"id:'99001', phase:1, nolog, pass, t:none, \
ctl:ruleRemoveTargetByTag=OWASP_CRS/WEB_ATTACK/SQL_INJECTION;/%{MATCHED_VAR_NAME}/"
SecRule ARGS "#rx ^2[0-9]{3}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9]$" \
"id:'99001', phase:1, nolog, pass, t:none, \
ctl:ruleRemoveTargetByTag=OWASP_CRS/WEB_ATTACK/SQL_INJECTION;MATCHED_VAR_NAME"
Is this possible with ModSecurity? Is there a way to use MATCHED_VAR_NAME for this use case? I would rather not have to add a rule for every argument name that might contain a timestamp.
Unfortunately, it is currently not possible to use Macro Expansion within the ctl action argument.
As evidence consider the following examples:
SecRule ARGS "#contains bob" "id:1,t:none,pass,ctl:ruleRemoveTargetById=2;ARGS:x"
SecRule ARGS "#contains hello" "id:2,deny,status:403"
When providing the following request: 'http://localhost/?x=bobhello' we will see the following in the debug log when evaluating the second rule
[04/Aug/2016:00:44:07 --0400] [localhost/sid#55e47aa583e0][rid#55e47ad7cb10][/][4] Recipe: Invoking rule 55e47ab14638; [file "/etc/httpd/modsecurity.d/includeOWASP.conf"] [line "12"] [id "2"].
[04/Aug/2016:00:44:07 --0400] [localhost/sid#55e47aa583e0][rid#55e47ad7cb10][/][5] Rule 55e47ab14638: SecRule "ARGS" "#contains hello" "phase:2,log,auditlog,id:2,deny,status:403"
[04/Aug/2016:00:44:07 --0400] [localhost/sid#55e47aa583e0][rid#55e47ad7cb10][/][4] Transformation completed in 0 usec.
[04/Aug/2016:00:44:07 --0400] [localhost/sid#55e47aa583e0][rid#55e47ad7cb10][/][9] fetch_target_exception: Found exception target list [ARGS:x] for rule id 2
[04/Aug/2016:00:44:07 --0400] [localhost/sid#55e47aa583e0][rid#55e47ad7cb10][/][9] fetch_target_exception: Target ARGS:x will not be processed.
[04/Aug/2016:00:44:07 --0400] [localhost/sid#55e47aa583e0][rid#55e47ad7cb10][/][4] Executing operator "contains" with param "hello" against ARGS:x skipped.
[04/Aug/2016:00:44:07 --0400] [localhost/sid#55e47aa583e0][rid#55e47ad7cb10][/][4] Rule returned 0.
However, When we provide the same request ('http://localhost/?x=bobhello') While have Macro Expansion within our ctl action (as follows):
SecRule ARGS "#contains bob" "id:1,t:none,pass,ctl:ruleRemoveTargetById=2;%{MATCHED_VAR_NAME}"
SecRule ARGS "#contains hello" "id:2,deny,status:403"
Our Debug log will appear as follows:
[04/Aug/2016:00:44:41 --0400] [localhost/sid#559f82a0b3e0][rid#559f82d2fb50][/][5] Rule 559f82ac76e8: SecRule "ARGS" "#contains hello" "phase:2,log,auditlog,id:2,deny,status:403"
[04/Aug/2016:00:44:41 --0400] [localhost/sid#559f82a0b3e0][rid#559f82d2fb50][/][4] Transformation completed in 0 usec.
[04/Aug/2016:00:44:41 --0400] [localhost/sid#559f82a0b3e0][rid#559f82d2fb50][/][9] fetch_target_exception: Found exception target list [%{MATCHED_VAR_NAME}] for rule id 2
[04/Aug/2016:00:44:41 --0400] [localhost/sid#559f82a0b3e0][rid#559f82d2fb50][/][4] Executing operator "contains" with param "hello" against ARGS:x.
[04/Aug/2016:00:44:41 --0400] [localhost/sid#559f82a0b3e0][rid#559f82d2fb50][/][9] Target value: "bobhello"
[04/Aug/2016:00:44:41 --0400] [localhost/sid#559f82a0b3e0][rid#559f82d2fb50][/][4] Operator completed in 2 usec.
[04/Aug/2016:00:44:41 --0400] [localhost/sid#559f82a0b3e0][rid#559f82d2fb50][/][4] Rule returned 1.
I cannot think of a method of accomplishing this goal without excessive overhead. At this point the best solution would likely be to manually whitelist each offending argument.
Related
I've run into a few situations where I would expect Snakemake would complain about ambiguous rules, but doesn't, so I'm trying to figure out exactly what the expected behavior is supposed to be.
For example, with this as rules1.smk:
rule rule1:
output: "something.txt"
rule rule2:
output: "{thing}.txt"
rule rule3:
output: "{thing}.{ext}"
If I request file.txt it complains as expected:
$ snakemake -n --debug-dag -s rules1.smk file.txt
Building DAG of jobs...
candidate job rule2
wildcards: thing=file
candidate job rule3
wildcards: thing=file, ext=txt
AmbiguousRuleException:
Rules rule2 and rule3 are ambiguous for the file file.txt.
...
But if I request something.txt, it goes straight to rule1 and stops at that:
$ snakemake -n --debug-dag -s rules1.smk something.txt
Building DAG of jobs...
candidate job rule1
wildcards:
selected job rule1
...
My question is, why does it do that? Shouldn't it complain that all three of these rules are ambiguous for that output?
My first thought was that rules that yield an output match with no wildcards might implicitly get a higher ruleorder defined than rules that use any number of wildcards, but I can't see anything like that in the documentation for ambiguous rules.
A slightly more complex example shows a little more about the behavior:
if not config.get("noruleorder"):
ruleorder: rule1 > rule1alt
rule rule1:
output: "something.txt"
rule rule1alt:
output: "something.txt"
rule rule2:
output: "{thing}.txt"
rule rule3:
output: "{thing}.{ext}"
That works by default, allowing that ruleorder directive:
$ snakemake -n --debug-dag -s rules2.smk something.txt
Building DAG of jobs...
candidate job rule1
wildcards:
selected job rule1
...
And obviously without ruleorder it can't work, since rule1 and rule1alt are as ambiguous as can be:
$ snakemake --config noruleorder=yep -n --debug-dag -s rules2.smk something.txt
Building DAG of jobs...
candidate job rule1alt
wildcards:
candidate job rule1
wildcards:
candidate job rule2
wildcards: thing=something
candidate job rule3
wildcards: thing=something, ext=txt
AmbiguousRuleException:
Rules rule1alt and rule1 are ambiguous for the file something.txt.
...
...but it's interesting that it then considers all the rules I would have thought would be candidates in the first place. I'm just not sure what that says about the candidate job logic. All this seems related to snakemake: Ambiguous rule not detected? but not quite identical.
This is with Snakemake 7.16.0.
Rules without wildcards are implicitly given a higher ruleorder than those with wildcards. This is noted way back in an old changelog for 3.2.2 as "rules without wildcards now beat other rules in case of ambiguity." There's actually a unit test for this that looks almost exactly like what I set up here. I just couldn't find any of this in the docs.
How I found this:
DAG.update loops over each job and breaks from the loop if it finds a job that is > than all other candidate jobs. Job.__gt__ just calls Rule.__gt__ which calls Ruleorder.compare which does this:
# if no ruleorder given, prefer rule without wildcards
wildcard_cmp = rule2.has_wildcards() - rule1.has_wildcards()
if wildcard_cmp != 0:
return wildcard_cmp
If I comment that out, I get the behavior I originally expected:
$ snakemake -n --debug-dag -s rules1.smk something.txt
Building DAG of jobs...
candidate job rule1
wildcards:
candidate job rule2
wildcards: thing=something
candidate job rule3
wildcards: thing=something, ext=txt
AmbiguousRuleException:
Rules rule1 and rule2 are ambiguous for the file something.txt.
The behavior and test were added in this commit. Unless it's already there and I'm just missing it this should probably get documented in the section about ambiguous rules.
I'm trying to configure modsecurity for Apache to limit the number of times a given resource can be accessed.
I wrote this code, and it works (I'm getting a 429 rejection as I wanted), but I can't reinitiate ip.counter at a certain point of time (last line).
SecAction initcol:ip=%{REMOTE_ADDRESS},pass,nolog,id:132
SecAction "phase:2,setvar:ip.counter=+1,pass,nolog,id:332"
SecRule IP:COUNTER "#ge 1" "phase:3,id:'9000080007',pause:10,deny,status:429,setenv:RATELIMITED,skip:1,nolog,id:232"
SecRule TIME "^10:37:00$" "phase:2,id:'9000080008',setvar:!ip.counter"
However, if I switch the last line to use TIME_HOUR instead, the SecRule does apply correctly:
SecRule TIME_HOUR "#eq 10" "phase:2,id:'9000080008',setvar:!ip.counter"
Any help please for using TIME variable in SecRule to match the exact time?
Congratulations on getting a very advanced recipe to work properly. This is really cool.
Now your rule does not work, because the online reference is wrong about the format of the TIME variable (The Handbook is correct though).
Here is how to debug this on ModSec debug log level 9:
SecRule TIME "#unconditionalMatch" "id:1000,phase:2,pass,log,msg:'Key : Value : |%{MATCHED_VAR_NAME}| : |%{MATCHED_VAR}|'"
Leads to:
...4c20][/][5] Rule 562b28db5420: SecRule "TIME" "#unconditionalMatch " "phase:2,auditlog,id:1007,pass,log,msg:'Key : Value : |%{MATCHED_VAR_NAME}| : |%{MATCHED_VAR}|'"
...4c20][/][4] Transformation completed in 0 usec.
...4c20][/][4] Executing operator "unconditionalMatch" with param "" against TIME.
...4c20][/][9] Target value: "20220829070111"
...4c20][/][4] Operator completed in 0 usec.
...4c20][/][9] Resolved macro %{MATCHED_VAR_NAME} to: TIME
...4c20][/][9] Resolved macro %{MATCHED_VAR} to: 20220829070111
...4c20][/][2] Warning. Unconditional match in SecAction. [file "/apache/conf/httpd.conf_pod_2022-08-29_06:58"] [line "209"] [id "1007"] [msg "Key : Value : |TIME| : |20220829070111|"]
...4c20][/][4] Rule returned 1.
...4c20][/][9] Match -> mode NEXT_RULE.
So as the title says I can't bring my workflow to execute anything, except the all rule...
When Executing the all rule it correctly finds all the input files, so the configfile is okay, every path is correct.
when trying to run without additional tags I get
Building DAG of jobs...
Checking status of 0 jobs.
Nothing to be done
Things I tried:
-f rcorrector -> only all rule
filenameR1.fcor_val1.fq -> MissingRuleException (No Typos)
--forceall -> only all rule
Some more fiddling I can't formulate clearly
please Help
from os import path
configfile:"config.yaml"
RNA_DIR = config["RAW_RNA_DIR"]
RESULT_DIR = config["OUTPUT_DIR"]
FILES = glob_wildcards(path.join(RNA_DIR, '{sample}R1.fastq.gz')).sample
############################################################################
rule all:
input:
r1=expand(path.join(RNA_DIR, '{sample}R1.fastq.gz'), sample=FILES),
r2=expand(path.join(RNA_DIR, '{sample}R2.fastq.gz'), sample=FILES)
#############################################################################
rule rcorrector:
input:
r1=path.join(RNA_DIR, '{sample}R1.fastq.gz'),
r2=path.join(RNA_DIR, '{sample}R2.fastq.gz')
output:
o1=path.join(RESULT_DIR, 'trimmed_reads/corrected/{sample}R1.cor.fq'),
o2=path.join(RESULT_DIR, 'trimmed_reads/corrected/{sample}R2.cor.fq')
#group: "cleaning"
threads: 8
params: "-t {threads}"
envmodules:
"bio/Rcorrector/1.0.4-foss-2019a"
script:
"scripts/Rcorrector.py"
############################################################################
rule FilterUncorrectabledPEfastq:
input:
r1=path.join(RESULT_DIR, 'trimmed_reads/corrected/{sample}R1.cor.fq'),
r2=path.join(RESULT_DIR, 'trimmed_reads/corrected/{sample}R2.cor.fq')
output:
o1=path.join(RESULT_DIR, "trimmed_reads/filtered/{sample}R1.fcor.fq"),
o2=path.join(RESULT_DIR, "trimmed_reads/filtered/{sample}R2.fcor.fq")
#group: "cleaning"
envmodules:
"bio/Jellyfish/2.2.6-foss-2017a",
"lang/Python/2.7.13-foss-2017a"
#TODO: load as module
script:
"/scripts/filterUncorrectable.py"
#############################################################################
rule trim_galore:
input:
r1=path.join(RESULT_DIR, "trimmed_reads/filtered/{sample}R1.fcor.fq"),
r2=path.join(RESULT_DIR, "trimmed_reads/filtered/{sample}R2.fcor.fq")
output:
o1=path.join(RESULT_DIR, "trimmed_reads/{sample}.fcor_val1.fq"),
o2=path.join(RESULT_DIR, "trimmed_reads/{sample}.fcor_val2.fq")
threads: 8
#group: "cleaning"
envmodules:
"bio/Trim_Galore/0.6.5-foss-2019a-Python-3.7.4"
params:
"--paired --retain_unpaired --phred33 --length 36 -q 5 --stringency 1 -e 0.1 -j {threads}"
script:
"scripts/trim_galore.py"
In snakemake, you define final output files of the pipeline as target files and define them as inputs in first rule of the pipeline. This rule is traditionally named as all (more recently as targets in snakemake doc).
In your code, rule all specifies input files of the pipeline, which already exists, and therefore snakemake doesn't see anything to do. It just instead needs to specify output files of interest from the pipeline.
rule all:
input:
expand(path.join(RESULT_DIR, "trimmed_reads/{sample}.fcor_val{read}.fq"), sample=FILES, read=[1,2]),
Why your attempted methods didn't work?
-f not working:
As per doc:
--force, -f
Force the execution of the selected target or the first rule regardless of already created output.
Default: False
In your code, this means rule all, which doesn't have output defined, and therefore nothing happened.
filenameR1.fcor_val1.fq
This doesn't match output of any of the rules and therefore the error MissingRuleException.
--forceall
Same reasoning as that for -f flag in your case.
--forceall, -F
Force the execution of the selected (or the first) rule and all rules it is dependent on regardless of already created output.
Default: False
I have both Apache and Modsecurity working together. I'm trying to limit hit rate by request's header (like "facebookexternalhit"). And then return a friendly "429 Too Many Requests" and "Retry-After: 3".
I know I can read a file of headers like:
SecRule REQUEST_HEADERS:User-Agent "#pmFromFile ratelimit-bots.txt"
But I'm getting trouble building the rule.
Any help would be really appreciated. Thank you.
After 2 days of researching and understanding how Modsecurity works, I finally did it. FYI I'm using Apache 2.4.37 and Modsecurity 2.9.2 This is what I did:
In my custom file rules: /etc/modsecurity/modsecurity_custom.conf I've added the following rule:
# Limit client hits by user agent
SecRule REQUEST_HEADERS:User-Agent "#pm facebookexternalhit" \
"id:400009,phase:2,nolog,pass,setvar:global.ratelimit_facebookexternalhit=+1,expirevar:global.ratelimit_facebookexternalhit=3"
SecRule GLOBAL:RATELIMIT_FACEBOOKEXTERNALHIT "#gt 1" \
"chain,id:4000010,phase:2,pause:300,deny,status:429,setenv:RATELIMITED,log,msg:'RATELIMITED BOT'"
SecRule REQUEST_HEADERS:User-Agent "#pm facebookexternalhit"
Header always set Retry-After "3" env=RATELIMITED
ErrorDocument 429 "Too Many Requests"
Explanation:
Note: I want to limit to 1 request every 3 seconds.
The first rule matches the request header user agent against "facebookexternalhit". If the match was succesful, it creates the ratelimit_facebookexternalhit property in the global collection with the initial value of 1 (it will increment this value with every hit matching the user agent). Then, it sets the expiration time of this var in 3 seconds. If we receive a new hit matching "facebookexternalhit" it will sum 1 to ratelimit_facebookexternalhit. If we don't receive hits matching "facebookexternalhit" after 3 seconds, ratelimit_facebookexternalhit will be gone and this process will be restarted.
If global.ratelimit_clients > 1 (we received 2 or more hits within 3 seconds) AND user agent matches "facebookexternalhit" (this AND condition is important because otherwise all requests will be denied if a match is produced), we set RATELIMITED=1, stop the action with a 429 http error, and log a custom message in Apache error log: "RATELIMITED BOT".
RATELIMITED=1 is set just to add the custom header "Retry-After: 3". In this case, this var is interpreted by Facebook's crawler (facebookexternalhit) and will retry operation in the specified time.
We map a custom return message (in case we want) for the 429 error.
You could improve this rule by adding #pmf and a .data file, then initializing global collection like initcol:global=%{MATCHED_VAR}, so you are not limited just to a single match by rule. I didn't test this last step (this is what I needed right now). I'll update my answer in case I do.
UPDATE:
I've adapted the rule to be able to have a file with all user agents I want to rate limit, so a single rule can be used across multiple bots/crawlers:
# Limit client hits by user agent
SecRule REQUEST_HEADERS:User-Agent "#pmf data/ratelimit-clients.data" \
"id:100008,phase:2,nolog,pass,setuid:%{tx.ua_hash},setvar:user.ratelimit_client=+1,expirevar:user.ratelimit_client=3"
SecRule USER:RATELIMIT_CLIENT "#gt 1" \
"chain,id:1000009,phase:2,deny,status:429,setenv:RATELIMITED,log,msg:'RATELIMITED BOT'"
SecRule REQUEST_HEADERS:User-Agent "#pmf data/ratelimit-clients.data"
Header always set Retry-After "3" env=RATELIMITED
ErrorDocument 429 "Too Many Requests"
So, the file with user agents (one per line) is located inside a subdirectory under the same directory of this rule: /etc/modsecurity/data/ratelimit-clients.data. Then we use #pmf to read and parse the file (https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#pmfromfile). We initialize the USER collection with the user agent: setuid:%{tx.ua_hash} (tx.ua_hash is in the global scope in /usr/share/modsecurity-crs/modsecurity_crs_10_setup.conf). And we simply use user as collection instead of global. That's all!
Might be better to use "deprecatevar",
And you can allow a bit bigger burst leneanancy
# Limit client hits by user agent
SecRule REQUEST_HEADERS:User-Agent "#pmf data/ratelimit-clients.data" \
"id:100008,phase:2,nolog,pass,setuid:%{tx.ua_hash},setvar:user.ratelimit_client=+1,deprecatevar:user.ratelimit_client=3/1"
SecRule USER:RATELIMIT_CLIENT "#gt 1" \
"chain,id:100009,phase:2,deny,status:429,setenv:RATELIMITED,log,msg:'RATELIMITED BOT'"
SecRule REQUEST_HEADERS:User-Agent "#pmf data/ratelimit-clients.data"
Header always set Retry-After "6" env=RATELIMITED
ErrorDocument 429 "Too Many Requests"
In modsecurity default-script:
base_rules/modsecurity_crs_20_protocol_violations.conf
there is a rule, 960011:
SecRule REQUEST_METHOD "^(?:GET|HEAD)$" \
"msg:'GET or HEAD Request with Body Content.',\
severity:'2',\
id:'960011',\
ver:'OWASP_CRS/2.2.9',\
rev:'1',\
maturity:'9',\
accuracy:'9',\
phase:1,\
block,\
logdata:'%{matched_var}',\
t:none,\
tag:'OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ',\
tag:'CAPEC-272',\
chain"
SecRule REQUEST_HEADERS:Content-Length "!^0?$"\
"t:none,\
setvar:'tx.msg=%{rule.msg}',\
setvar:tx.anomaly_score=+%{tx.critical_anomaly_score},\
setvar:'tx.%{rule.id}-OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ-%{matched_var_name}=%{matched_var}'"
I only want to disable logging for this rule (it gives too many false positives),
and therefore add my own script
base_rules/z99_logging_suppress.conf
to remove the default-rule and create a new identical rule -- only without logging:
SecRuleRemoveById 960011
SecRule REQUEST_METHOD "^(?:GET|HEAD)$" \
"msg:'GET or HEAD Request with Body Content.',\
severity:'2',\
id:'9960011',\
ver:'OWASP_CRS/2.2.9',\
rev:'1',\
maturity:'9',\
accuracy:'9',\
phase:1,\
block,nolog,\
logdata:'%{matched_var}',\
t:none,\
tag:'OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ',\
tag:'CAPEC-272',\
chain"
SecRule REQUEST_HEADERS:Content-Length "!^0?$"\
"t:none,\
setvar:'tx.msg=%{rule.msg}',\
setvar:tx.anomaly_score=+%{tx.critical_anomaly_score},\
setvar:'tx.%{rule.id}-OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ-%{matched_var_name}=%{matched_var}'"
The only differences to the original rule are the new id 9960011, and the nolog additions:
...
id:'9960011',\
...
block,nolog,\
...
But when I restart httpd with this additional rule, I get error:
AH00526: Syntax error on line 18 of /path/base_rules/z99_logging_suppress.conf:
ModSecurity: Execution phases can only be specified by chain starter rules.
The same strategy --- SecRuleRemoveById + then re-create it with new id --- works for all other default-rules I tried, but not for this one.
Anyone can tell me why that is?
It basically says that the phase command can only be in the first rule in a chain and not in a subsequent rule which forms part of the chain.
There is nothing wrong with the rule as you have written it, phase is only specified in the first SecRule. In fact I've tried it on my instance and it works. So either one of two things has gone wrong:
You have copied and pasted it incorrectly into this question.
The rule above where you have defined this, has chain in it and so has left an open chain, that your rule 9960011 is then effectively trying to continue on from.
Or something else weird is happening! But I'm going with 1 or 2 for now :-)