I want to block every country except mine, so I downloaded the GeoLite2 database and added it in the crs-setup.conf file. Under -=[ Block Countries ]=- I also added every country code for testing.
This did not work and after trying multiple alternative "country blocking" rules I looked into the debug log and saw that the rule itself was working, but it wasn't finding any geo data for the IP:
Recipe: Invoking rule 72bef6b0; [file "/etc/modsecurity/rules/REQUEST-910-IP-REPUTATION.conf"] [line "75"] [id "910100"].
Rule 72bef6b0: SecRule "TX:HIGH_RISK_COUNTRY_CODES" "!#rx ^$" "phase:2,log,auditlog,id:910100,drop,t:none,msg:'Client IP is from a HIGH Risk Country Location',logdata:%{MATCHED_VAR},tag:application-multi,tag:language-multi,tag:platform-multi,tag:attack-reputation-ip,tag:paranoia-level/1,tag:OWASP_CRS,ver:OWASP_CRS/3.3.2,severity:CRITICAL,chain"
Transformation completed in 8 usec.
Executing operator "!rx" with param "^$" against TX:high_risk_country_codes.
Target value: "AD AE AF AG AI AL AM AO AQ AR AS AT AU AW AX AZ BA BB BD BE BF BG BH BI BJ BL BM BN BO BQ BR BS BT BV BW BY BZ CA CC CD CF CG CH CI CK CL CM CN CO CR CU CV CW CX CY CZ DE DJ DK DM DO DZ EC EE EG EH ER ES ET FI FJ FK FM FO FR GA GB GD GE GF GG GH GI GL GM GN GP GQ GR GS GT GU GW GY HK HM HN HR HT HU ID IE IL IM IN IO IQ IR IS IT JE JM JO JP KE KG KH KI KM KN KP KR KW KY KZ LA LB LC LI LK LR LS LT LU LV LY MA MC MD ME MF MG MH MK ML MM MN MO MP MQ MR MS MT MU MV MW MX MY MZ NA NC NE NF NG NI NL NO NP NR NU NZ OM PA PE PF PG PH PK PL PM PN PR PS PT PW PY QA RE RO RS RU RW SA SB SC SD SE SG SH SI SJ SK SL SM SN SO SR SS ST SV SX SY SZ TC TD TF TG TH TJ TK TL TM TN TO TR TT TV TW TZ UA UG UM US UY UZ VA VC VE VG VI VN VU WF WS YE YT ZA ZM ZW"
Operator completed in 20 usec.
Rule returned 1.
Match -> mode NEXT_RULE.
Recipe: Invoking rule 72eb4298; [file "/etc/modsecurity/rules/REQUEST-910-IP-REPUTATION.conf"] [line "77"].
Rule 72eb4298: SecRule "TX:REAL_IP" "#geoLookup " "chain"
Transformation completed in 2 usec.
Executing operator "geoLookup" with param "" against TX:real_ip.
Target value: "###.##.#.###"
GEO: Looking up "###.##.#.###".
GEO: Using address "###.##.#.###" (0x########). ##########
No geo data for "###.##.#.###" (country -4431872).
Operator completed in 10205 usec.
Rule returned 0.
However the IP is in the database as I checked it with geoip2 in Python and it returned the correct country for said IP.
Is there anything obvious I missed?
ModSecurity does NOT support new GeoIP2 format of GeoIP database so old, legacy, format need to be used.
Related
I have the below query.. I need to list the sales rep as columns and the information in the DESCRIP under the appropriate sales rep. Pivot doesn't work as I am not aggregating anything. Haven't found something that is clicking with me so I am asking for your help.. Thank you for your time.
SALES_REP DESCRIP
AG (AXI01) EGCG EXTENDED RELEASE CAPSULES (RD)
AG (DLI01) NWE-01 ESSENTIAL DIGESTIVES ()
AG (KID01) KIDNEYHOOD S3 (UT)
AG (AXI01) AB-4166 DISSOLUTION TESTING (RD)
AG (AXI01) MCC SPHERES COATED W BLACK IRON OXIDE (RD)
AG (ELY01) MATTER REFORMULATION (RON/A)
BD (SMA01) VEGETARIAN PROTEIN BLEND W/GREENS (RD)
BD (MIT01) ARGIINE AAKG GARN AND TABLETS (ANGEL)
BD (DLI01) BISMUTH/FURAZOLIDONE/CLARITHROMYCIN COMBINATION DRUG PRODUCT_UTECH GUM (NL)
BD (00390) COAT 8 KILOS OF TABLET CORES WITH ABOUT 1 % COATING THAT HAS ADDED SAFETRACES DNA MARKER ()
BD (EEX02) GRANULATE SOY PROTEIN WITH ACACIA ()
BD (LIQ01) GRANULATE SAMPLES WITH WATER ONLY , SPRAY DRIED LEMON, ORANGE , GRAPEFUIT POWDERS. BASE ALREADY CONTAINS 25-30% ACACIA GUM ()
BD (BRA01) FOR QUOTING PURPOSES , DEVELOP FORMULA FOR DEXBROMPHENIRAMINE MALEATE 2 MG AND PHENAZOPYRIDE HCL (AZO) 100 & 200 MG TABS ()
CB (NUS04) JUNGAMALS JAPAN 2021 25 BOTTLE R&D PILOT FOR STABILITY TESTS (ANGEL)
CB (NUS01) US DIGESTIVE 90 COUNT VERSION OF NUS292C (ANGEL)
CB (LIM01) QUICK DISSOLVE TABLET - RTA SHOT (BERRY) (UT)
CB (LIM01) QUICK DISSOLVE TABLET - THRIVE-HGH (TROPICAL) (UT)
CB (AMB01) AMBRA/LOVIDIA XR FORMULA WITH GYMNEMA SLYVESTRE 465 MG (RD)
CB (SHA02) RITESTART KIDS & TEENS REFORMULATION (UT)
CB (BAS01) SEROVITAL B CAPSULE FORMULA OPTIONS - YELLOW PWDR IN CLEAR & "00" WHITE CASPULE SAMPLES (USHA)
CB (NUS01) TR90 FIT US PLACEBO 20 KG RUN 60 CNT APPROX 600-700 UNLABELED BOTTLES (ANGEL)
MG (AME02) POWDER BLENDING (ANGEL)
MG (ZER01) CHRONDOITIN SULFATE COATING X2 (AG)
MG (ELU01) TOBRAMYCIN GRANULES (NL)
MG (BRI01) BRILLIANT HEALTH TRANSMIT GRAN AND TABLET (UT)
RND (DLI01) EFFERVESCENT PAIN PROCESS VALIDATION (RD)
RND (DLI01) GRAN D MAGNET QUALIFICATION (RD)
RND (DLI01) BLENDER CAPACITY STUDY (RD)
RND (DLI01) COATING PAN A&B QUALIFICATION (RD)
RND (WYE06) SODD-006 STABILITY EVALUATION (NL)
RND (WYE06) CAL160 PROCESS VALIDATION (RD)
SH (LIV02) FORMULATION OF HYDRATION DRINK MIX (USHA)
SH (KOB01) ASPIRIN CAFFEINE POWDER (USHA)
SH (COP01) APPLE CIDER VINEGAR EFFERVESCENT TABLETS (USHA)
SH (COP01) SURVIVAL TABLETS 720CT (NATE)
SH (USA01) GRANULATE 10 KILO POWDER BLEND - USANA (USHA)
TV (LIF01) PETANDIM LIF-07T AND LIF-13T NEW OMEGA (ANGEL)
TV (LIF01) L-CARNITINE ISSUE NRF1 JP (ANGEL)
TV (SHA01) SOY-02 REPLACEMENT IN SHK-06 AND SHK-22/25 PRODUCTS (AG)
TV (ELY01) PROBIOTIC CAPSULES ()
TV (VYM01) VYMUNE CHEWABLE TABLETS (UT)
TV (QUL01) IMMUNITY BOOSTER (UT)
TV (DOT01) DOT-06 UPDATED FORMULA WITH NEW OIL, STEVIA # 95% NEED 2 OPTIONS ()
TV (DOT01) MITO2MAX ()
TV (DOT01) TERRAZYME ()
TV (LIF02) GRANULATE 2 PREBLENDED POWDER PRODUCTS , PARTYAID AND FOCUSAID . USE WATER ONLY ()
TV (PAR01) CALCIUM CITRATE WITH PROBIOTIC STICKPACK ()
SELECT SALES_REP, '(' + CUSTOMER_ID + ') ' + PROJECT_DESCRIPTION + ' (' + IsNull(RD_REP,'') + ')' as DESCRIP
FROM DLI_RND_PROJECTS
WHERE (STATUS = N'OPEN')
ORDER BY SALES_REP
So something like this
AG BD CB ETC
-------------------------------------------------
AXI01... SMA01... NUS04...
DLI01... MIT01... NUS01...
KID01... DLI01... LIM01...
Hmmm . . . I think you want:
select max(case when sales_rep = 'AG' then col end) as ag,
max(case when sales_rep = 'BD' then col end) as bd,
. . .
from (select t.*, row_number() over (partition by sales_rep, col order by col) as seqnum
from t cross apply
(values (substring(descr, 2, 5))) v(col)
) t
group by seqnum;
In my datasets, I have one variable which contains 30% missing value.
I am trying to use tree based model but not getting clear picture how to implement it.
data['X'].value_counts()
OUTPUT-----
? 39454
MC 32223
HM 6197
SP 4892
BC 4569
MD 3473
CP 2493
UN 2366
CM 1932
OG 1020
PO 585
DM 536
CH 145
WC 130
OT 94
MP 79
SI 52
FR 1
The approach which I am trying to implement is:
Suppose this variable has 24 distinct category. And the above is the value count output. ? is the missing value and I should impute the value among rest of the value mentioned with the help of tree based model.
Different categories are MC HM SP BC MD CP UN CM OG PO DM CH WC OT MP ST FR ? and count of ? is 39454. So we have 39454 missing values that we should impute with the help of tree based model
Now, with the help of above values I have to train a model and predict the missing value.
I would recommend below :
Take non-missing data and perform clustering
assign labels for missing data using appropriate cluster
df_lyrics['Lyrics']
0 \n\n--Male--\nAaaaa Aaaaa\n--Female--\nAaaaaa\...
1 \n\n--Male1--\nAnkhiyon Hi Ankhiyon Mein\nRati...
2 \n\n--Male1--\nAray Peeli Chotiyaan,\nHawaeyn ...
3 \n\nPyar Itna Na Kar\nYeh Dil Jaata Hai Bhar\n...
4 Zaraa Maara Maara Sa\nJaane Kyun Dil Ye Ban Ba...
...
1286 \n\nKaara fankaara kab aaye re\nKaara fankaar...
1287 \n\nZameen-o-aasmaan ne kya baat ki hai\nGira...
1288 \n\nMaula Wa Sallim wassalim da-iman abadan\n...
1289 \n\nBhavara\nRe ga re ga re ga re ga pa ma ga...
1290 \n\nArre udi udi udi... udi jaye..\n\nUdi udi...
Name: Lyrics, Length: 1291, dtype: object
I want to convert all these into a single paragraph....
Please help me.
You can join the strings using pd.Series sum:
print(df_lyrics['Lyrics'].sum())
I made a 3x3 table that all cell is stroked but the outer lines are thick by Excel. Then exported to PDF and got the following stream (decoded from Flate).
q
49.56 728.5 169.25 60.12 re
W* n
/P <</MCID 0>> BDC /GS6 gs
0 g
49.56 729.58 2.88 59.04 re
f*
0.14 w
/GS7 gs
0 G
2 J 1 j
105.68 785.68 m
105.68 732.52 l
S
105.62 732.46 0.96 53.28 re
f*
I don't believe that this stream is complete to draw the table. Is something missed or a kind of interpolation working? Of course table is rendered properly on Acrobat Reader.
The image is just to explain and is not what I made so its size differs.
Update
Please refer to the actual file:
https://drive.google.com/file/d/1We2jri-Y04fBYJcZ96HIs05DPeTSDOIc/view?usp=sharing
Actually the content stream is a bit longer:
q
49.56 728.5 169.25 60.12 re
W*
n
/P <</MCID 0 >> BDC
/GS6 gs
0 g
49.56 729.58 2.88 59.04 re
f*
0.14 w
/GS7 gs
0 G
2 J
1 j
105.68 785.68 m
105.68 732.52 l
S
105.62 732.46 0.96 53.28 re
f*
160.76 785.68 m
160.76 732.52 l
S
160.7 732.46 0.96001 53.28 re
f*
214.85 729.58 2.88 56.16 re
f*
52.44 785.74 165.29 2.88 re
f*
52.5 768.76 m
214.79 768.76 l
S
52.44 767.86 162.41 0.96 re
f*
52.5 750.28 m
214.79 750.28 l
S
52.44 749.38 162.41 0.96 re
f*
52.44 729.58 165.29 2.88 re
f*
Q
EMC
All lines are drawn using filled narrow rectangles. The inner lines additionally are drawn as stroked single-edge paths. Those latter stroked paths are not necessary for the appearance.
You probably should check the code for FLATE decoding the content stream which returned you only a partial result.
I am a beginner to Python. I am analyzing bus headways for each stop along a bus route. For each stop, I have a list of headways. The count of headway can be different for each stop. To visualize data, I want to plot boxplots on the same page so that you can observe how bus bunching occurs over the route. For this, I developed a code that reads bus data from a .csv file into a stop dictionary with the name as key and values as an object (I trace some other aspects of the stop but not included here for brevity). The trouble I am having is related to the boxplot. I thought pandas would provide some ease to do this. But, I had lots of trouble trying to set up a dataframe, because my dictionary includes objects. You may have other ideas. I simplified my code to a minimum so that you can still follow what I did. As a side note, I was trying to learn how to use classes while I was working on this analysis. That’s why you see bunch of classes in my code. In my full code, I deal with duplicate vehicles and outliers in their own methods.
stops={}
stopNamesA=[]
headwaysA=[]
class Data:
def __init__(self):
self.depart = 0
self.vehicle = 0
class Stop:
def __init__(self):
self.vehicles = []
self.departs = []
self.headways=[]
self.stopName =""
def AddData(self, line):
fields = line.split(",")
self.stopName = fields[3]
self.vehicles.append(fields[0])
x = fields[4]
self.departs.append(datetime.datetime.strptime(x[:-1], "%m/%d/%y %I:%M:%S %p"))
def CalcHeadway(self):
for i in range(len(self.departs)-1):
dt = self.departs[i]
dt2 = self.departs[i+1]
self.headways.append(datetime.timedelta.total_seconds(dt2 - dt))
with open('data.csv','r') as f:
for line in f:
fields = line.split(",")
sid = str(fields[3])
if (fields[1] == 'X2' and fields[2] == 'WEST'):
if sid in stops.keys():
s = stops[sid]
else:
s = Stop()
stops[sid] = s
s.AddData(line)
for key, value in stops.items():
value.CalcHeadway()
The data looks like the following (I truncated other parts again):
5401 X2 WEST H ST NW + 7TH ST NW 10/3/16 7:58:48 AM
2835 X2 WEST H ST NW + 7TH ST NW 10/3/16 8:16:49 AM
2460 X2 WEST H ST NW + 7TH ST NW 10/3/16 8:20:12 AM
2460 X2 WEST H ST NW + 7TH ST NW 10/3/16 8:20:38 AM
2460 X2 WEST H ST NW + 7TH ST NW 10/3/16 8:20:57 AM
5404 X2 WEST I ST + 14TH ST 10/3/16 8:01:55 AM
2835 X2 WEST I ST + 14TH ST 10/3/16 8:24:01 AM
2853 X2 WEST I ST + 14TH ST 10/3/16 9:27:07 AM
5404 X2 WEST I ST + 14TH ST 10/3/16 9:45:43 AM
2835 X2 WEST I ST + 14TH ST 10/3/16 9:57:31 AM
2831 X2 WEST MINNESOTA AVE NE + BENNING RD NE 10/3/16 8:02:41 AM
2821 X2 WEST MINNESOTA AVE NE + BENNING RD NE 10/3/16 8:17:42 AM
5420 X2 WEST MINNESOTA AVE NE + BENNING RD NE 10/3/16 8:34:43 AM
2853 X2 WEST MINNESOTA AVE NE + BENNING RD NE 10/3/16 8:44:14 AM
5401 X2 WEST MINNESOTA AVE NE + BENNING RD NE 10/3/16 9:02:20 AM
First, as the Error suggests, 'Series' object has no attribute 'boxplot'. You can draw a boxplot from a Series by Series.plot.box().
However since you want multiple boxes to appear, it makes sense to use a dataframe. So what you need is a DataFrame to plot your boxplot from.
If I understand your needs correctly, you'd need a DataFrame with 26 columns, one column per bus stop.
import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame()
df["I ST + 14TH ST"] = [1107.0, 1359.0, 1859.0, 1190.0, 1071.0, 904.0]
df["BENNING RD NE + 19TH ST NE"] = [1132.0, 1503.0, 1448.0, 1344.0, 958.0, 771.0]
#......
df["H ST NW + 5TH ST NW"] = [1182.0, 1315.0, 1691.0, 1193.0, 956.0, 729.0]
df.boxplot(rot=45)
plt.tight_layout()
plt.show()
It seems that in order to get a working dataframe out of the stops dictionary, one can do.
stops_for_drawing = {}
for key, val in stops.iteritems():
stops_for_drawing.update({key: val.headways})
df = pd.DataFrame(stops_for_drawing)