Given this code in iPython
df1=df["BillingContactCountry"].value_counts()
df1
I get
United States 4138
Germany 1963
United Kingdom 732
Switzerland 528
Australia 459
Canada 369
Japan 344
France 303
Netherlands 285
I want to get a series with count larger than 303, what should I do?
You needboolean indexing:
print (df1[df1 > 303])
United States 4138
Germany 1963
United Kingdom 732
Switzerland 528
Australia 459
Canada 369
Japan 344
Name: BillingContactCountry, dtype: int64
Related
In a dataframe like this:
...
match team opponent venue
233 3b0345fb Brazil Argentina Home
234 3b2357fb Argentina Brazil Away
427 3b0947fb England Poland Home
...
how can I select one dataframe slice, based on a column value (df[df['team']=='England']), like this:
...
match team opponent venue
559 4a3eae2f England Poland Home
...
And add inverted rows of that slice to the original dataframe, changing 'Home' with 'Away', ending up with:
...
match team opponent venue
233 3b0345fb Brazil Argentina Home
234 3b2357fb Argentina Brazil Away
559 3b0947fb England Poland Home
560 3b0947fb Poland England Away
...
Note: This slice should contain n rows and produce n inverted rows.
You can use:
df2 = df[df['team'].eq('England')].copy()
df2[['team', 'opponent']] = df2[['opponent', 'team']]
df2['venue'] = df2['venue'].map({'Home': 'Away', 'Away': 'Home})
out = pd.concat([df, df2])
print(out)
Output:
match team opponent venue
233 3b0345fb Brazil Argentina Home
234 3b2357fb Argentina Brazil Away
427 3b0947fb England Poland Home
427 3b0947fb Poland England Away
If you want to invert all:
df2 = df.copy()
df2[['team', 'opponent']] = df2[['opponent', 'team']]
df2['venue'] = df2['venue'].map({'Home': 'Away', 'Away': 'Home})
out = pd.concat([df, df2])
output:
match team opponent venue
233 3b0345fb Brazil Argentina Home
234 3b2357fb Argentina Brazil Away
427 3b0947fb England Poland Home
233 3b0345fb Argentina Brazil Away
234 3b2357fb Brazil Argentina Home
427 3b0947fb Poland England Away
My datasets:
df1:
Country
Year
Dem
Turkiye
1993
152
Spain
1993
223
Spain
1994
166
USA
1993
123
df2 (it has many more columns but I only want to use "Index":
Country
Year
Index
Turkiye
1993
0
Spain
1993
1
Spain
1994
1
USA
1993
1
Use pandas.merge as shown below :
df1.merge(df2[['Country', 'Year', 'Index']], on=['Country', 'Year'], how='left')
Look at the following data file（cou.data） which has four fields separated by tab.
Four fields are:
country name
land area
population
continent
As for country name or continent name which has two words, two words are separated by space.
(Data are not accurately confirmed, just for test purpose)
USSR 8649 275 Asia
Cananda 3852 25 North America
China 3705 1032 Asia
USA 3615 237 North America
Brazil 3286 134 South America
India 1267 746 Asia
Mexico 762 78 North America
France 211 55 Europe
Japan 144 120 Asia
Germany 96 61 Europe
England 94 56 Europe
Taiwan 55 144 Asia
North Korea 44 2134 Asia
awk 'BEGIN { FS = "\t" } { print $1, "---", $4 }' cou.data
I got the output which exactly meets my anticipation:
USSR --- Asia
Cananda --- North America
China --- Asia
USA --- North America
Brazil --- South America
India --- Asia
Mexico --- North America
France --- Europe
Japan --- Asia
Germany --- Europe
England --- Europe
Taiwan --- Asia
North Korea --- Asia
Then I replace \t by one space (" ")
That is :
awk 'BEGIN { FS = " " } { print $1, "---", $4 }' cou.data
The output I got is not understandable to me
USSR --- Asia
Cananda --- North
China --- Asia
USA --- North
Brazil --- South
India --- Asia
Mexico --- North
France --- Europe
Japan --- Asia
Germany --- Europe
England --- Europe
Taiwan --- Asia
North --- 2134
Line 2,4,5,7,13 each have one space and the other lines have no space(s) at all.
As for lines that have no space, why $1, $4 still can be printed ?
As for line 2,4,5,7,13, I thought $1 should be printed like this:
Cananda 3852 25 North
USA 3615 237 North
Brazil 3286 134 South
Mexico 762 78 North
North
And $4 does not exist.
Where did I get wrong ?
So problem here is string/country names on 1st field which are having spaces in their names for example North Korea. So when you are setting FS as \t this string will be considered as a single field on the other hand when you will set FS as space this will be considered as 2 different fields. That is why you are seeing difference between field numbers after changing the FS values in your codes.
I would suggest your first attempt is good enough to get your expected values.
I am trying to understand how the range pattern matching work in Awk
Here is the full data that I am practicing with
Raw Data
-----------------------------------------
USSR 8649 275 Asia
Canada 3852 25 North America
China 3705 1032 Asia
USA 3615 237 North America
Brazil 3286 134 South America
India 1267 746 Asia
Mexico 762 78 North America
France 211 55 Europe
Japan 144 120 Asia
Germany 96 61 Europe
England 94 56 Europe
If I write this code
$ awk '/Asia/, /Europe/' countries.awk
I get
USSR 8649 275 Asia
Canada 3852 25 North America
China 3705 1032 Asia
USA 3615 237 North America
Brazil 3286 134 South America
India 1267 746 Asia
Mexico 762 78 North America
France 211 55 Europe
Japan 144 120 Asia
Germany 96 61 Europe
It doesn't output England.
And If I write this
$ awk '/Europe/, /Asia/' countries.awk
I get
France 211 55 Europe
Japan 144 120 Asia
Germany 96 61 Europe
England 94 56 Europe
What is the behavior here? Why do I not get England on the first one?
Awk process input lines one at a time, the syntax you used is likely to print lines from the start to the end pattern, represented by country names. When you used
awk '/Asia/, /Europe/'
The start of pattern Asia happens more than once. As you can see from the line numbers below, numbers 3,5,8 and 11 represent the start of the pattern and the pattern ends at lines 10 and 12. Observe carefully the sub-ranges of lines between 8-10 and 11-12. The last end pattern Europe for the last Asia ends at line 12, that is the reason you are not seeing England in the first case.
But when you used
awk '/Europe/, /Asia/'
The line containing the first start pattern Europe starts at line 10 and ends at 11 another two pattern start at 12 and 13 without an end pattern Asia, so it would obviously print all the lines until Asia appears. So you are seeing England in the second case.
$ cat -n file
1 Raw Data
2 -----------------------------------------
3 USSR 8649 275 Asia
4 Canada 3852 25 North America
5 China 3705 1032 Asia
6 USA 3615 237 North America
7 Brazil 3286 134 South America
8 India 1267 746 Asia
9 Mexico 762 78 North America
10 France 211 55 Europe
11 Japan 144 120 Asia
12 Germany 96 61 Europe
13 England 94 56 Europe
Never use range expressions as they make trivial tasks very slightly briefer but then need a complete rewrite or duplicate conditions when your requirements change. Always use a flag instead:
awk '/Asia/{f=1} f{print} /Europe/{f=0}' countries.awk
I bet if you started with that you wouldn't even have had to ask this question as the logic is clear and explicit.
I have two multi-level Series and would like to merge them according to both index. The first Series looks like this:
# of restaurants
BORO CUISINE
BRONX American 425
Chinese 330
Pizza 206
BROOKLYN American 1254
Chinese 750
Cafe/Coffee/Tea 350
The second one has more rows and is like this:
# of votes
BORO CUISINE
BRONX American 2425
Caribbean 320
Chinese 3130
Pizza 3336
BROOKLYN American 21254
Caribbean 2320
Chinese 7250
Cafe/Coffee/Tea 3350
Pizza 13336
Setup:
s1 = pd.Series({('BRONX', 'American'): 425, ('BROOKLYN', 'Chinese'): 750, ('BROOKLYN', 'Cafe/Coffee/Tea'): 350, ('BRONX', 'Pizza'): 206, ('BROOKLYN', 'American'): 1254, ('BRONX', 'Chinese'): 330})
s2 = pd.Series({('BRONX', 'Caribbean'): 320, ('BRONX', 'American'): 2425, ('BROOKLYN', 'Chinese'): 7250, ('BROOKLYN', 'Cafe/Coffee/Tea'): 3350, ('BRONX', 'Pizza'): 3336, ('BROOKLYN', 'American'): 21254, ('BROOKLYN', 'Pizza'): 13336, ('BRONX', 'Chinese'): 3130, ('BROOKLYN', 'Caribbean'): 2320})
s1 = s1.rename_axis(['BORO','CUISINE']).rename('restaurants')
s2 = s2.rename_axis(['BORO','CUISINE']).rename('votes')
print (s1)
BORO CUISINE
BRONX American 425
Chinese 330
Pizza 206
BROOKLYN American 1254
Chinese 750
Cafe/Coffee/Tea 350
Name: restaurants, dtype: int64
print (s2)
BORO CUISINE
BRONX American 2425
Caribbean 320
Chinese 3130
Pizza 3336
BROOKLYN American 21254
Caribbean 2320
Chinese 7250
Cafe/Coffee/Tea 3350
Pizza 13336
Name: votes, dtype: int64
Use concat with parameter join if need inner join:
print (pd.concat([s1,s2], axis=1, join='inner'))
restaurants votes
BORO CUISINE
BRONX American 425 2425
Chinese 330 3130
Pizza 206 3336
BROOKLYN American 1254 21254
Cafe/Coffee/Tea 350 3350
Chinese 750 7250
#join='outer' is by default, so can be omited
print (pd.concat([s1,s2], axis=1))
restaurants votes
BORO CUISINE
BRONX American 425.0 2425
Caribbean NaN 320
Chinese 330.0 3130
Pizza 206.0 3336
BROOKLYN American 1254.0 21254
Cafe/Coffee/Tea 350.0 3350
Caribbean NaN 2320
Chinese 750.0 7250
Pizza NaN 13336
Another solution is use merge with reset_index:
#by default how='inner', so can be omited
print (pd.merge(s1.reset_index(), s2.reset_index(), on=['BORO','CUISINE']))
BORO CUISINE restaurants votes
0 BRONX American 425 2425
1 BRONX Chinese 330 3130
2 BRONX Pizza 206 3336
3 BROOKLYN American 1254 21254
4 BROOKLYN Chinese 750 7250
5 BROOKLYN Cafe/Coffee/Tea 350 3350
#outer join
print (pd.merge(s1.reset_index(), s2.reset_index(), on=['BORO','CUISINE'], how='outer'))
BORO CUISINE restaurants votes
0 BRONX American 425.0 2425
1 BRONX Chinese 330.0 3130
2 BRONX Pizza 206.0 3336
3 BROOKLYN American 1254.0 21254
4 BROOKLYN Chinese 750.0 7250
5 BROOKLYN Cafe/Coffee/Tea 350.0 3350
6 BRONX Caribbean NaN 320
7 BROOKLYN Caribbean NaN 2320
8 BROOKLYN Pizza NaN 13336