Why AWK print is strange when I set FS = " " instead of FS = "\t"? - awk

Look at the following data file(cou.data) which has four fields separated by tab.
Four fields are:
country name
land area
population
continent
As for country name or continent name which has two words, two words are separated by space.
(Data are not accurately confirmed, just for test purpose)
USSR 8649 275 Asia
Cananda 3852 25 North America
China 3705 1032 Asia
USA 3615 237 North America
Brazil 3286 134 South America
India 1267 746 Asia
Mexico 762 78 North America
France 211 55 Europe
Japan 144 120 Asia
Germany 96 61 Europe
England 94 56 Europe
Taiwan 55 144 Asia
North Korea 44 2134 Asia
awk 'BEGIN { FS = "\t" } { print $1, "---", $4 }' cou.data
I got the output which exactly meets my anticipation:
USSR --- Asia
Cananda --- North America
China --- Asia
USA --- North America
Brazil --- South America
India --- Asia
Mexico --- North America
France --- Europe
Japan --- Asia
Germany --- Europe
England --- Europe
Taiwan --- Asia
North Korea --- Asia
Then I replace \t by one space (" ")
That is :
awk 'BEGIN { FS = " " } { print $1, "---", $4 }' cou.data
The output I got is not understandable to me
USSR --- Asia
Cananda --- North
China --- Asia
USA --- North
Brazil --- South
India --- Asia
Mexico --- North
France --- Europe
Japan --- Asia
Germany --- Europe
England --- Europe
Taiwan --- Asia
North --- 2134
Line 2,4,5,7,13 each have one space and the other lines have no space(s) at all.
As for lines that have no space, why $1, $4 still can be printed ?
As for line 2,4,5,7,13, I thought $1 should be printed like this:
Cananda 3852 25 North
USA 3615 237 North
Brazil 3286 134 South
Mexico 762 78 North
North
And $4 does not exist.
Where did I get wrong ?

So problem here is string/country names on 1st field which are having spaces in their names for example North Korea. So when you are setting FS as \t this string will be considered as a single field on the other hand when you will set FS as space this will be considered as 2 different fields. That is why you are seeing difference between field numbers after changing the FS values in your codes.
I would suggest your first attempt is good enough to get your expected values.

Related

Pandas - add row with inverted values based on condition

In a dataframe like this:
...
match team opponent venue
233 3b0345fb Brazil Argentina Home
234 3b2357fb Argentina Brazil Away
427 3b0947fb England Poland Home
...
how can I select one dataframe slice, based on a column value (df[df['team']=='England']), like this:
...
match team opponent venue
559 4a3eae2f England Poland Home
...
And add inverted rows of that slice to the original dataframe, changing 'Home' with 'Away', ending up with:
...
match team opponent venue
233 3b0345fb Brazil Argentina Home
234 3b2357fb Argentina Brazil Away
559 3b0947fb England Poland Home
560 3b0947fb Poland England Away
...
Note: This slice should contain n rows and produce n inverted rows.
You can use:
df2 = df[df['team'].eq('England')].copy()
df2[['team', 'opponent']] = df2[['opponent', 'team']]
df2['venue'] = df2['venue'].map({'Home': 'Away', 'Away': 'Home})
out = pd.concat([df, df2])
print(out)
Output:
match team opponent venue
233 3b0345fb Brazil Argentina Home
234 3b2357fb Argentina Brazil Away
427 3b0947fb England Poland Home
427 3b0947fb Poland England Away
If you want to invert all:
df2 = df.copy()
df2[['team', 'opponent']] = df2[['opponent', 'team']]
df2['venue'] = df2['venue'].map({'Home': 'Away', 'Away': 'Home})
out = pd.concat([df, df2])
output:
match team opponent venue
233 3b0345fb Brazil Argentina Home
234 3b2357fb Argentina Brazil Away
427 3b0947fb England Poland Home
233 3b0345fb Argentina Brazil Away
234 3b2357fb Brazil Argentina Home
427 3b0947fb Poland England Away

How to find IF single rows meet a criteria ELSE aggregate multiple rows within a group

I have some accounting data where I need to select a single row within a group if it meets a dollar amount criteria OR if it does not I need to sum/combine multiple rows in that group to see if that group meets the criteria. Example data:
Continent
Region
Sales Amount
South America
North
$300
South America
South
$100
South America
West
$500
South America
East
$200
North America
North
$100
North America
South
$50
North America
West
$50
North America
East
$400
Europe
North
$100
Europe
South
$200
Europe
West
$100
Europe
East
$100
Asia
North
$75
Asia
South
$100
Asia
West
$100
Asia
East
$100
Africa
North
$500
Africa
South
$700
Africa
West
$100
Africa
East
$100
In the above example, I want to find all continents that have single regions/rows with $500 in sales OR I want to find countries where 2 or more regions can be combined to meet the $500 amount. My expected result would be:
Continent
Region_1
Region_2
Sales Amount_1
Sales Amount_2
Canada
West
not applicable
$500
USA
North,East
not applicable
$500
Europe
North,South,West,East
not applicable
$500
Asia
does not meet criteria
not applicable
does not meet criteria
Africa
South
North
$700
$500
Region_2 is only applicable if more than one region within a continent meets the sales amount criteria of $500 on its own.

Can not use group by function

Data on Table:-
wkt Partners Team Opponent Runs Balls
1 S Hope & E Lewis WEST INDIES SOUTH AFRICA 43 66
2 S Hope & S Hetmyer WEST INDIES SOUTH AFRICA 70 79
3 D Bravo & S Hetmyer WEST INDIES SOUTH AFRICA 84 97
1 J Malan & Q Kock SOUTH AFRICA WEST INDIES 3 4
2 J Malan & F Plessis SOUTH AFRICA WEST INDIES 32 44
3 J Malan & R Dussen SOUTH AFRICA WEST INDIES 100 90
1 S Dhawan & R Sharma INDIA IRELAND 3 8
2 V Kohli & R Sharma INDIA IRELAND 102 70
I want to return the pair of partners, team they belong to, opponent they play against only once for each wkt where runs are highest for that particular wkt
For above table I'd like result as follow
wkt Partners Team Opponent Runs Balls
1 S Hope & E Lewis WEST INDIES SOUTH AFRICA 43 66
2 V Kohli & R Sharma INDIA IRELAND 102 70
3 J Malan & R Dussen SOUTH AFRICA WEST INDIES 100 90
Following is the code that I've used
SELECT wkt, Partners, Team, Opponent, max(Runs), Balls
FROM Partnerships
GROUP BY wkt
But I've been stuck with following error
Column 'Partnerships.Partners' is invalid in the select list because it is not contained in either an aggregate function or the GROUP BY clause.
How about row_number()?
select p.*
from (select p.*, row_number() over (partition by wkt order by runs desc) as seqnum
from Partnerships p
) p
where seqnum = 1;

How does Awk's pattern matching for strings works?

I am trying to understand how the range pattern matching work in Awk
Here is the full data that I am practicing with
Raw Data
-----------------------------------------
USSR 8649 275 Asia
Canada 3852 25 North America
China 3705 1032 Asia
USA 3615 237 North America
Brazil 3286 134 South America
India 1267 746 Asia
Mexico 762 78 North America
France 211 55 Europe
Japan 144 120 Asia
Germany 96 61 Europe
England 94 56 Europe
If I write this code
$ awk '/Asia/, /Europe/' countries.awk
I get
USSR 8649 275 Asia
Canada 3852 25 North America
China 3705 1032 Asia
USA 3615 237 North America
Brazil 3286 134 South America
India 1267 746 Asia
Mexico 762 78 North America
France 211 55 Europe
Japan 144 120 Asia
Germany 96 61 Europe
It doesn't output England.
And If I write this
$ awk '/Europe/, /Asia/' countries.awk
I get
France 211 55 Europe
Japan 144 120 Asia
Germany 96 61 Europe
England 94 56 Europe
What is the behavior here? Why do I not get England on the first one?
Awk process input lines one at a time, the syntax you used is likely to print lines from the start to the end pattern, represented by country names. When you used
awk '/Asia/, /Europe/'
The start of pattern Asia happens more than once. As you can see from the line numbers below, numbers 3,5,8 and 11 represent the start of the pattern and the pattern ends at lines 10 and 12. Observe carefully the sub-ranges of lines between 8-10 and 11-12. The last end pattern Europe for the last Asia ends at line 12, that is the reason you are not seeing England in the first case.
But when you used
awk '/Europe/, /Asia/'
The line containing the first start pattern Europe starts at line 10 and ends at 11 another two pattern start at 12 and 13 without an end pattern Asia, so it would obviously print all the lines until Asia appears. So you are seeing England in the second case.
$ cat -n file
1 Raw Data
2 -----------------------------------------
3 USSR 8649 275 Asia
4 Canada 3852 25 North America
5 China 3705 1032 Asia
6 USA 3615 237 North America
7 Brazil 3286 134 South America
8 India 1267 746 Asia
9 Mexico 762 78 North America
10 France 211 55 Europe
11 Japan 144 120 Asia
12 Germany 96 61 Europe
13 England 94 56 Europe
Never use range expressions as they make trivial tasks very slightly briefer but then need a complete rewrite or duplicate conditions when your requirements change. Always use a flag instead:
awk '/Asia/{f=1} f{print} /Europe/{f=0}' countries.awk
I bet if you started with that you wouldn't even have had to ask this question as the logic is clear and explicit.

Series with count larger than certain number

Given this code in iPython
df1=df["BillingContactCountry"].value_counts()
df1
I get
United States 4138
Germany 1963
United Kingdom 732
Switzerland 528
Australia 459
Canada 369
Japan 344
France 303
Netherlands 285
I want to get a series with count larger than 303, what should I do?
You needboolean indexing:
print (df1[df1 > 303])
United States 4138
Germany 1963
United Kingdom 732
Switzerland 528
Australia 459
Canada 369
Japan 344
Name: BillingContactCountry, dtype: int64