How to Assign Data after Groupby and Concat in pandas - pandas

Question

Iam newbie in python. I have huge a dataframe with millions of rows and id. my data looks like this:
Time ID X Y
8:00 A 23 100
9:00 B 24 110
10:00 B 25 120
11:00 C 26 130
12:00 C 27 140
13:00 A 28 150
14:00 A 29 160
15:00 D 30 170
16:00 C 31 180
17:00 B 32 190
18:00 A 33 200
19:00 C 34 210
20:00 A 35 220
21:00 B 36 230
22:00 C 37 240
23:00 B 38 250
I sorted the data on id and time.
Time ID X Y
8:00 A 23 100
13:00 A 28 150
14:00 A 29 160
18:00 A 33 200
20:00 A 35 220
9:00 B 24 110
10:00 B 25 120
17:00 B 32 190
21:00 B 36 230
23:00 B 38 250
11:00 C 26 130
12:00 C 27 140
16:00 C 31 180
19:00 C 34 210
22:00 C 37 240
15:00 D 30 170
and I want to pick only "The first and the last" of the id and eliminate the rest. The result looked like this:
Time ID X Y
8:00 A 23 100
20:00 A 35 220
9:00 B 24 110
23:00 B 38 250
11:00 C 26 130
22:00 C 37 240
15:00 D 30 170
I used this code:
df = pd.read_csv("data.csv")
g = df.groupby('ID')
g_1 = pd.concat([g.head(1),g.tail(1)]).drop_duplicates().sort_values('ID').reset_index(drop=True)
g_1.to_csv('result.csv')
but I want to assign or give annotation every row as "first" and "last" in the new column.
my expected result looks like this:
Time ID X Y Annotation
8:00 A 23 100 First
20:00 A 35 220 Last
9:00 B 24 110 First
23:00 B 38 250 Last
11:00 C 26 130 First
22:00 C 37 240 Last
15:00 D 30 170
anyone could help me with this? please give me advice thank you.

Answer 1

You can use groupby agg,first and last. Works perfectly for the column Annotation. As a bonus, this will work on the original dataframe so no need to sort
df3.groupby('ID').agg(['first', 'last']).stack().reset_index().rename(columns = {'level_1': 'Annotation'})
ID Annotation Time X Y
0 A first 8:00 23 100
1 A last 20:00 35 220
2 B first 9:00 24 110
3 B last 23:00 38 250
4 C first 11:00 26 130
5 C last 22:00 37 240
6 D first 15:00 30 170
7 D last 15:00 30 170

Answer 2

No need groupby using drop_duplicatesafter you sort
df=pd.concat([df.drop_duplicates(['ID']).assign(sign='first'),df.drop_duplicates(['ID'],keep='last').assign(sign='last')]).sort_values('ID')
df
Time ID X Y sign
0 8:00 A 23 100 first
4 20:00 A 35 220 last
5 9:00 B 24 110 first
9 23:00 B 38 250 last
10 11:00 C 26 130 first
14 22:00 C 37 240 last
15 15:00 D 30 170 first
15 15:00 D 30 170 last

Answer 3

Try:
df.groupby('ID').agg(['first','last'])\
.stack(1).reset_index()\
.rename(columns={'level_1':'Annotation'})
Output:
ID Annotation Time X Y
0 A first 8:00 23 100
1 A last 20:00 35 220
2 B first 9:00 24 110
3 B last 23:00 38 250
4 C first 11:00 26 130
5 C last 22:00 37 240
6 D first 15:00 30 170
7 D last 15:00 30 170