How to groupby, iterate over rows and count in pandas? - pandas

Question

I have a dataframe with columns city, date, source and count of each source.
I need to group by city, then iterate over rows of date column with the following condition: check each row and if the difference between them is <30 days then find start_date(min) and end_date(max) for the group of dates that meet condition, and add them as a separate columns. Also, count the source type and add again as separate columns.
city date source count
0 Alexandria 2022-10-13 black 117
1 Alexandria 2022-10-14 black 85
2 Alexandria 2022-10-15 black 63
3 Alexandria 2022-10-16 black 190
4 Alexandria 2022-10-17 black 389
5 Alexandria 2022-10-18 black 284
6 Alexandria 2022-10-19 black 179
7 Amsterdam 2018-08-05 red 1
8 Amsterdam 2018-08-28 red 111
9 Amsterdam 2019-08-17 red 1669
10 Amsterdam 2019-08-18 red 1584
11 Amsterdam 2019-08-19 red 940
12 Amsterdam 2019-08-21 red 1498
13 Amsterdam 2019-08-22 red 2281
14 Amsterdam 2019-08-23 red 2038
15 Amsterdam 2019-08-24 red 1516
16 Amsterdam 2019-08-25 red 1952
17 Amsterdam 2019-08-26 red 1434
18 Amsterdam 2019-08-27 red 881
19 Amsterdam 2019-08-29 red 1482
20 Amsterdam 2019-08-30 red 978
21 Amsterdam 2019-08-31 red 1423
22 Amsterdam 2019-09-01 red 1120
23 Amsterdam 2019-09-02 red 1117
24 Amsterdam 2019-09-06 red 1
To reproduce my dataframe:
# initialize list of lists
data1 = [['Alexandria', '2022-10-13', 'black', 117],
['Alexandria', '2022-10-14', 'black', 85],
['Alexandria', '2022-10-15', 'black', 63],
['Alexandria', '2022-10-16', 'black', 190],
['Alexandria', '2022-10-17', 'black', 389],
['Alexandria', '2022-10-18', 'black', 284],
['Alexandria', '2022-10-19', 'black', 179],
['Amsterdam', '2018-08-05', 'red', 1],
['Amsterdam', '2018-08-28', 'red', 111],
['Amsterdam', '2019-08-17', 'red', 1669],
['Amsterdam', '2019-08-18', 'red', 1584],
['Amsterdam', '2019-08-19', 'red', 940],
['Amsterdam', '2019-08-21', 'red', 1498],
['Amsterdam', '2019-08-22', 'red', 2281],
['Amsterdam', '2019-08-23', 'red', 2038],
['Amsterdam', '2019-08-24', 'red',1516],
['Amsterdam', '2019-08-25', 'red', 1952],
['Amsterdam', '2019-08-26', 'red', 1434],
['Amsterdam', '2019-08-27', 'red', 881],
['Amsterdam', '2019-08-29', 'red', 1482],
['Amsterdam', '2019-08-30', 'red', 978],
['Amsterdam', '2019-08-31', 'red', 1423],
['Amsterdam', '2019-09-01', 'red', 1120],
['Amsterdam', '2019-09-02', 'red',1117],
['Amsterdam', '2019-09-06', 'red', 1],
]
# Create the pandas DataFrame
df_b = pd.DataFrame(data1, columns=['city', 'date', 'source','count'])
The output needed(looks like pivot table):
So, for Alexandria i have only 1 round where dates are less than 30 days between them: from 2022-10-13 till 2022-10-19 (6th row). These 2 dates are min and max.
For Amsterdam we have 2 rounds of dates that have less that 30 days between them: 1st round are rows 7-8, 2nd round are rows 9-24.
Between 8 and 9 rows for Amsterdam there are more than 30 days, meaning it starts a new round.
city start_date end_date black red
0 Alexandria 2022-10-13 2022-10-19 1307 0
1 Amsterdam 2018-08-05 2018-08-28 0 112
2 Amsterdam 2019-08-17 2019-09-06 0 21914
To reproduce the output:
# initialize list of lists
data = [['Alexandria', '2022-10-13', '2022-10-19', 1307, 0],
['Amsterdam', '2018-08-05', '2018-08-28', 0, 112],
['Amsterdam', '2019-08-17', '2019-09-06', 0,21914 ]]
# Create the pandas DataFrame
df_a = pd.DataFrame(data, columns=['city', 'start_date', 'end_date','black', 'red'])

Answer 1

I know this does not fully answer the question, but is hopefully helpful in getting you partly there, since you did not show what you have tried (if anything). The 30 days part presumably would involve some custom way of grouping and aggregating the data which I have not attempted here. As such, this answer shows how to:
Create start_date and end_date columns for minimum and maximum dates grouped by city, respectively
Sum the count for each source value into new columns for each identified source value (black and red)
Merge the two resultant dataframes into a single one
This answer assumes the original data is in a dateframe named df.
Minimum and maximum dates
I know this is not exactly what is wanted due to the 30 days logic, but shows a straightforward grouping by city with minimum and maximum dates for each.
date_agg_df = df.groupby("city").agg({"date": ["min", "max"]}).reset_index()
date_agg_df.columns = ["city", "start_date", "end_date"]
Data in date_agg_df:
city start_date end_date
0 Alexandria 2022-10-13 2022-10-19
1 Amsterdam 2018-08-05 2019-09-06
Sum the count of each source value
black_red_df = df.fillna(0).pivot_table(index="city", columns="source", values="count", aggfunc="sum").fillna(0).reset_index()
black_red_df.columns = ["city", "black", "red"]
Data in black_red_df:
city black red
0 Alexandria 1307.0 0.0
1 Amsterdam 0.0 22026.0
Merge the dataframes
mrg_df = pd.merge(date_agg_df, black_red_df, on="city")
Data in mrg_df:
city start_date end_date black red
0 Alexandria 2022-10-13 2022-10-19 1307.0 0.0
1 Amsterdam 2018-08-05 2019-09-06 0.0 22026.0