Pandas aggregating across time windows with user logs - pandas

Question

I am trying to take a dataframe of logs and aggregate counts across time windows, specifically before a Purchase. The goal is to create features that can be used to predict a future purchase.
Here is my original df
user_id activity_date activity_type
0 2013-07-11 EmailOpen
0 2013-07-11 FormSubmit
0 2013-07-15 EmailOpen
0 2013-07-17 Purchase
0 2013-07-18 EmailOpen
and I would like my result to look like:
user_id EmailOpen_count FormSubmit_count Days_since_start Purchase
0 2 1 6 1
0 1 0 1 0
The above idea is I have aggregated before the purchase, and since that user had only one purchase, the next row will aggregate everything after the last purchase.
I tried to extract the Purchase dates first and then do an iterative approach but ran it all night with no success. Here's how I was going to extract the dates, but even this took way too long and I am sure that building the new dataframe would have taken millennia:
purchase_dict = {}
for user in list_of_users:
# Stores list of days when purchase was made for each user.
days_bought = list(df[df['user_id'] == user][df['activity_type'] == 'Purchase']['activity_date'])
purchase_dict[user] = days_bought
I'm wondering if there is a semi-efficient way with groupbys, agg, time_between, etc. Thanks!

Answer 1

Perhaps a bit clunky, and needing some column renaming at the end, but this appears to work for me (with new testing data):
user_id activity_date activity_type
0 2013-07-11 EmailOpen
0 2013-07-11 FormSubmit
0 2013-07-15 EmailOpen
0 2013-07-17 Purchase
0 2013-07-18 EmailOpen
1 2013-07-12 Purchase
1 2013-07-12 FormSubmit
1 2013-07-15 EmailOpen
1 2013-07-18 Purchase
1 2013-07-18 EmailOpen
2 2013-07-09 EmailOpen
2 2013-07-10 Purchase
2 2013-07-15 EmailOpen
2 2013-07-22 Purchase
2 2013-07-23 EmailOpen
# Convert to datetime
df['activity_date'] = pd.to_datetime(df['activity_date'])
# Create shifted flag to identify purchase
df['x'] = (df['activity_type'] == 'Purchase').astype(int).shift().fillna(method='bfill')
# Calculate time window as cumsum of this shifted flag
df['time_window'] = df.groupby('user_id')['x'].cumsum()
# Pivot to count activities by user ID and time window
df2 = df.pivot_table(values='activity_date', index=['user_id', 'time_window'],
columns='activity_type', aggfunc=len, fill_value=0)
# Create separate table of days elapsed by user ID & time window
time_elapsed = ( df.groupby(['user_id', 'time_window'])['activity_date'].max()
- df.groupby(['user_id', 'time_window'])['activity_date'].min() )
# Merge dataframes
df3 = df2.join(time_elapsed)
yields
EmailOpen FormSubmit Purchase activity_date
user_id time_window
0 0.0 2 1 1 6 days
1.0 1 0 0 0 days
1 0.0 0 0 1 0 days
1.0 1 1 1 6 days
2.0 1 0 0 0 days
2 0.0 1 0 1 1 days
1.0 1 0 1 7 days
2.0 1 0 0 0 days
Edit per comments:
To add in time elapsed by type of activity:
time_since_activity = ( df.groupby(['user_id', 'time_window'])['activity_date'].max()
- df.groupby(['user_id', 'time_window', 'activity_type'])['activity_date'].max() )
df4 = df3.join(time_since_activity.unstack('activity_type'), rsuffix='_time')
yielding
EmailOpen FormSubmit ... FormSubmittime Purchasetime
user_id time_window ...
0 0.0 2 1 ... 6 days 0 days
1.0 1 0 ... NaT NaT
1 0.0 0 0 ... NaT 0 days
1.0 1 1 ... 6 days 0 days
2.0 1 0 ... NaT NaT
2 0.0 1 0 ... NaT 0 days
1.0 1 0 ... NaT 0 days
2.0 1 0 ... NaT NaT