Count different actions within one hour in python - pandas

Question

I am starting to work with time series. I have one of a user doing bank transfers to different countries, however the most frequent country to where he/she is doing the transfers is X, but there are transfers also to the countries Y and Z. Let's say:
date id country
2020-01-01T00:00:00.000Z id_01 X
2020-01-01T00:20:00.000Z id_02 X
2020-01-01T00:25:00.000Z id_03 Y
2020-01-01T00:35:00.000Z id_04 X
2020-01-01T00:45:00.000Z id_05 Z
2020-01-01T01:00:00.000Z id_06 X
2020-01-01T10:20:00.000Z id_07 X
2020-01-01T10:25:00.000Z id_08 X
2020-01-01T13:00:00.000Z id_09 X
2020-01-01T18:45:00.000Z id_10 Z
2020-01-01T18:55:00.000Z id_11 X
Since the most frequent country is X, I would like to count iteratively how many transactions have been done within one hour (in the whole list of events) to countries different than X.
The format of the expected output for this particular case would be:
date id country
2020-01-01T00:25:00.000Z id_03 Y
2020-01-01T00:45:00.000Z id_05 Z
Starting from 2020-01-01T00:00:00.000Z, within one hour there are two Y, Z transactions. Then starting from 2020-01-01T00:20:00.000Z, within one hour, there are the same transactions, and so on. Then, starting from 2020-01-01T10:20:00.000Z, within one hour, all are X. Starting from 2020-01-01T18:45:00.000Z, within one hour, there is only one Z.
I am trying with a double for loop and .value_counts(), but I'm not sure of what I am doing.

Answer 1

Have you considered using a time-series database for this? It could make your life easier if you are doing a lot of event-based aggregations with arbitrary time intervals. Time-series databases abstract this for you so all you need is to send a query and get the results into pandas. It's also going to run considerably faster.
For example hourly aggregations can be done using the following syntax in QuestDB.
select timestamp, country, count() from yourTable SAMPLE BY 1h
this will return results like this
| timestamp | country | count |
| 2020-06-22T00:00:00 | X | 234 |
| 2020-06-22T00:00:00 | Y | 493 |
| 2020-06-22T01:00:00 | X | 12 |
| 2020-06-22T01:00:00 | Y | 66 |
You can adjust this to monthly or weekly or 5-minute resolution results without having to re-write your logic, all you need to do is change the 1h to 1M,7d or 5m or pass this as an argument.
Now, to get results one hour before and after the timestamp of your target transaction, you could add a timestamp interval search to the above. For example assuming your target transaction happened on 2010-01-01T06:47:00.000000Z, the resulting search would be
select hour, country, count() from yourTable
where timestamp = '2010-01-01T05:47:00.000000Z;2h'
sample by 1h;
If this is something which would work for you, there is a tutorial on how to run this type of query in QuestDB and get the results into pandas here

Answer 2

IIUC, you can select only the rows not X, then use diff once forward and once backward (within 1 hour before and after) and you want where any of the two diff is below a Timedelta of 1h.
#convert to datetime
df['date'] = pd.to_datetime(df['date'])
#mask not X and select only these rows
mX = df['country'].ne('X')
df_ = df[mX].copy()
# mask within an hour before and after
m1H = (df_['date'].diff().le(pd.Timedelta(hours=1)) |
df_['date'].diff(-1).le(pd.Timedelta(hours=1)) )
# selet only the rows meeting criteria on X and 1H
df_ = df_[m1H]
print (df_)
date id country
2 2020-01-01 00:25:00+00:00 id_03 Y
4 2020-01-01 00:45:00+00:00 id_05 Z

Answer 3

You can try :
df['date'] = pd.to_datetime(df.date)
(df.country != 'X').groupby(by=df.date.dt.hour).sum()
First it turns your date columns into a datetime. Then, you test if country is 'X', and group by hour, and sum the number of countries that are different than 'X'. Groups are based on hours, and not on rolling elasped time. Hope it solves your problem!