Shifting a Pandas column, and then take the mean of the next 3 values (post_shift) - pandas

Question

I have a dataframe, df which looks like this
Open High Low Close Volume
Date
2007-03-22 2.65 2.95 2.64 2.86 176389
2007-03-23 2.87 2.87 2.78 2.78 63316
2007-03-26 2.83 2.83 2.51 2.52 54051
2007-03-27 2.61 3.29 2.60 3.28 589443
2007-03-28 3.65 4.10 3.60 3.80 1114659
2007-03-29 3.91 3.91 3.33 3.57 360501
2007-03-30 3.70 3.88 3.66 3.71 185787
I'm attempting to create a new column, that first shifts the Open column 3 rows (df.Open.shift(-3)) and then takes the average of itself and the next 2 values.
So for example the above dataframe's Open column would be shifted -3 rows and look something like this:
Date
2007-03-22 2.610
2007-03-23 3.650
2007-03-26 3.910
2007-03-27 3.700
2007-03-28 3.710
2007-03-29 3.710
2007-03-30 3.500
I then want to take the forward mean of the next 3 values(including itself) via iteration.
So the first iteration would 2.610 (first value) + 3.650 + 3.910(which are the next values) divided by 3.
Then we take the next value 3.650 (first value) and do the same. Creating a column of values.
At first I tried something like :
df['Avg'] =df.Open.shift(-3).iloc[0:3].mean()
But this doesn't iterate through all the values of Open.shift
This next loop seems to work but is very slow, and I was told it's bad practice to use for loops in Pandas.
for i in range(0, len(df.Open)):
df['Avg'][i] =df.Open.shift(-3).iloc[i:i+4].mean()
I tried to thinking of ways to use apply
df.Open.shift(-3).apply(loc[0:4].mean())
df.Open.shift(-3).apply(lambda x: x[0:4].mean())
but these seems to give errors such as
TypeError: 'float' object is not subscriptable etc
I can't think of an elegant way of doings this.
Thank you.

Answer 1

You can use pandas rolling_mean. Since it uses backward window, it will give you the first two rows as 2.61 (value itself) and 3.13(mean of row 0 and row 1). To handle that, you can use shift(-2) to shift the values by 2 rows.
pd.rolling_mean(df, window=3, min_periods=1).shift(-2)
output:
open
date
2007-03-22 3.390000
2007-03-23 3.753333
2007-03-26 3.773333
2007-03-27 3.706667
2007-03-28 3.640000
2007-03-29 NaN
2007-03-30 NaN

Answer 2

numpy solution
As promised
NOTE: HUGE CAVEAT
This is an advanced technique and is not recommended for any beginner!!!
Using this might actually shave your poodle bald by accident. BE CAREFUL!
as_strided
from numpy.lib.stride_tricks import as_strided
import numpy as np
import pandas as pd
# I didn't have your full data for all dates
# so I created my own array
# You should be able to just do
# o = df.Open.values
o = np.array([2.65, 2.87, 2.83, 2.61, 3.65, 3.91, 3.70, 3.71, 3.71, 3.50])
# because we shift 3 rows, I trim with 3:
# because it'll be rolling 3 period mean
# add two np.nan at the end
# this makes the strides cleaner.. sortof
# whatever, I wanted to do it
o = np.append(o[3:], [np.nan] * 2)
# strides are the size of the chunk of memory
# allocated to each array element. there will
# be a stride for each numpy dimension. for
# a one dimensional array, I only want the first
s = o.strides[0]
# it gets fun right here
as_strided(o, (len(o) - 2, 3), (s, s))
# ^ \___________/ \__/
# | \ \______
# object to stride --- size of array --- \
# to make memory chunk
# to step forward
# per dimension
[[ 2.61 3.65 3.91]
[ 3.65 3.91 3.7 ]
[ 3.91 3.7 3.71]
[ 3.7 3.71 3.71]
[ 3.71 3.71 3.5 ]
[ 3.71 3.5 nan]
[ 3.5 nan nan]]
Now we just take the mean. All together
o = np.array([2.65, 2.87, 2.83, 2.61, 3.65, 3.91, 3.70, 3.71, 3.71, 3.50])
o = np.append(o[3:], [np.nan] * 2)
s = o.strides[0]
as_strided(o, (len(o) - 2, 3), (s, s)).mean(1)
array([ 3.39 , 3.75333333, 3.77333333, 3.70666667, 3.64 ,
nan, nan])
You can wrap it in a pandas series
pd.Series(
as_strided(o, (len(o) - 2, 3), (s, s)).mean(1),
df.index[3:],
)