I have a Pandas DataFrame:
Out[57]:
lastrun rate
0 2013-11-04 12:15:02 0
1 2013-11-04 13:14:50 4
2 2013-11-04 14:14:48 10
3 2013-11-04 16:14:59 16
I would like to convert that into an hourly time series and interpolate missing values (15:00) so that I end up with:
2013-11-04 12:00:00 0
2013-11-04 13:00:00 4
2013-11-04 14:00:00 10
2013-11-04 15:00:00 13
2013-11-04 16:00:00 16
How do I convert / map the dataframe data to a time series in Pandas?
Assuming your 'lastrun' has datetime objects:
In [22]: s = df.set_index('lastrun').resample('H')['rate']
In [23]: s
Out[23]:
lastrun
2013-11-04 12:00:00 0
2013-11-04 13:00:00 4
2013-11-04 14:00:00 10
2013-11-04 15:00:00 NaN
2013-11-04 16:00:00 16
Freq: H, dtype: float64
In [24]: s.interpolate()
Out[24]:
lastrun
2013-11-04 12:00:00 0
2013-11-04 13:00:00 4
2013-11-04 14:00:00 10
2013-11-04 15:00:00 13
2013-11-04 16:00:00 16
Freq: H, dtype: int64
That's if you want linear interpolation. There's a bunch more options in the upcoming .13 release!
Related
I have a dataframe like this:
df11 = pd.DataFrame(
{
"Start_date": ["2018-01-31 12:00:00", "2018-02-28 16:00:00", "2018-02-27 22:00:00"],
"End_date": ["2019-01-31 21:45:00", "2019-03-24 22:00:00", "2018-02-28 01:00:00"],
}
)
Start_date End_date
0 2018-01-31 12:00:00 2019-01-31 21:45:00
1 2018-02-28 16:00:00 2019-03-24 22:00:00
2 2018-02-27 22:00:00 2018-02-28 01:00:00
I need to check the overlap time duration in specific periods in seconds. My expected results are like this:
Start_date End_date 12h-16h 16h-22h 22h-00h 00h-02h30
0 2018-01-31 12:00:00 2019-01-31 21:45:00 14400 20700 0 0
1 2018-02-28 16:00:00 2019-03-24 22:00:00 0 21600 0 0
2 2018-02-27 22:00:00 2018-02-28 01:00:00 0 0 7200 3600
I know it`s completely wrong and I´ve tried other solutions. This is one of my attempts:
df11['12h-16h']=np.where(df11['Start_date']<timedelta(hours=16, minutes=0, seconds=0) & df11['End_date']>timedelta(hours=12, minutes=0, seconds=0),(np.minimum(df11['End_date'],timedelta(hours=16, minutes=0, seconds=0)))-(np.maximum(df11['Start_date'],timedelta(hours=12, minutes=0, seconds=0)))
I only want to replace 0 which lies between exactly two numbers with its average value.
My dataset looks like below:
time value
9:45:00 0
10:00:00 0
10:15:00 0
10:30:00 10
10:45:00 0
11:00:00 10
11:15:00 10
11:30:00 0
11:45:00 10
12:00:00 0
12:15:00 0
12:30:00 0
12:45:00 10
13:00:00 0
13:15:00 0
I want it to look like this:
time value
9:45:00 0
10:00:00 0
10:15:00 0
10:30:00 10
10:45:00 10
11:00:00 10
11:15:00 10
11:30:00 10
11:45:00 10
12:00:00 0
12:15:00 0
12:30:00 0
12:45:00 10
13:00:00 0
13:15:00 0
in this, since the 0 between 11:45 to 12:45 is not exactly between two numbers (ie multiple zeros), we are not filling in these values
How about this?
from io import StringIO as sio
data = sio("""
time value
9:45:00 0
10:00:00 0
10:15:00 0
10:30:00 10
10:45:00 0
11:00:00 10
11:15:00 10
11:30:00 0
11:45:00 10
12:00:00 0
12:15:00 0
12:30:00 0
12:45:00 10
13:00:00 0
13:15:00 0
""")
import pandas as pd
df = pd.read_csv(data, sep='\s+')
df['flag_to_fill'] = (df['value']==0) & (df['value'].shift(1)!=0) & (df['value'].shift(-1)!=0)
df.loc[df['flag_to_fill'], 'value'] = 0.5*(df['value'].shift(1) + df['value'].shift(-1))
df
I have a DataFrame (df) as follow where 'date' is a datetime index (Y-M-D):
df :
values
date
2010-01-01 10
2010-01-02 20
2010-01-03 - 30
I want to create a new df with interpolated datetime index as follow:
values
date
2010-01-01 12:00:00 10
2010-01-01 17:00:00 15 # mean value betw. 2010-01-01 and 2010-01-02
2010-01-02 12:00:00 20
2010-01-02 17:00:00 - 5 # mean value betw. 2010-01-02 and 2010-01-03
2010-01-03 12:00:00 -30
Can anyone help me on this?
I believe need add 12 hours to index first, then reindex by union new indices with 17 and last interpolate:
df1 = df.set_index(df.index + pd.Timedelta(12, unit='h'))
idx = (df.index + pd.Timedelta(17, unit='h')).union(df1.index)
df2 = df1.reindex(idx).interpolate()
print (df2)
values
date
2010-01-01 12:00:00 10.0
2010-01-01 17:00:00 15.0
2010-01-02 12:00:00 20.0
2010-01-02 17:00:00 -5.0
2010-01-03 12:00:00 -30.0
2010-01-03 17:00:00 -30.0
I have a column in DataFrame a Start_Time (datatype: Object) with values as :-
6:00:00
7:00:00
8:01:00
and want to change it to
6:30:00
7:30:00
8:30:00
Maybe you can use to_datetime and time:
print df
a
0 6:00:00
1 7:00:00
2 8:01:00
b = pd.to_datetime(df['a']) + pd.Timedelta(minutes=30)
print b
0 2016-02-17 06:30:00
1 2016-02-17 07:30:00
2 2016-02-17 08:31:00
Name: a, dtype: datetime64[ns]
print b.dt.time
0 06:30:00
1 07:30:00
2 08:31:00
Name: a, dtype: object
print b.dt.time.values
[datetime.time(6, 30) datetime.time(7, 30) datetime.time(8, 31)]
But if you need reset minutes:
print df
a
0 6:00:00
1 7:00:00
2 8:01:00
df['a'] = pd.to_datetime(df['a'])
print df
a
0 2016-02-17 06:00:00
1 2016-02-17 07:00:00
2 2016-02-17 08:01:00
df['a'] = df['a'].map(lambda x: datetime.datetime(x.year, x.month, x.day, x.hour))
print df
a
0 2016-02-17 06:00:00
1 2016-02-17 07:00:00
2 2016-02-17 08:00:00
b = df['a'] + pd.Timedelta(minutes=30)
print b.dt.time
0 06:30:00
1 07:30:00
2 08:30:00
Name: a, dtype: object
print b.dt.time.values
[datetime.time(6, 30) datetime.time(7, 30) datetime.time(8, 30)]
I'm using pandas 0.12.0. I have a DataFrame that looks like:
date ms
0 2013-06-03 00:10:00 75.846318
1 2013-06-03 00:20:00 78.408277
2 2013-06-03 00:30:00 75.807990
3 2013-06-03 00:40:00 70.509438
4 2013-06-03 00:50:00 71.537499
I want to generate a third column, "tod", which contains just the time portion of the date (i.e. call .time() on each value). I'm somewhat of a pandas newbie, so I suspect this is trivial but I'm just not seeing how to do it.
Just apply the Timestamp time method to items in the date column:
In [11]: df['date'].apply(lambda x: x.time())
# equivalently .apply(pd.Timestamp.time)
Out[11]:
0 00:10:00
1 00:20:00
2 00:30:00
3 00:40:00
4 00:50:00
Name: date, dtype: object
In [12]: df['tod'] = df['date'].apply(lambda x: x.time())
This gives a column of datetime.time objects.
Using the method Andy created on Index is faster than apply
In [93]: df = DataFrame(randn(5,1),columns=['A'])
In [94]: df['date'] = date_range('20130101 9:05',periods=5)
In [95]: df['time'] = Index(df['date']).time
In [96]: df
Out[96]:
A date time
0 0.053570 2013-01-01 09:05:00 09:05:00
1 -0.382155 2013-01-02 09:05:00 09:05:00
2 0.357984 2013-01-03 09:05:00 09:05:00
3 -0.718300 2013-01-04 09:05:00 09:05:00
4 0.531953 2013-01-05 09:05:00 09:05:00
In [97]: df.dtypes
Out[97]:
A float64
date datetime64[ns]
time object
dtype: object
In [98]: df['time'][0]
Out[98]: datetime.time(9, 5)