I have a dataframe in pandas called 'munged_data' with two columns 'entry_date' and 'dob' which i have converted to Timestamps using pd.to_timestamp.I am trying to figure out how to calculate ages of people based on the time difference between 'entry_date' and 'dob' and to do this i need to get the difference in days between the two columns ( so that i can then do somehting like round(days/365.25). I do not seem to be able to find a way to do this using a vectorized operation. When I do munged_data.entry_date-munged_data.dob i get the following :
internal_quote_id
2 15685977 days, 23:54:30.457856
3 11651985 days, 23:49:15.359744
4 9491988 days, 23:39:55.621376
7 11907004 days, 0:10:30.196224
9 15282164 days, 23:30:30.196224
15 15282227 days, 23:50:40.261632
However i do not seem to be able to extract the days as an integer so that i can continue with my calculation.
Any help appreciated.
Using the Pandas type Timedelta available since v0.15.0 you also can do:
In[1]: import pandas as pd
In[2]: df = pd.DataFrame([ pd.Timestamp('20150111'),
pd.Timestamp('20150301') ], columns=['date'])
In[3]: df['today'] = pd.Timestamp('20150315')
In[4]: df
Out[4]:
date today
0 2015-01-11 2015-03-15
1 2015-03-01 2015-03-15
In[5]: (df['today'] - df['date']).dt.days
Out[5]:
0 63
1 14
dtype: int64
You need 0.11 for this (0.11rc1 is out, final prob next week)
In [9]: df = DataFrame([ Timestamp('20010101'), Timestamp('20040601') ])
In [10]: df
Out[10]:
0
0 2001-01-01 00:00:00
1 2004-06-01 00:00:00
In [11]: df = DataFrame([ Timestamp('20010101'),
Timestamp('20040601') ],columns=['age'])
In [12]: df
Out[12]:
age
0 2001-01-01 00:00:00
1 2004-06-01 00:00:00
In [13]: df['today'] = Timestamp('20130419')
In [14]: df['diff'] = df['today']-df['age']
In [16]: df['years'] = df['diff'].apply(lambda x: float(x.item().days)/365)
In [17]: df
Out[17]:
age today diff years
0 2001-01-01 00:00:00 2013-04-19 00:00:00 4491 days, 00:00:00 12.304110
1 2004-06-01 00:00:00 2013-04-19 00:00:00 3244 days, 00:00:00 8.887671
You need this odd apply at the end because not yet full support for timedelta64[ns] scalars (e.g. like how we use Timestamps now for datetime64[ns], coming in 0.12)
Not sure if you still need it, but in Pandas 0.14 i usually use .astype('timedelta64[X]') method
http://pandas.pydata.org/pandas-docs/stable/timeseries.html (frequency conversion)
df = pd.DataFrame([ pd.Timestamp('20010101'), pd.Timestamp('20040605') ])
df.ix[0]-df.ix[1]
Returns:
0 -1251 days
dtype: timedelta64[ns]
(df.ix[0]-df.ix[1]).astype('timedelta64[Y]')
Returns:
0 -4
dtype: float64
Hope that will help
Let's specify that you have a pandas series named time_difference which has type
numpy.timedelta64[ns]
One way of extracting just the day (or whatever desired attribute) is the following:
just_day = time_difference.apply(lambda x: pd.tslib.Timedelta(x).days)
This function is used because the numpy.timedelta64 object does not have a 'days' attribute.
To convert any type of data into days just use pd.Timedelta().days:
pd.Timedelta(1985, unit='Y').days
84494
Related
I have a dataframe, df, with datetimeindex and a single column, like this:
I need to count how many non-zero entries i have at each month. For example, according to those images, in January i would have 2 entries, in February 1 entry and in March 2 entries. I have more months in the dataframe, but i guess that explains the problem.
I tried using pandas groupby:
df.groupby(df.index.month).count()
But that just gives me total days at each month and i don't saw any other parameter in count() that i could use here.
Any ideas?
Try index.to_period()
For example:
In [1]: import pandas as pd
import numpy as np
x_df = pd.DataFrame(
{
'values': np.random.randint(low=0, high=2, size=(120,))
} ,
index = pd.date_range("2022-01-01", periods=120, freq="D")
)
In [2]: x_df
Out[2]:
values
2022-01-01 0
2022-01-02 0
2022-01-03 1
2022-01-04 0
2022-01-05 0
...
2022-04-26 1
2022-04-27 0
2022-04-28 0
2022-04-29 1
2022-04-30 1
[120 rows x 1 columns]
In [3]: x_df[x_df['values'] != 0].groupby(lambda x: x.to_period("M")).count()
Out[3]:
values
2022-01 17
2022-02 15
2022-03 16
2022-04 17
can you try this:
#drop nans
import numpy as np
dfx['col1']=dfx['col1'].replace(0,np.nan)
dfx=dfx.dropna()
dfx=dfx.resample('1M').count()
My Pandas df:
import pandas as pd
import io
data = """date value
"2015-09-01" 71.925000
"2015-09-06" 71.625000
"2015-09-11" 71.333333
"2015-09-12" 64.571429
"2015-09-21" 72.285714
"""
df = pd.read_table(io.StringIO(data), delim_whitespace=True)
df.date = pd.to_datetime(df.date)
I Given a user input date ( 01-09-2015).
I would like to keep only those date where difference between date and input date is multiple of 5.
Expected output:
input = 01-09-2015
df:
date value
0 2015-09-01 71.925000
1 2015-09-06 71.625000
2 2015-09-11 71.333333
3 2015-09-21 72.285714
My Approach so far:
I am taking the delta between input_date and date in pandas and saving this delta in separate column.
If delta%5 == 0, keep the row else drop. Is this the best that can be done?
Use boolean indexing for filter by mask, here convert input values to datetimes and then timedeltas to days by Series.dt.days:
input1 = '01-09-2015'
df = df[df.date.sub(pd.to_datetime(input1)).dt.days % 5 == 0]
print (df)
date value
0 2015-09-01 71.925000
1 2015-09-06 71.625000
2 2015-09-11 71.333333
4 2015-09-21 72.285714
I have a column of timedelta in pandas. It is in the format x days 00:00:00. I want to filter out and flag the rows which have a value >=30 minutes. I have no clue how to do that using pandas. I tried booleans and if statements but it didn't work. Any help would be appreciated.
You can convert timedeltas to seconds by total_seconds and compare with scalar:
df = df[df['col'].dt.total_seconds() < 30]
Or compare with Timedelta:
df = df[df['col'] < pd.Timedelta(30, unit='s')]
Sample:
df = pd.DataFrame({'col':pd.to_timedelta(['25:10:01','00:01:20','00:00:20'])})
print (df)
col
0 1 days 01:10:01
1 0 days 00:01:20
2 0 days 00:00:20
df = df[df['col'].dt.total_seconds() < 30]
print (df)
col
2 00:00:20
I'm trying to accomplish the following...
I got a Pandas dataframe that have a number of entries, indexed with DatetimeIndex which looks a bit like this:
bro_df.info()
<class 'bat.log_to_dataframe.LogToDataFrame'>
DatetimeIndex: 3596641 entries, 2017-12-14 13:52:01.633070 to 2018-01-03 09:59:53.108566
Data columns (total 20 columns):
conn_state object
duration timedelta64[ns]
history object
id.orig_h object
id.orig_p int64
id.resp_h object
id.resp_p int64
local_orig bool
local_resp bool
missed_bytes int64
orig_bytes int64
orig_ip_bytes int64
orig_pkts int64
proto object
resp_bytes int64
resp_ip_bytes int64
resp_pkts int64
service object
tunnel_parents object
uid object
dtypes: bool(2), int64(9), object(8), timedelta64[ns](1)
memory usage: 528.2+ MB
What I'm interested in is getting a slice of this data that takes the last entry, 2018-01-03 09:59:53.108566' in this case, and then subtracts an hour from that. This should give me the last hours worth of entries.
What I've tried to do so far is the following:
last_entry = bro_df.index[-1:]
first_entry = last_entry - pd.Timedelta('1 hour')
Which gives me what to me looks like fairly correct values, as per:
print(first_entry)
print(last_entry)
DatetimeIndex(['2018-01-03 08:59:53.108566'], dtype='datetime64[ns]', name='ts', freq=None)
DatetimeIndex(['2018-01-03 09:59:53.108566'], dtype='datetime64[ns]', name='ts', freq=None)
This is also sadly where I get stuck. I've tried various things with bro_df.loc and bro_df.iloc and so on but all I get is different errors for datatypes and not in index etc. Which leads me to think that I possibly might need to convert the first_entry, last_entry variables to another type?
Or I might as usual be barking up entirely the wrong tree.
Any assistance or guidance would be most appreciated.
Cheers, Mike
It seems you need create scalars by indexing [0] and select by loc:
df = bro_df.loc[first_entry[0]: last_entry[0]]
Or select by exact indexing:
df = bro_df[first_entry[0]: last_entry[0]]
Sample:
rng = pd.date_range('2017-04-03', periods=10, freq='2H 24T')
bro_df = pd.DataFrame({'a': range(10)}, index=rng)
print (bro_df)
a
2017-04-03 00:00:00 0
2017-04-03 02:24:00 1
2017-04-03 04:48:00 2
2017-04-03 07:12:00 3
2017-04-03 09:36:00 4
2017-04-03 12:00:00 5
2017-04-03 14:24:00 6
2017-04-03 16:48:00 7
2017-04-03 19:12:00 8
2017-04-03 21:36:00 9
last_entry = bro_df.index[-1:]
first_entry = last_entry - pd.Timedelta('3 hour')
print (last_entry)
DatetimeIndex(['2017-04-03 21:36:00'], dtype='datetime64[ns]', freq='144T')
print (first_entry)
DatetimeIndex(['2017-04-03 18:36:00'], dtype='datetime64[ns]', freq=None)
print (last_entry[0])
2017-04-03 21:36:00
print (first_entry[0])
2017-04-03 18:36:00
df = bro_df.loc[first_entry[0]: last_entry[0]]
print (df)
a
2017-04-03 19:12:00 8
2017-04-03 21:36:00 9
df1 = bro_df[first_entry[0]: last_entry[0]]
print (df1)
a
2017-04-03 19:12:00 8
2017-04-03 21:36:00 9
I have a Pandas dataframe, the index/labels are date. I just want to get out the starting date (ie the first entry) and the ending date (ie the last entry). What is the best way to do that?
Any help would be much appreciated.
You could use the index's format method. For example,
In [44]: df = pd.DataFrame({'foo':1}, index=pd.date_range('2000-1-1', periods=5, freq='D'))
In [45]: df
Out[45]:
foo
2000-01-01 1
2000-01-02 1
2000-01-03 1
2000-01-04 1
2000-01-05 1
[5 rows x 1 columns]
In [46]: df.index[[0,-1]].format()
Out[46]: ['2000-01-01', '2000-01-05']
To get the index of adataframe as a list use:
df.index.format()