All, I have a dataframe with a date column and an hour column. I am trying to combine those into a single timestamp. I tried many solutions available using datetime.datetime.combine and just implicitly extracting month day and year and creating a datetime stamp with it but all lead to some error.
idOnController date eventTime Energy hour
0 5014 2018-05-31 2018-05-31 01:00:00 26.619 0
2 5014 2018-06-02 2018-06-02 02:00:00 29.251 0
3 5014 2018-06-03 2018-06-03 03:00:00 30.635 0
The datatypes are as follows
idOnController int64
date object
eventTime datetime64[ns]
Energy float64
hour int64
dtype: object
I am looking to combine date and hour into a timestamp that looks like eventTime and then replace eventTime with that value.
You can do:
df['new_date'] = pd.to_datetime(df['date']) + df['hour'] * pd.to_timedelta('1H')
Output of df.dtypes:
idOnController int64
date object
eventTime datetime64[ns]
Energy float64
hour int64
new_date datetime64[ns]
dtype: object
If you want to have the string timestamps you can do
df['new_date'] = df['new_date'].dt.strftime('%Y-%m-%d %H:%M:%S')
Another way of doing this would be (a bit more verbose though!):
df['date'] = pd.to_datetime(df['date'])
df['year'] = df.date.dt.year
df['month'] = df.date.dt.month
df['day'] = df.date.dt.day
df['date'] = pd.to_datetime(df[['year','month','day','hour']])
Related
I have a dataframe with mixed date formats across and within columns. When trying to convert them from object to datetime type, I get an error due to column date1 having a mixed format. I can't see how to fix it in this case. Also, how could I remove the seconds from both columns (date1 and date2)?
Here's the code I attempted:
df = pd.DataFrame(np.array([[10, "2021-06-13 12:08:52.311 UTC", "2021-03-29 12:44:33.468"],
[36, "2019-12-07 12:18:02 UTC", "2011-10-15 10:14:32.118"]
]),
columns=['col1', 'date1', 'date2'])
df
>>
col1 date1 date2
0 10 2021-06-13 12:08:52.311 UTC 2021-03-29 12:44:33.468
1 36 2019-12-07 12:18:02 UTC 2011-10-15 10:14:32.118
# Converting from object to datetime
df["date1"]= pd.to_datetime(df["date1"], format="%Y-%m-%d %H:%M:%S.%f UTC")
df["date2"]= pd.to_datetime(df["date2"], format="%Y-%m-%d %H:%M:%S.%f")
>>
ValueError: time data '2019-12-07 12:18:02 UTC' does not match format '%Y-%m-%d %H:%M:%S.%f UTC' (match)
for conversion to datetime, i found the infer_datetime_format to be helpful.
could not get it to work on the complete dataframe, it is able to convert one column at a time.
In [19]: pd.to_datetime(df["date1"], infer_datetime_format=True)
Out[19]:
0 2021-06-13 12:08:52.311000+00:00
1 2019-12-07 12:18:02+00:00
Name: date1, dtype: datetime64[ns, UTC]
In [20]: pd.to_datetime(df["date2"], infer_datetime_format=True)
Out[20]:
0 2021-03-29 12:44:33.468
1 2011-10-15 10:14:32.118
Name: date2, dtype: datetime64[ns]
If atleast all formats start with this format "%Y-%m-%d %H:%M" , then you can just slice all strings till that point and use them
In [32]: df['date1'].str.slice(stop=16)
Out[32]:
0 2021-06-13 12:08
1 2019-12-07 12:18
Name: date1, dtype: object
for getting rid of the seconds in your datetime values, instead of simply getting rid of those values, you can use round , you can also check floor and ceil whatever suits your use case better.
In [28]: pd.to_datetime(df["date1"], infer_datetime_format=True).dt.round('T')
Out[28]:
0 2021-06-13 12:09:00+00:00
1 2019-12-07 12:18:00+00:00
Name: date1, dtype: datetime64[ns, UTC]
In [29]: pd.to_datetime(df["date2"], infer_datetime_format=True).dt.round('T')
Out[29]:
0 2021-03-29 12:45:00
1 2011-10-15 10:15:00
Name: date2, dtype: datetime64[ns]
I have a column of timestamps that span over 24 hours. I want to convert these to differentiate between days. I've done this by converting to timedelta. The result is displayed below.
The question I have is, can these be converted or re-arranged again to provide random datetimes. e.g. dd:mm:yyyy hh:mm:ss.
import pandas as pd
df = pd.DataFrame({
'Time' : ['8:00','18:00','28:00'],
})
df['Time'] = [x + ':00' for x in df['Time']]
df['Time'] = pd.to_timedelta(df['Time'])
Out:
Time
0 0 days 08:00:00
1 0 days 18:00:00
2 1 days 04:00:00
Intended Output:
Time
0 1/01/1904 08:00:00 AM
1 1/01/1904 18:00:00 PM
2 2/01/1904 04:00:00 AM
The input timestamps will never go over more than 2 days. Is there a package that can achieve this or would a dummy start and end dates.
After you convert the Time just adding the date part
df.Time+pd.to_datetime('1904-01-01')
0 1904-01-01 08:00:00
1 1904-01-01 18:00:00
2 1904-01-02 04:00:00
Name: Time, dtype: datetime64[ns]
I have a dataframe in pandas called 'munged_data' with two columns 'entry_date' and 'dob' which i have converted to Timestamps using pd.to_timestamp.I am trying to figure out how to calculate ages of people based on the time difference between 'entry_date' and 'dob' and to do this i need to get the difference in days between the two columns ( so that i can then do somehting like round(days/365.25). I do not seem to be able to find a way to do this using a vectorized operation. When I do munged_data.entry_date-munged_data.dob i get the following :
internal_quote_id
2 15685977 days, 23:54:30.457856
3 11651985 days, 23:49:15.359744
4 9491988 days, 23:39:55.621376
7 11907004 days, 0:10:30.196224
9 15282164 days, 23:30:30.196224
15 15282227 days, 23:50:40.261632
However i do not seem to be able to extract the days as an integer so that i can continue with my calculation.
Any help appreciated.
Using the Pandas type Timedelta available since v0.15.0 you also can do:
In[1]: import pandas as pd
In[2]: df = pd.DataFrame([ pd.Timestamp('20150111'),
pd.Timestamp('20150301') ], columns=['date'])
In[3]: df['today'] = pd.Timestamp('20150315')
In[4]: df
Out[4]:
date today
0 2015-01-11 2015-03-15
1 2015-03-01 2015-03-15
In[5]: (df['today'] - df['date']).dt.days
Out[5]:
0 63
1 14
dtype: int64
You need 0.11 for this (0.11rc1 is out, final prob next week)
In [9]: df = DataFrame([ Timestamp('20010101'), Timestamp('20040601') ])
In [10]: df
Out[10]:
0
0 2001-01-01 00:00:00
1 2004-06-01 00:00:00
In [11]: df = DataFrame([ Timestamp('20010101'),
Timestamp('20040601') ],columns=['age'])
In [12]: df
Out[12]:
age
0 2001-01-01 00:00:00
1 2004-06-01 00:00:00
In [13]: df['today'] = Timestamp('20130419')
In [14]: df['diff'] = df['today']-df['age']
In [16]: df['years'] = df['diff'].apply(lambda x: float(x.item().days)/365)
In [17]: df
Out[17]:
age today diff years
0 2001-01-01 00:00:00 2013-04-19 00:00:00 4491 days, 00:00:00 12.304110
1 2004-06-01 00:00:00 2013-04-19 00:00:00 3244 days, 00:00:00 8.887671
You need this odd apply at the end because not yet full support for timedelta64[ns] scalars (e.g. like how we use Timestamps now for datetime64[ns], coming in 0.12)
Not sure if you still need it, but in Pandas 0.14 i usually use .astype('timedelta64[X]') method
http://pandas.pydata.org/pandas-docs/stable/timeseries.html (frequency conversion)
df = pd.DataFrame([ pd.Timestamp('20010101'), pd.Timestamp('20040605') ])
df.ix[0]-df.ix[1]
Returns:
0 -1251 days
dtype: timedelta64[ns]
(df.ix[0]-df.ix[1]).astype('timedelta64[Y]')
Returns:
0 -4
dtype: float64
Hope that will help
Let's specify that you have a pandas series named time_difference which has type
numpy.timedelta64[ns]
One way of extracting just the day (or whatever desired attribute) is the following:
just_day = time_difference.apply(lambda x: pd.tslib.Timedelta(x).days)
This function is used because the numpy.timedelta64 object does not have a 'days' attribute.
To convert any type of data into days just use pd.Timedelta().days:
pd.Timedelta(1985, unit='Y').days
84494
i have the following data frame :
correction = ['2.0','-2.5','4.5','-3.0']
date = ['2015-05-19 20:45:00','2017-04-29 17:15:00','2011-05-09 10:40:00','2016-12-18 16:10:00']
i want to convert correction as hours and add it to the date. i tried the following code, but it get the error.
df['correction'] = pd.to_timedelta(df['correction'],unit='h')
df['date'] =pd.DatetimeIndex(df['date'])
df['date'] = df['date'] + df['correction']
I get the error in converting correction to timedelta as:
ValueError: no units specified
For me works cast to float column correction:
df['correction'] = pd.to_timedelta(df['correction'].astype(float),unit='h')
df['date'] = pd.DatetimeIndex(df['date'])
df['date'] = df['date'] + df['correction']
print (df)
correction date
0 02:00:00 2015-05-19 22:45:00
1 -1 days +21:30:00 2017-04-29 14:45:00
2 04:30:00 2011-05-09 15:10:00
3 -1 days +21:00:00 2016-12-18 13:10:00
I need your help guys
I have information with wrong time format.
For example:
it shows 1245 or 1837 etc. I want them to be like in correct format:
12:45 PM or 6:37 PM.
How can I convert it?
Thanks!
I think you need convert to_datetime and then strftime or dt.time:
See also http://strftime.org/.
df = pd.DataFrame({'date':[1245, 1837]})
print (df)
date
0 1245
1 1837
print (pd.to_datetime(df['date'], format='%H%M'))
0 1900-01-01 12:45:00
1 1900-01-01 18:37:00
Name: date, dtype: datetime64[ns]
#for string output
print (pd.to_datetime(df['date'], format='%H%M').dt.strftime('%I:%M %p'))
0 12:45 PM
1 06:37 PM
Name: date, dtype: object
#for time output
print (pd.to_datetime(df['date'], format='%H%M').dt.time)
0 12:45:00
1 18:37:00
Name: date, dtype: object