Pandas DataFrame Time Conversion - pandas

How to change for example '01:31:41' the value display in stopwatch which is into seconds value. E.g this is 1 min and 31 second, so roughly about 91 seconds.

split string into a list object
create a timedelta object without miliseconds as they aren't needed
we use the self method of timedelta total_seconds()
from datetime import timedelta
original_time_string = "01:30:12"
list_string = original_time_string.split(":")
print((timedelta(minutes=int(list_string[0]),seconds=int(list_string[1])).total_seconds()))

Related

TypeError: dtype datetime64[ns] cannot be converted to timedelta64[ns]

I have a column of years from the sunspots dataset.
I want to convert column 'year' in integer e.g. 1992 to datetime format then find the time delta and eventually compute total seconds (cumulative) to represent the time index column of a time series.
I am trying to use the following code but I get the error
TypeError: dtype datetime64[ns] cannot be converted to timedelta64[ns]
sunspots_df['year'] = pd.to_timedelta(pd.to_datetime(sunspots_df['year'], format='%Y') ).dt.total_seconds()
pandas.Timedelta "[r]epresents a duration, the difference between two dates or times." So you're trying to get Python to tell you the difference between a particular datetime and...nothing. That's why it's failing.
If it's important that you store your index this way (and there may be better ways), then you need to pick a start datetime and compute the difference to get a timedelta.
For example, this code...
import pandas as pd
df = pd.DataFrame({'year': [1990,1991,1992]})
diff = (pd.to_datetime(df['year'], format='%Y') - pd.to_datetime('1990', format='%Y'))\
.dt.total_seconds()
...returns a series whose values are seconds from January 1st, 1990. You'll note that it doesn't invoke pd.to_timedelta(), because it doesn't need to: the result of the subtraction is automatically a pd.timedelta column.

Convert DateTime to TimeStamp Pandas

The objective of this post is to be able to convert the columns [‘Open Date’, 'Close date’] to timestamp format
I have tried with the functions / examples from these links with any results.
Convert datetime to timestamp in Neo4j
Convert datetime pandas
Pandas to_dict() converts datetime to Timestamp
Really appreciate any ideas / comments / examples on how to do so.
Data Base Image
Column Characteristics:
Open Date datetime64[ns] and pandas.core.series.Series
Close date datetime64[ns] and pandas.core.series.Series
Finally I been using these libraries
import pandas as pd
import numpy as np
from datetime import datetime, date, time, timedelta
You convert first to numpy array by values and transform (cast) to int64 - output is in nanoseconds , which means divide by 10 ** 9:
df['open_ts'] = df['Open_Date'].datetime.values.astype(np.int64)
df['close_ts'] = df['Close_Date'].datetime.values.astype(np.int64)
OR
If you want to avoid using numpy, you can also try:
df['open_ts'] = pd.to_timedelta(df['Open_Date'], unit='ns').dt.total_seconds().astype(int)
df['close_ts'] = pd.to_timedelta(df['Close_Date'], unit='ns').dt.total_seconds().astype(int)
Try them and report it back here

How can I filter a pandas data frame based on a datetime column between current time and 10 hours ago?

I have a pandas DataFrame which includes a datetime column and I want to filter the data frame between the current hour and 10 hours ago. I have tried different ways to do it but still I cannot handle it. Because when I want to use pandas, the column type is Series and I can't use timedelta to compare them. If I use a for loop to compare the column as a string to my time interval, it is not efficient.
The table is like this:
And I want to filter the 'dateTime' column between current time and 10 hours ago, then filter based on 'weeks' > 80.
I have tried these codes as well But they have not worked:
filter_criteria = main_table['dateTime'].sub(today).abs().apply(lambda x: x.hours <= 10)
main_table.loc[filter_criteria]
This returns an error:
TypeError: unsupported operand type(s) for -: 'str' and 'datetime.datetime'
Similarly this code has the same problem:
main_table.loc[main_table['dateTime'] >= (datetime.datetime.today() - pd.DateOffset(hours=10))]
And:
main_table[(pd.to_datetime('today') - main_table['dateTime'] ).dt.hours.le(10)]
In all of the code above main_table is the name of my data frame.
How can I filter them?
First you need to make sure that your datatype in datetime column is correct. you can check it by using:
main_table.info()
If it is not datetime (i.e, object) convert it:
# use proper formatting if this line does not work
main_table['dateTime'] = pd.to_datetime(main_table['dateTime'])
Then you need to find the datetime object of ten hour before current time (ref):
from datetime import datetime, timedelta
date_time_ten_before = datetime.now() - timedelta(hours = 10)
All it remains is to filter the column:
main_table_10 = main_table[main_table['dateTime'] >= date_time_ten_before]

Pandas returning datetime instead of datetime.time when time is 00:00:00

I am reading a sheet from excel file that contains block of hours.
Excel sheet
cls.model_slots_result = pandas.read_excel(model_static_results_xlsx, sheet_name='model_slots',
engine='openpyxl')
When the time is 00:00:00, pandas load it as a Timestamp ('1899-12-30 00:00:00') i.e. Datetime in the dataframe even though other values (block of hours) of the same column are stored as datetime.time.
Dataframe image
As of now I have created a function that checks if the type is datetime.time, if not it extracts time from datetime.
cls.model_slots_result['START_TIME'] = cls.model_slots_result['START_TIME'].apply(my_to_datetime)
cls.model_slots_result['END_TIME'] = cls.model_slots_result['END_TIME'].apply(my_to_datetime)
def my_to_datetime(datetime_obj):
if not isinstance(datetime_obj, time):
return datetime_obj.time()
return datetime_obj
Is there any way to make pandas return datetime.time object for 00:00:00 instead of a datetime object or make pandas read the the column as datetime.time?
Thanks

String to datetime in pandas reversed

I am dealing with time objects saved as strings in the form 57:44.6 (second, minute, hour).I am trying to convert the column elements to datetime using pd_todatetime. There results are Nat. How can i change the format of the string to HH:MM:SS (6:44:57)before converting?
provide the appropriate format specifier '%S:%M.%H'. Ex:
import pandas as pd
s = pd.Series(['57:44.6'])
dts = pd.to_datetime(s, format='%S:%M.%H')
# dts
# 0 1900-01-01 06:44:57
# dtype: datetime64[ns]