Having a Pandas DataFrame with a column of TimeStamp yyyy-mm-dd HH:MM:SS timezone (e.g. 2020-06-01 04:26:00-05:00), how to extract new column with only yyyy-mm-dd HH
Tried:
df.index = df.Time.to_period(freq='T').index
Result in: yyyy-mm-dd HH:MM
you can use:
df['new_date']=df['your_date_columns'].dt.strftime('%Y-%m-%d %H')
Related
I have a impala table where date column values are stored in dd-mm-yy HH:mm:ss string format, e.g.
30-11-20 12:34:45
I want to convert them into yyyy-mm-dd HH:mm:ss.SSS, e.g.
2020-11-30 12:34:45.000
If anyone could suggest a way to achieve this!!
first use to_timestamp to convert to timestamp. Then use from_timestamp to convert to string.
select from_timestamp(
to_timestamp('30-11-20 12:34:45','dd-MM-yy HH:mm:ss')
,'yyyy-MM-dd HH:mm:ss.SSS') as str_timestamp
A column in dataframe keeps date like:
2019-06-19 23:04:36
2018-06-29 20:06:56
2019-03-04 11:12:35
2019-07-12 21:16:44
I tried the below code but it gives not correct results:
df['timestamps'] = pd.to_datetime(df['datetimes']).astype('int64') / 10**9
The results are like these:
1.465506e+09
1.465516e+09
1.465503e+09
If I convert them again to date, I get incorrect date time:
df['new'] = df['timestamps'].apply(lambda x: pd.Timestamp(x))
1970-01-01 00:00:01.465506396
1970-01-01 00:00:01.465506397
1970-01-01 00:00:01.465506397
Something is not correct...
What is the way to convert date time that is as string like "2019-06-19 23:04:36" to timestamp?
Thank you.
I have a Pandas Dataframe that stores date in the format 19-Jul-18. I am trying to convert it to 2018-07-19
I tried doing pd.to_datetime(df['date']) but it dint help.
In hive there is some data I have. Now I want to convert the start_timestamp into unix_timestamp in second. How to do that? Because the start_timestamp has two formats:
First format:
2018-03-22 02:54:35
Second format:
May 15 2018 5:15PM
First format is 'yyyy-MM-dd HH:mm:ss', second is 'MMM dd yyyy hh:mm:aa'. If the format is wrong, unix_timestamp function will return NULL. Try to convert using one format, if NULL, try to convert using the other format. This can be done using coalesce function:
select
coalesce(unix_timestamp(start_timestamp ,'yyyy-MM-dd HH:mm:ss'),
unix_timestamp(start_timestamp ,'MMM dd yyyy hh:mm:aa')
) as UnixTimestamp
from my_table;
Use from_unixtime() to convert it back to given format if necessary, like in this answer.
See patterns examples here: SimpleDateFormat
After loading data from a csv file, I set the index to the "Date" column and then convert the index to datetime.
df1=pd.read_csv('Data.csv')
df1=df1.set_index('Date')
df1.index=pd.to_datetime(df1.index)
However after conversion the datetime format shows it has been misinterpreted:
original date was e.g. 01-10-2014 00:00:00
but Pandas converts it to 2014-01-10 00:00:00
How can I get Pandas to respect or recognize the original date format?
Thank you
Your datestrings were being interpreted as month first, you need to specify the correct format:
df1.index=pd.to_datetime(df1.index, format='%d-%m-%Y %H:%M:%S')
so that it doesn't interpret the first part as the month
In [128]:
pd.to_datetime('01-10-2014 00:00:00', format='%d-%m-%Y %H:%M:%S')
Out[128]:
Timestamp('2014-10-01 00:00:00')