I'm trying to fill down a column based on 2 conditions. In this case, whether the index (time series) falls between sunrise and sunset, in which case I want 1 in a new column called "sunlight'. Otherwise, I want the value to be zero. I'm new to pandas from excel so I'm trying to do this as I would there, probably wrongly.
df['sunlight'] = 0
mask1 = df.index > df['sunrise']
mask2 = df.index < df['sunset']
df[mask1 & mask2]
df.loc[df[mask1 & mask2],'sunlight'] = 1
df
enter image description here
Index
sunrise
sunset
Sunlight
08:18:00
08:19:17
15:56:43
0
08:19:00
08:19:17
15:56:43
0
08:20:00
08:19:17
15:56:43
1
08:21:00
08:19:17
15:56:43
1
08:22:00
08:19:17
15:56:43
1
Let`s look on a DataFrame with only on day of data with a frequency of one hour (not minutes) as an example.
df = pd.DataFrame({'sunrais':[pd.to_datetime('2020-01-01 08:19:17')]*24,
'sunset':[pd.to_datetime('2020-01-01 15:46:43')]*24 },
index=pd.date_range('2020-01-01 00:00:00', '2020-01-01 23:00:00', freq='H')
)
If you now cast the truth value as integer you can multiply both selections in one step.
df['sunlight'] = (df['sunrais']<df.index).astype(int) * (df.index<df['sunset']).astype(int)
The the output looks like this:
sunrais sunset sunlight
2020-01-01 07:00:00 2020-01-01 08:19:17 2020-01-01 15:46:43 0
2020-01-01 08:00:00 2020-01-01 08:19:17 2020-01-01 15:46:43 0
2020-01-01 09:00:00 2020-01-01 08:19:17 2020-01-01 15:46:43 1
2020-01-01 10:00:00 2020-01-01 08:19:17 2020-01-01 15:46:43 1
Related
Date 1
Date 2
Date 3
Date 4
LineCount
Month_Gap
2020-01-01
2019-10-01
2019-09-06
1
2020-01-01
2019-10-01
2019-09-13
2019-09-06
2
0
2020-01-01
2019-10-01
2019-08-13
2019-09-06
2
1
If the LineCount is 1, then Month_Gap should be the maximum month difference between (Date1 & Date3) and (Date2 & Date3). Date3 will always be in between Date1 and Date2.
In this Case, the output should be the max month difference between (2020/01/01 - 2019/09/06) and (2019/10/01 - 2019/09/06), which is 3 months:
Date 1
Date 2
Date 3
Date 4
LineCount
Month_Gap
2020-01-01
2019-10-01
2019-09-06
1
3
2020-01-01
2019-10-01
2019-09-13
2019-09-06
2
0
2020-01-01
2019-10-01
2019-08-13
2019-09-06
2
1
I was trying something like this but not sure how to go about it -
CASE WHEN LineCount = 1 THEN MAX(DATE_DIFF(.....), which won't work I guess.
The pattern you should use is
SELECT TIMESTAMPDIFF("MONTH", LEAST(date1,date2,date3,date4), GREATEST(date1,date2,date3,date4)) as `maximum_difference`;
This will simply look through your columns, find the least and greatest, and return the result.
SELECT
CASE WHEN LineCount = 1 THEN GREATEST(DATE_DIFF('month', Date3, Date1),
DATE_DIFF('month', Date3, Date2)) END AS Month_Gap
I have the following dataframe
df = pd.DataFrame({'user':['Dan','Dan','Dan','Dan','Ron'], 'start':['2020-01-01 17:00:00',
'2020-01-01 16:20:00','2020-01-01 17:00:00', '2020-01-01 06:30:00','2020-01-01 17:00:00'],
'end':['2020-01-01 21:00:00', '2020-01-02 01:00:00','2020-01-01 21:15:00',
'2020-01-01 10:00:00','2020-01-01 21:00:00']})
user
start
end
Dan
2020-01-01 17:00:00
2020-01-01 21:00:00
Dan
2020-01-01 16:20:00
2020-01-02 01:00:00
Dan
2020-01-01 17:00:00
2020-01-01 21:15:00
Dan
2020-01-01 06:30:00
2020-01-01 10:00:00
Ron
2020-01-01 17:00:00
2020-01-01 21:00:00
For the same user, I would like to leave just one of the records which have overlapping start-end time intervals (no matter which) for example:
user
start
end
Dan
2020-01-01 17:00:00
2020-01-01 21:00:00
Dan
2020-01-01 06:30:00
2020-01-01 10:00:00
Ron
2020-01-01 17:00:00
2020-01-01 21:00:00
My try, for a given user to run:
idx, single_user, single_start, single_end = df.to_records()[0]
a = df.loc[((df['user'] == single_user) & ((df['start'] < single_start) & (df['end']< single_end))
|((df['start']> single_start) &(df['end'] > single_end)))].index.tolist()
a.append(df.iloc[idx].name)
And obtain the result using:
test_df.iloc[a]
There must be a better way, is there a pandas method to tackle it?
Here is a potential approach:
# ensure datetime type for meaningful comparisons
df['end'] = pd.to_datetime(df['end'])
df['start'] = pd.to_datetime(df['start'])
# sort by start time
df = df.sort_values(by='start')
# valid data have a start time greater than the previous end time
# or no previous time (NaT in the shift)
s = df.groupby('user')['end'].shift()
df[df['start'].gt(s)|s.isna()]
Output:
user start end
3 Dan 2020-01-01 06:30:00 2020-01-01 10:00:00
1 Dan 2020-01-01 16:20:00 2020-01-02 01:00:00
4 Ron 2020-01-01 17:00:00 2020-01-01 21:00:00
I had a look to the "sql-like" windows function for pandas, and to "rolling".
However, it seems to me I can't have a condition on timestamps in the index, but maybe I'm wrong.
So far, I've been writing this very inefficient code to have an hourly average as a window function.
Anyone knowing a quicker and nicer method?
def avg_on_hour(data: pd.Series()):
new_series = pd.Series()
start_date = data.index.min()
end_date = data.index.max()
delta = dt.timedelta(hours=1)
this_time = start_date
while this_time < end_date:
this_date = this_time.date()
this_hour = this_time.hour
day_slice = data[(data.index.date == this_date) & (data.index.hour == this_hour)]
day_avg = day_slice.mean()
day_slice.iloc[:] = day_avg
new_series = new_series.append(day_slice, verify_integrity=True)
this_time = this_time + delta
return new_series
Example:
Pandas has rolling on datetime, given that the series is datetime indexed
# sample data:
np.random.seed(1)
size=10
s = pd.Series(np.random.rand(size),
index=pd.date_range('2020-01-01', freq='7T', periods=size))
# rolling mean
series.rolling('1H').mean()
Output:
2020-01-01 00:00:00 0.417022
2020-01-01 00:07:00 0.568673
2020-01-01 00:14:00 0.379154
2020-01-01 00:21:00 0.359948
2020-01-01 00:28:00 0.317310
2020-01-01 00:35:00 0.279815
2020-01-01 00:42:00 0.266450
2020-01-01 00:49:00 0.276339
2020-01-01 00:56:00 0.289720
2020-01-01 01:03:00 0.303252
Freq: 7T, dtype: float64
Update: from your comment, it looks like you are looking for groupby:
s.groupby(s.index.floor('H')).transform('mean')
or
s.groupby(pd.Grouper(freq='H')).transform('mean')
Output:
2020-01-01 00:00:00 0.289720
2020-01-01 00:07:00 0.289720
2020-01-01 00:14:00 0.289720
2020-01-01 00:21:00 0.289720
2020-01-01 00:28:00 0.289720
2020-01-01 00:35:00 0.289720
2020-01-01 00:42:00 0.289720
2020-01-01 00:49:00 0.289720
2020-01-01 00:56:00 0.289720
2020-01-01 01:03:00 0.538817
Freq: 7T, dtype: float64
I have weekly hourly FX data which I need to resample into '1D' or '24hr' bins Monday through Thursday 12:00pm and at 21:00 on Friday, totaling 5 days per week:
Date rate
2020-01-02 00:00:00 0.673355
2020-01-02 01:00:00 0.67311
2020-01-02 02:00:00 0.672925
2020-01-02 03:00:00 0.67224
2020-01-02 04:00:00 0.67198
2020-01-02 05:00:00 0.67223
2020-01-02 06:00:00 0.671895
2020-01-02 07:00:00 0.672175
2020-01-02 08:00:00 0.672085
2020-01-02 09:00:00 0.67087
2020-01-02 10:00:00 0.6705800000000001
2020-01-02 11:00:00 0.66884
2020-01-02 12:00:00 0.66946
2020-01-02 13:00:00 0.6701600000000001
2020-01-02 14:00:00 0.67056
2020-01-02 15:00:00 0.67124
2020-01-02 16:00:00 0.6691699999999999
2020-01-02 17:00:00 0.66883
2020-01-02 18:00:00 0.66892
2020-01-02 19:00:00 0.669345
2020-01-02 20:00:00 0.66959
2020-01-02 21:00:00 0.670175
2020-01-02 22:00:00 0.6696300000000001
2020-01-02 23:00:00 0.6698350000000001
2020-01-03 00:00:00 0.66957
So the number of hours in each some days of the week is uneven, ie "Monday" = 00:00:00 Monday through 12:00:00 Monday, "Tuesday" (and also Weds, Thu) = i.e. 13:00:00 Monday though 12:00:00 Tuesday, and Friday = 13:00:00 through 21:00:00
In trying to find a solution I see that base is now deprecated, and offset/origin methods aren't working as expected, likely due to uneven number of rows per day:
df.rate.resample('24h', offset=12).ohlc()
I've spent hours attempting to find a solution
How can one simply bin into ohlc() columns all data rows between each 12:00:00 timestamp?
the desired output would look something like this:
Out[69]:
open high low close
2020-01-02 00:00:00.0000000 0.673355 0.673355 0.673355 0.673355
2020-01-03 00:00:00.0000000 0.673110 0.673110 0.668830 0.669570
2020-01-04 00:00:00.0000000 0.668280 0.668280 0.664950 0.666395
2020-01-05 00:00:00.0000000 0.666425 0.666425 0.666425 0.666425
Is this what you are looking for, using both origin and offset as parameters:
df.resample('24h', origin='start_day', offset='13h').ohlc()
For your example, this gives me:
open high low close
datetime
2020-01-01 13:00:00 0.673355 0.673355 0.66884 0.66946
2020-01-02 13:00:00 0.670160 0.671240 0.66883 0.66957
Since the period lengths are unequal, IMO it is necessary to craft the mapping wheel yourself. Speaking precisely, the 1.5-day length on Monday makes it impossible for freq='D' to do the mapping correctly at once.
The hand-crafted code is also able to guard against records outside the well-defined periods.
Data
A slightly different timestamp is used to demonstrate the correctness of the code. The days are from Mon. to Fri.
import pandas as pd
import numpy as np
from datetime import datetime
import io
from pandas import Timestamp, Timedelta
df = pd.read_csv(io.StringIO("""
rate
Date
2020-01-06 00:00:00 0.673355
2020-01-06 23:00:00 0.673110
2020-01-07 00:00:00 0.672925
2020-01-07 12:00:00 0.672240
2020-01-07 13:00:00 0.671980
2020-01-07 23:00:00 0.672230
2020-01-08 00:00:00 0.671895
2020-01-08 12:00:00 0.672175
2020-01-08 23:00:00 0.672085
2020-01-09 00:00:00 0.670870
2020-01-09 12:00:00 0.670580
2020-01-09 23:00:00 0.668840
2020-01-10 00:00:00 0.669460
2020-01-10 12:00:00 0.670160
2020-01-10 21:00:00 0.670560
2020-01-10 22:00:00 0.671240
2020-01-10 23:00:00 0.669170
"""), sep=r"\s{2,}", engine="python")
df.set_index(pd.to_datetime(df.index), inplace=True)
Code
def find_day(ts: Timestamp):
"""Find the trading day with irregular length"""
wd = ts.isoweekday()
if wd == 1:
return ts.date()
elif wd in (2, 3, 4):
return ts.date() - Timedelta("1D") if ts.hour <= 12 else ts.date()
elif wd == 5:
if ts.hour <= 12:
return ts.date() - Timedelta("1D")
elif 13 <= ts.hour <= 21:
return ts.date()
# out of range or nulls
return None
# map the timestamps, and set as new index
df.set_index(pd.DatetimeIndex(df.index.map(find_day)), inplace=True)
# drop invalid values and collect ohlc
ans = df["rate"][df.index.notnull()].resample("D").ohlc()
Result
print(ans)
open high low close
Date
2020-01-06 0.673355 0.673355 0.672240 0.672240
2020-01-07 0.671980 0.672230 0.671895 0.672175
2020-01-08 0.672085 0.672085 0.670580 0.670580
2020-01-09 0.668840 0.670160 0.668840 0.670160
2020-01-10 0.670560 0.670560 0.670560 0.670560
I ended up using a combination of grouby and datetime day of the week identification to arrive at my specific solution
# get idxs of time to rebal (12:00:00)-------------------------------------
df['idx'] = range(len(df)) # get row index
days = [] # identify each row by day of week
for i in range(len(df.index)):
days.append(df.index[i].date().weekday())
df['day'] = days
dtChgIdx = [] # stores "12:00:00" rows
justDates = df.index.date.tolist() # gets just dates
res = [] # removes duplicate dates
[res.append(x) for x in justDates if x not in res]
justDates = res
grouped_dates = df.groupby(df.index.date) # group entire df by dates
for i in range(len(grouped_dates)):
tempDf = grouped_dates.get_group(justDates[i]) # look at each grouped dates
if tempDf['day'][0] == 6:
continue # skip Sundays
times = [] # gets just the time portion of index
for y in range(len(tempDf.index)):
times.append(str(tempDf.index[y])[-8:])
tempDf['time'] = times # add time column to df
tempDf['dayCls'] = np.where(tempDf['time'] == '12:00:00',1,0) # idx "12:00:00" row
dtChgIdx.append(tempDf.loc[tempDf['dayCls'] == 1, 'idx'][0]) # idx value
I have a dataframe (df), contains datetime columns startdate, enddate and volume of product
If I want to look at one particular date that fit in between startdate and enddate and its total volume, i can do it with no problem at all (see code).
However if I create a second dataframe (call it report), create a list of date that I would like to look at the total volume of product from first df, I came up with an error:
Can only compare identically-labeled Series objects
I read up on things like dropping index on the second df or sorting dates but they don't seem to work
So my working code for requesting volume fitted within startdate and enddate, say first of july 2019:
df[(df['StartDate'] >= '2019-07-01') & (df['EndDate'] <= '2019-10-31')]['Volume'].sum()
but if i create a second df (report):
report = pd.Series(pd.date_range('today', periods=len(df), freq='D').normalize(),name='Date')
report = pd.DataFrame(report)
and request what i want to see:
report['trial'] = df[(df['StartDate'] >= report.Date) & (df['EndDate'] <= report.Date)]['Volume'].sum()
got this error: 'Can only compare identically-labeled Series objects'
Any advice/suggestions welcome, thanks!
First, some sample data:
np.random.seed(42)
dates = pd.date_range('2019-01-01', '2019-12-01', freq='MS')
df = pd.DataFrame({
'StartDate': dates,
'EndDate': dates + pd.offsets.MonthEnd(),
'Volume': np.random.randint(1, 10, len(dates))
})
StartDate EndDate Volume
0 2019-01-01 2019-01-31 7
1 2019-02-01 2019-02-28 4
2 2019-03-01 2019-03-31 8
3 2019-04-01 2019-04-30 5
4 2019-05-01 2019-05-31 7
5 2019-06-01 2019-06-30 3
6 2019-07-01 2019-07-31 7
7 2019-08-01 2019-08-31 8
8 2019-09-01 2019-09-30 5
9 2019-10-01 2019-10-31 4
10 2019-11-01 2019-11-30 8
11 2019-12-01 2019-12-31 8
And the report dates:
reports = pd.to_datetime(['2019-01-15', '2019-02-15', '2019-08-15'])
Using numpy's array broadcasting:
start = df['StartDate'].values
end = df['EndDate'].values
d = reports.values[:, None]
df[np.any((start <= d) & (d <= end), axis=0)]
Result:
StartDate EndDate Volume
0 2019-01-01 2019-01-31 7
1 2019-02-01 2019-02-28 4
7 2019-08-01 2019-08-31 8