Generate time series dates after a certain date in each group in Pyspark dataframe - dataframe
I have this dataframe -
data = [(0,1,1,201505,3),
(1,1,1,201506,5),
(2,1,1,201507,7),
(3,1,1,201508,2),
(4,2,2,201750,3),
(5,2,2,201751,0),
(6,2,2,201752,1),
(7,2,2,201753,1)
]
cols = ['id','item','store','week','sales']
data_df = spark.createDataFrame(data=data,schema=cols)
display(data_df)
What I want it this -
data_new = [(0,1,1,201505,3,0),
(1,1,1,201506,5,0),
(2,1,1,201507,7,0),
(3,1,1,201508,2,0),
(4,1,1,201509,0,0),
(5,1,1,201510,0,0),
(6,1,1,201511,0,0),
(7,1,1,201512,0,0),
(8,2,2,201750,3,0),
(9,2,2,201751,0,0),
(10,2,2,201752,1,0),
(11,2,2,201753,1,0),
(12,2,2,201801,0,0),
(13,2,2,201802,0,0),
(14,2,2,201803,0,0),
(15,2,2,201804,0,0)]
cols_new = ['id','item','store','week','sales','flag',]
data_df_new = spark.createDataFrame(data=data_new,schema=cols_new)
display(data_df_new)
So basically, I want 8 (this can also be 6 or 10) weeks of data for each item-store groupby combination. Wherever the 52/53 weeks for the year ends, I need the weeks for the next year, as I have mentioned in the sample. I need this in PySpark, thanks in advance!
Related
pandas equivalent of percentage growth
Is there any way how to calculate the percentage growth (into the future) in pandas?
pandas have .pct_change method to calculate the percent change of some columns.
I would like to perform this in the future - my function does the work. however, I find it kind of weird to be using a for-loop for some calculations
def cf_future_projection(
cashflow_of_last_year: float,
cashflow_pct_grow: float,
last_observed_year: int,
n_year_future: int = 5,
) -> dict:
grow_values = {}
grow_values[last_observed_year + 1] = cashflow_of_last_year * (
1 + cashflow_pct_grow
)
for year in range(1, n_year_future):
grow_values[last_observed_year + 1 + year] = grow_values[
last_observed_year + 1 + year - 1
] * (1 + cashflow_pct_grow)
return grow_values
cf_future_projection(150, 0.15, 2020, 15)
Any way how to do that in pandas and without a for loop?
When you run cf_future_projection(150, 0.15, 2020, 15), the basic calculation you are performing is 150*(1+0.15)^n for n years into the future, so I think that your function, while nicely written, is unnecessarily complicated.
I don't know exactly what your use case is, but I think if you want to create a dataframe with new rows at the end, adding news rows one at a time is an expensive operation, and you probably don't want to use a for loop with dataframes as you mentioned. You might be better off taking the last row of an existing dataframe, creating a new dataframe with projected future values, and concatenating the original and new dataframes together.
For example, let's say you're starting with a dataframe that looks like:
df = pd.DataFrame({'year':[2019,2020],'value':[140,150]})
To do something similar to cf_future_projection(150, 0.15, 2020, 15), we can take the starting value from the row of the dataframe corresponding to 2020, and then use a list comprehension to create our new future values and future years. If you like, you can wrap this operation in a function
year,value = df[df['year'] == 2020].values[0]
n = 5
year_future = [2020 + i for i in range(1,n+1)]
value_future = [value*(1+0.15)**i for i in range(1,n+1)]
df_future = pd.DataFrame({'year':year_future,'value':value_future})
df_future = pd.concat([df,df_future])
Result:
>>> df_future
year value
0 2019 140.000000
1 2020 150.000000
0 2021 172.500000
1 2022 198.375000
2 2023 228.131250
3 2024 262.350937
4 2025 301.703578
Pandas Cumulative sum over 1 indice but not the other 3
I have a dataframe with 4 variables DIVISION, QTR, MODEL_SCORE, MONTH with the sum of variable X aggregated by those 4. I would like to effective partition the data by DIVISION,QTR, and MODEL SCORE and keep a running total order the MONTH FIELD order smallest to largest. The idea being it would reset if it got to a new permutation of the other 3 columns df = df.groupby(['DIVISION','MODEL','QTR','MONTHS'])['X'].sum() I'm trying df['cumsum'] = df.groupby(level=3)['X'].cumsum() having tried all numbers I can think in the level argument. It seems be able to work any way other than what I want. EDIT: I know the below isn't formatted ideally, but basically as long as the only variable changing was MONTH the cumulative sum would continue but any other variable would cause it to reset. DIVSION QTR MODEL MONTHS X CUMSUM A 1 1 1 10 10 A 1 1 2 20 30 A 1 2 1 5 5
I'm sorry for all the trouble I believe the answer was way simpler than I was making it to be. After df = df.groupby(['DIVISION','MODEL','QTR','MONTHS'])['X'].sum() I was supposed to reset the index I did not want a multi-index and this appears to have worked. df = df.reset_index() df['cumsum'] = df.groupby(['DIVISION','MODEL','QTR'])['X'].cumsum()
How can I optimize my for loop in order to be able to run it on a 320000 lines DataFrame table?
I think I have a problem with time calculation.
I want to run this code on a DataFrame of 320 000 lines, 6 columns:
index_data = data["clubid"].index.tolist()
for i in index_data:
for j in index_data:
if data["clubid"][i] == data["clubid"][j]:
if data["win_bool"][i] == 1:
if (data["startdate"][i] >= data["startdate"][j]) & (
data["win_bool"][j] == 1
):
NW_tot[i] += 1
else:
if (data["startdate"][i] >= data["startdate"][j]) & (
data["win_bool"][j] == 0
):
NL_tot[i] += 1
The objective is to determine the number of wins and the number of losses from a given match taking into account the previous match, this for every clubid.
The problem is, I don't get an error, but I never obtain any results either.
When I tried with a smaller DataFrame ( data[0:1000] ) I got a result in 13 seconds. This is why I think it's a time calculation problem.
I also tried to first use a groupby("clubid"), then do my for loop into every group but I drowned myself.
Something else that bothers me, I have at least 2 lines with the exact same date/hour, because I have at least two identical dates for 1 match. Because of this I can't put the date in index.
Could you help me with these issues, please?
As I pointed out in the comment above, I think you can simply sum the vector of win_bool by group. If the dates are sorted this should be equivalent to your loop, correct?
import pandas as pd
dat = pd.DataFrame({
"win_bool":[0,0,1,0,1,1,1,0,1,1,1,1,1,1,0],
"clubid": [1,1,1,1,1,1,1,2,2,2,2,2,2,2,2],
"date" : [1,2,1,2,3,4,5,1,2,1,2,3,4,5,6],
"othercol":["a","b","b","b","b","b","b","b","b","b","b","b","b","b","b"]
})
temp = dat[["clubid", "win_bool"]].groupby("clubid")
NW_tot = temp.sum()
NL_tot = temp.count()
NL_tot = NL_tot["win_bool"] - NW_tot["win_bool"]
If you have duplicate dates that inflate the counts, you could first drop duplicates by dates (within groups):
# drop duplicate dates
temp = dat.drop_duplicates(["clubid", "date"])[["clubid", "win_bool"]].groupby("clubid")
Pandas: Creating an index year ago for multiple items
I can create an index vs the previous year when I have just one item, but I'm trying to figure out how to do this when I have multiple items. Here is my data set:
rng = pd.date_range('1/1/2011', periods=3, freq='Y')
rng = np.repeat(rng,3)
country = ["USA","Brazil","Japan"]*3
df = pd.DataFrame({'Country':country,'date':rng,'value':range(20,29)})
If I only had one item/country I can do something like this:
df['pct_iya'] = 100*(df['value'].pct_change()+1)
I'm trying to get this to work with multiple items. Here is the expected result:
Maybe this could work with a groupby, but my attempt did not work...
df['pct_iya2'] = df.groupby(['Country','date'])['value'].pct_change()
Answer: Use a group by (excluding date) and than add one to the percent change (ex +15percent goes from .15 to 1.15), then multiple you 100. df['pct_iya'] = 100*(df.groupby(['Country'])['value'].pct_change()+1)
Make a plot by occurence of a col by hour of a second col
I have this df : and i would like to make a graph by half hour of how many row i have by half hour without including the day. Just a graph with number of occurence by half hour not including the day. 3272 8711600410367 2019-03-11T20:23:45.415Z d7ec8e9c5b5df11df8ec7ee130552944 home 2019-03-11T20:23:45.415Z DISPLAY None 3273 8711600410367 2019-03-11T20:23:51.072Z d7ec8e9c5b5df11df8ec7ee130552944 home 2019-03-11T20:23:51.072Z DISPLAY None Here is my try : df["Created"] = pd.to_datetime(df["Created"]) df.groupby(df.Created.dt.hour).size().plot() But it's not by half hour I would like to show all half hour on my graph
One way you could do this is split up coding for hours and half-hours, and then bring them together. To illustrate, I extended your data example a bit:
import pandas as pd
df = pd.DataFrame({'Created':['2019-03-11T20:23:45.415Z', '2019-03-11T20:23:51.072Z', '2019-03-11T20:33:03.072Z', '2019-03-11T21:10:10.072Z']})
df["Created"] = pd.to_datetime(df["Created"])
First create a 'Hours column':
df['Hours'] = df.Created.dt.hour
Then create a column that codes half hours. That is, if the minutes are greater than 30, count it as half hour.
df['HalfHours'] = [0.5 if x>30 else 0 for x in df.Created.dt.minute]
Then bring them together again:
df['Hours_and_HalfHours'] = df['Hours']+df['HalfHours']
Finally, count the number of rows by groupby, and plot:
df.groupby(df['Hours_and_HalfHours']).size().plot()