I have a df as follows:
appid month tag totalvalue
0 1234 02-'22 B 50.00
1 1234 02-'22 BA 10.00
2 1234 01-'22 B 100.00
3 2345 03-'22 BA 25.00
4 2345 03-'22 B 100.00
5 2345 04-'22 BB 100.00
Output what I want is follows:
appid month tag totalvalue %tage
0 1234 02-'22 B 50.00 1.0
1 1234 02-'22 BA 10.00 0.2
2 1234 01-'22 B 100.00 1.0
3 2345 03-'22 BA 25.00 0.25
4 2345 03-'22 B 100.00 1.0
5 2345 04-'22 BB 100.00 inf
I want to have group variables based on appid & month. Moreover want to check if there are tag=B is available in that group just divide other tag's totalvalue with it. If not shows the inf
I have tried with df.groupby(['appid', 'month'])['totalvalue'] but unable to replicate them with condition of tag=B as denominator over groupby object.
IIUC, you can use a groupby.transform('first') on the masked totalvalue, then use it a divider:
m = df['tag'].eq('B')
df['%tage'] = (df['totalvalue']
.div(df['totalvalue'].where(m)
.groupby([df['appid'], df['month']])
.transform('first').fillna(0))
)
output:
appid month tag totalvalue %tage
0 1234 02-'22 B 50.0 1.00
1 1234 02-'22 BA 10.0 0.20
2 1234 01-'22 B 100.0 1.00
3 2345 03-'22 BA 25.0 0.25
4 2345 03-'22 B 100.0 1.00
5 2345 04-'22 BB 100.0 inf
Related
Following df:
appid tag totalvalue
0 1234 B 50.00
1 1234 BA 10.00
2 2345 B 100.00
3 2345 BA 25.00
4 2345 BCS 15.00
What we want is to group the df with appid and have some analysis based on tag column, is such that if each tag is divided by tag='B' with totalvalue. Just like follows:
appid tag total %tage(B)
0 1234 B 50.00 1
1 1234 BA 10.00 0.2
2 2345 B 100.00 1
3 2345 BA 25.00 0.4
4 2345 BCS 15.00 0.15
You can use groupby:
gmax = df['totalvalue'].where(df['tag'] == 'B').groupby(df['appid']).transform('max')
df['%tage(B)'] = 1 / (gmax / df['totalvalue'])
print(df)
# Output
appid tag totalvalue %tage(B)
0 1234 B 50.0 1.00
1 1234 BA 10.0 0.20
2 2345 B 100.0 1.00
3 2345 BA 25.0 0.25
4 2345 BCS 15.0 0.15
as_of_date
industry
sector
deal
year
quarter
stage
amount
yield
0
2022-01-01
Mortgage
RMBS
XYZ
2022
NaN
A
111
0.1
1
2022-01-01
Mortgage
RMBS
XYZ
2022
1
A
222
0.2
2
2022-01-01
Mortgage
RMBS
XYZ
2022
2
A
333
0.3
3
2022-01-01
Mortgage
RMBS
XYZ
2022
3
A
444
0.4
4
2022-01-01
Mortgage
RMBS
XYZ
2022
4
A
555
0.5
5
2022-01-01
Mortgage
RMBS
XYZ
2022
Nan
B
123
0.6
6
2022-01-01
Mortgage
RMBS
XYZ
2022
1
B
234
0.7
7
2022-01-01
Mortgage
RMBS
XYZ
2022
2
B
345
0.8
8
2022-01-01
Mortgage
RMBS
XYZ
2022
3
B
456
0.9
9
2022-01-01
Mortgage
RMBS
XYZ
2022
4
B
567
1.0
For each group (as_of_date, industry, sector, deal, year, stage), I need to display all the amounts and yields in one line
I have tried this -
df.groupby(['as_of_date', 'industry', 'sector', 'deal', 'year', 'stage'])['amount', 'yield' ].apply(lambda df: df.reset_index(drop=True)).unstack().reset_index()
but this is not working correctly.
Basically, I need this as output rows -
2022-01-01 Mortgage RMBS XYZ 2022 A 111 222 333 444 555 0.1 0.2 0.3 0.4 0.5
2022-01-01 Mortgage RMBS XYZ 2022 B 123 234 345 456 567 0.6 0.7 0.8 0.9 1.0
What would be the correct way to achieve this with Pandas? Thank you
This can be calculated by creating a list for each column first, then combined this (using +), and turning this into a string, removing the [, ], ,:
df1 = df.groupby(['as_of_date', 'industry', 'sector', 'deal', 'year', 'stage']).apply(
lambda x: str(list(x['amount']) + list(x['yield']))[1:-1].replace(",", ""))
df1
#Out:
#as_of_date industry sector deal year stage
#2022-01-01 Mortgage RMBS XYZ 2022 A 111 222 333 444 555 0.1 0.2 0.3 0.4 0.5
# B 123 234 345 456 567 0.6 0.7 0.8 0.9 1.0
Maybe this?
df.groupby(['as_of_date', 'industry', 'sector', 'deal', 'year', 'stage']).agg(' '.join).reset_index()
does this answer your question?
df2 = df.pivot(index=['as_of_date','industry','sector','deal','year', 'stage'], columns=['quarter']).reset_index()
to flatten the columns names
df2.columns = df2.columns.to_series().str.join('_')
df2
as_of_date_ industry_ sector_ deal_ year_ stage_ amount_1 amount_2 amount_3 amount_4 amount_NaN amount_Nan yield_1 yield_2 yield_3 yield_4 yield_NaN yield_Nan
0 2022-01-01 Mortgage RMBS XYZ 2022 A 222.0 333.0 444.0 555.0 111.0 NaN 0.2 0.3 0.4 0.5 0.1 NaN
1 2022-01-01 Mortgage RMBS XYZ 2022 B 234.0 345.0 456.0 567.0 NaN 123.0 0.7 0.8 0.9 1.0 NaN 0.6
I have a dataframe as following:
Company Date relTweet GaplastRel
XYZ 3/2/2020 1
XYZ 3/3/2020 1
XYZ 3/4/2020 1
XYZ 3/5/2020 1
XYZ 3/5/2020 0
XYZ 3/6/2020 1
XYZ 3/8/2020 1
ABC 3/9/2020 0
ABC 3/10/2020 1
ABC 3/11/2020 0
ABC 3/12/2020 1
The relTweet displays whether the tweet is relevant (1) or not (0).
\nI need to find the days difference (GaplastRel) between each successive rows for each company, with a condition that the previous day's tweet should be relevant tweet (i.e. relTweet =1 ). e.g. For the first record relTweet should be 0. For the 2nd record, relTweet should be 1 as the last relevant tweet was made one day ago.
Below is the example of needed output:
Company Date relTweet GaplastRel
XYZ 3/2/2020 1 0
XYZ 3/3/2020 1 1
XYZ 3/4/2020 1 1
XYZ 3/5/2020 1 1
XYZ 3/5/2020 0 1
XYZ 3/6/2020 1 1
XYZ 3/8/2020 1 2
ABC 3/9/2020 0 0
ABC 3/10/2020 1 0
ABC 3/11/2020 0 1
ABC 3/12/2020 1 2
Following is my code:
dataDf['Date'] = pd.to_datetime(dataDf['Date'], format='%m/%d/%Y')
dataDf['relTweet'] = (dataDf.groupby('Company', group_keys=False)
.apply(lambda g: g['Date'].diff().replace(0, np.nan).ffill()))
This code gives the days difference between successive rows for each company without conisidering the relTweet =1 condition. I am not sure how to apply the condition.
Following is the output of the above code:
Company Date relTweet GaplastRel
XYZ 3/2/2020 1 NaT
XYZ 3/3/2020 1 1 days
XYZ 3/4/2020 1 1 days
XYZ 3/5/2020 1 1 days
XYZ 3/5/2020 0 0 days
XYZ 3/6/2020 1 1 days
XYZ 3/8/2020 1 2 days
ABC 3/9/2020 0 NaT
ABC 3/10/2020 1 1 days
ABC 3/11/2020 0 1 days
ABC 3/12/2020 1 1 days
Change your mind sometime we need merge_asof rather than groupby
df1=df.loc[df['relTweet']==1,['Company','Date']]
df=pd.merge_asof(df,df1.assign(Date1=df1.Date),by='Company',on='Date', allow_exact_matches=False)
df['GaplastRel']=(df.Date-df.Date1).dt.days.fillna(0)
df
Out[31]:
Company Date relTweet Date1 GaplastRel
0 XYZ 2020-03-02 1 NaT 0.0
1 XYZ 2020-03-03 1 2020-03-02 1.0
2 XYZ 2020-03-04 1 2020-03-03 1.0
3 XYZ 2020-03-05 1 2020-03-04 1.0
4 XYZ 2020-03-05 0 2020-03-04 1.0
5 XYZ 2020-03-06 1 2020-03-05 1.0
6 XYZ 2020-03-08 1 2020-03-06 2.0
7 ABC 2020-03-09 0 NaT 0.0
8 ABC 2020-03-10 1 NaT 0.0
9 ABC 2020-03-11 0 2020-03-10 1.0
10 ABC 2020-03-12 1 2020-03-10 2.0
I am stuck in a case since 2 days and I need your assistance please.
the table as below :
Sample Table
SR UID Flag Fees
1 AAA 1 100.00
2 AAA 0 0.00
3 AAA 0 0.00
4 AAA 1 120.00
5 AAA 0 0.00
6 AAA 0 0.00
7 AAA 1 140.00
1 BBB 1 200.00
2 BBB 0 0.00
3 BBB 0 0.00
4 BBB 0 0.00
5 BBB 0 0.00
6 BBB 0 0.00
7 BBB 1 400.00
how I can use First_value function to replace the 0.00 values in fees column with the first value where Flag =1 partition by UID
the result should be as the following
Results
SR UID Flag Fees First_Value
1 AAA 1 100.00 100.00
2 AAA 0 0.00 100.00
3 AAA 0 0.00 100.00
4 AAA 1 120.00 120.00
5 AAA 0 0.00 120.00
6 AAA 0 0.00 120.00
7 AAA 1 140.00 140.00
1 BBB 1 200.00 200.00
2 BBB 0 0.00 200.00
3 BBB 0 0.00 200.00
4 BBB 0 0.00 200.00
5 BBB 0 0.00 200.00
6 BBB 0 0.00 200.00
7 BBB 1 400.00 400.00
One idea, using FIRST_VALUE like the Op was trying to:
CREATE TABLE #Sample (SR int, UID char(3), Flag bit, Fees decimal(5,2));
INSERT INTO #Sample
VALUES
(1,'AAA',1,100.00),
(2,'AAA',0,0.00 ),
(3,'AAA',0,0.00 ),
(4,'AAA',1,120.00),
(5,'AAA',0,0.00 ),
(6,'AAA',0,0.00 ),
(7,'AAA',1,140.00),
(1,'BBB',1,200.00),
(2,'BBB',0,0.00 ),
(3,'BBB',0,0.00 ),
(4,'BBB',0,0.00 ),
(5,'BBB',0,0.00 ),
(6,'BBB',0,0.00 ),
(7,'BBB',1,400.00);
WITH Groups AS(
SELECT *,
ROW_NUMBER() OVER (PARTITION BY UID ORDER BY SR) -
ROW_NUMBER() OVER (PARTITION BY UID, Flag ORDER BY SR) AS Grp
FROM #Sample)
SELECT SR, UID, Flag, Fees,
FIRST_VALUE(Fees) OVER (PARTITION BY UID, CASE Grp WHEN 0 THEN 1 ELSE Grp END ORDER BY SR)
FROM Groups
ORDER BY UID, SR;
DROP TABLE #Sample
I have a following situation
table1
PricelistID ValidTo Currency Markup
1 2013-12-31 USD 1.2
2 2013-12-31 USD 1.25
3 2013-12-31 USD 1.3
4 2013-12-31 USD 1.1
table2
PricelistID PeriodID Price
1 1 10.00
1 2 20.00
2 3 15.00
3 4 50.00
4 5 30.00
Now I have a situation where I need to add new pricelist with values(2013-12-31, USD, 1.5) (only in in table 1) that would inherit values of pricelistID 4 in table 2, how can I do it?
Query should return
PricelistiD ValidTo Currency Markup PeriodID Price
5 2013-12-31 USD 1.5 5 30.00