From the given table of inflation rates below, I want to obtain the countries with negative inflation rates for two consecutive years.
2017 2018 2019 2020 2021 2022
Country
Turkey NaN 47.0 -7.0 -19.0 38.0 260.0
Argentina NaN 33.0 56.0 -22.0 15.0 8.0
Suriname NaN -68.0 -37.0 695.0 56.0 13.0
Zimbabwe NaN 106.0 2306.0 118.0 -83.0 -21.0
Lebanon NaN 2.0 -36.0 2826.0 82.0 39.0
Sudan NaN 96.0 -19.0 220.0 19.0 34.0
Venezuela NaN 1482.0 -70.0 -88.0 15.0 -89.0
I have seen some solutions in SO that use list comprehension or loops. I wonder if this task is possible without them.
I attempted to convert the dataframe into 1s and 0s, in which 1.0 indicates a negative inflation.
2017 2018 2019 2020 2021 2022
Country
Turkey NaN 0.0 1.0 1.0 0.0 0.0
Argentina NaN 0.0 0.0 1.0 0.0 0.0
Suriname NaN 1.0 1.0 0.0 0.0 0.0
Zimbabwe NaN 0.0 0.0 0.0 1.0 1.0
Lebanon NaN 0.0 1.0 0.0 0.0 0.0
Sudan NaN 0.0 1.0 0.0 0.0 0.0
Venezuela NaN 0.0 1.0 1.0 0.0 1.0
However, I am stuck at this point. I tried to use np.prod function but this returns 0 if at least one column as 0.0 data.
Any ideas about how to solve this problem?
You can first set an integer mask for the negative values (1 means negative). Then compute a rolling min on the axis 1, of the min is 1 all values are. This is generalizable to any number of consecutive columns.
N = 2
m1 = df.lt(0).astype(int)
m2 = m.rolling(N, axis=1).min().eq(1).any(axis=1)
df[m2]
Output:
2017 2018 2019 2020 2021 2022
Country
Turkey NaN 47.0 -7.0 -19.0 38.0 260.0
Suriname NaN -68.0 -37.0 695.0 56.0 13.0
Zimbabwe NaN 106.0 2306.0 118.0 -83.0 -21.0
Venezuela NaN 1482.0 -70.0 -88.0 15.0 -89.0
NB. One needs to work with integers as rolling is currently limited to numeric types
Alternative with a single mask for N=2
m = df.lt(0)
df[(m&m.shift(axis=1)).any(axis=1)]
Try this:
match = (df.lt(0) & df.shift(axis=1).lt(0)).any(axis=1)
df[match]
How it works:
df.lt(0): current year inflation is less than 0
df.shift(axis=1).lt(0): previous year inflation is less than 0
.any(axis=1): any such occurrence in the country.
Given your dataframe, this is what would work for me:
set the Country as an index so I just have digits in my df values
Define new column for check of 'Two sequential negatives' in columns using df.shift(axis=1).
So it would look like:
df.set_index('Country',inplace=True)
df['TwoNegatives'] = ((df.values < 0) & ((df.shift(axis=1)).values <0)).any(axis=1)
Try with rolling
out = df[df.le(0).T.rolling(window=2).sum().ge(2).any()]
Out[15]:
2017 2018 2019 2020 2021 2022
Country
Turkey NaN 47.0 -7.0 -19.0 38.0 260.0
Suriname NaN -68.0 -37.0 695.0 56.0 13.0
Zimbabwe NaN 106.0 2306.0 118.0 -83.0 -21.0
Venezuela NaN 1482.0 -70.0 -88.0 15.0 -89.0
def function1(ss:pd.Series):
ss.loc['col1']=ss.rolling(2).apply(lambda ss1:ss1.iloc[0]<0 and ss1.iloc[1]<0).eq(1).any()
return ss
df1.set_index('Country').apply(function1,axis=1).query('col1')
out
2017 2018 2019 2020 2021 2022 col1
Country
Turkey NaN 47.0 -7.0 -19.0 38.0 260.0 True
Suriname NaN -68.0 -37.0 695.0 56.0 13.0 True
Zimbabwe NaN 106.0 2306.0 118.0 -83.0 -21.0 True
Venezuela NaN 1482.0 -70.0 -88.0 15.0 -89.0 True
Related
Under the variable names, there is an extra row that I do not want in my data set
fdi_autocracy = fdi_autocracy.pivot(index=["Country", "regime", "Year"],
columns="partner_regime",
values =['FDI_outward', "FDI_inward", "total_fdi"],
).reset_index()
Country regime Year FDI_outward FDI_inward total_fdi
partner_regime 0.0 0.0 0.0
0 Albania 0.0 1995 NaN NaN NaN
1 Albania 0.0 1996 NaN NaN NaN
2 Albania 0.0 1997 NaN NaN NaN
3 Albania 0.0 1998 NaN NaN NaN
4 Albania 0.0 1999 NaN NaN NaN
What I want is following:
Country regime Year FDI_outward FDI_inward total_fdi
0 Albania 0.0 1995 NaN NaN NaN
1 Albania 0.0 1996 NaN NaN NaN
2 Albania 0.0 1997 NaN NaN NaN
3 Albania 0.0 1998 NaN NaN NaN
4 Albania 0.0 1999 NaN NaN NaN
IIUC, you don't need the partner_regime?
this removes that title
fdi_autocracy.rename_axis(columns=[None, None])
I have this df:
CODE DATE TMAX TMIN PP
0 000130 1991-01-01 NaN NaN 0.0
1 000130 1991-01-02 31.2 NaN 0.0
2 000130 1991-01-03 32.0 21.2 0.0
3 000130 1991-01-04 NaN NaN 0.0
4 000130 1991-01-05 NaN 22.0 0.0
... ... ... ... ...
34995 000135 1997-04-24 NaN NaN 0.0
34996 000135 1997-04-25 NaN NaN 4.0
34997 000135 1997-04-26 NaN 22.1 0.0
34998 000135 1997-04-27 31.0 NaN 5.0
34999 000135 1997-04-28 28.8 24.0 0.0
I'm counting the NaN values by CODE column, in columns TMAX TMIN and PP. So i'm using this code.
dfna=df[['TMAX','TMIN','PP']].isna().groupby(df.CODE).sum()
But i want to start counting NaN values since the first non NaN value.
Expected df:
CODE TMAX TMIN PP
000130 2 1 0
000135 0 1 0
...
...
How can i do this?
Thanks in advance.
Think in term of the whole frame, you can use ffill to remove the later nan values. So you can use this to detect the nan's that come after the first valid values:
df.isna() & df.ffill().notna()
Now, you can try groupby.apply
(df[['TMAX','TMIN','PP']].groupby(df['CODE'])
.apply(lambda d: (d.isna() & d.ffill().notna()).sum())
)
Output:
TMAX TMIN PP
CODE
130 2 1 0
135 0 1 0
I am not entirely new to data science, but rather novice with pandas.
My data looks like this:
Date Obser_Type
0 2001-01-05 A
1 2002-02-06 A
2 2002-02-06 B
3 2004-03-07 C
4 2005-04-08 B
5 2006-05-09 A
6 2007-06-10 C
7 2007-07-11 B
I would like to get the following output with the proportions for the different kinds of observations as of total (i.e. accumulated from the beginning up to and including the specified year) and within each year:
Year A_%_total B_%_total C_%_total A_%_Year B_%_Year C_%_Year
0 2001 100 0 0 100 0 0
1 2002 67 33 0 50 50 0
2 2004 50 25 25 0 0 100
3 2005 40 40 20 0 100 0
4 2006 50 33 17 100 0 0
5 2007 37,5 37,5 25 0 50 50
I tried various approaches involving groupby, multiindexing, count etc but to no avail. I got either errors or something unsatisfying.
After extensively digging Stack Overflow and the rest of the internet for days, I am stumped.
The medieval way would be a bucket of loops and ifs, but what is the proper way to do this?
I have used appropriate values for the numbers. I don't know the aggregation logic of each of them, but I decided to create a composition ratio for 'Obser_Type' and a composition ratio for 'year'.
Add a new column for year data
2.Aggregate and create DF
3.Creating the Composition Ratio
4.Aggregate and create DF
5.Creating the Composition Ratio
6.Combining the two DF's
import pandas as pd
import numpy as np
import io
data = '''
Date Obser_Type Value
0 2001-01-05 A 34
1 2002-02-06 A 39
2 2002-02-06 B 67
3 2004-03-07 C 20
4 2005-04-08 B 29
5 2006-05-09 A 10
6 2007-06-10 C 59
7 2007-07-11 B 43
'''
df = pd.read_csv(io.StringIO(data), sep=' ')
df['Date'] = pd.to_datetime(df['Date'])
df['yyyy'] = df['Date'].dt.year
df1 = df.groupby(['yyyy','Obser_Type'])['Value'].agg(sum).unstack().fillna(0)
df1 = df1.apply(lambda x: x/sum(x), axis=0).rename(columns={'A':'A_%_total','B':'B_%_total','C':'C_%_total'})
df2 = df.groupby(['Obser_Type','yyyy'])['Value'].agg(sum).unstack().fillna(0)
df2 = df2.apply(lambda x: x/sum(x), axis=0)
df2 = df2.unstack().unstack().rename(columns={'A':'A_%_Year','B':'B_%_Year','C':'C_%_Year'})
pd.merge(df1, df2, on='yyyy')
Obser_Type A_%_total B_%_total C_%_total A_%_Year B_%_Year C_%_Year
yyyy
2001 0.409639 0.000000 0.000000 1.000000 0.000000 0.000000
2002 0.469880 0.482014 0.000000 0.367925 0.632075 0.000000
2004 0.000000 0.000000 0.253165 0.000000 0.000000 1.000000
2005 0.000000 0.208633 0.000000 0.000000 1.000000 0.000000
2006 0.120482 0.000000 0.000000 1.000000 0.000000 0.000000
2007 0.000000 0.309353 0.746835 0.000000 0.421569 0.578431
Thank you very much for your answer. However, i probably should have made it more clear that the actual dataframe is much bigger and has much more types of observations than A B C, so listing them manually would be inconvenient. My scope here is just the statistics for the different types of observations, not their associated numerical values.
I was able to build something and would like to share:
# convert dates to datetimes
#
df[‚Date'] = pd.to_datetime(df[‚Date'])
# get years from the dates
#
df[‚Year'] = df.Date.dt.year
# get total number of observations per type of observation and year in tabular form
#
grouped = df.groupby(['Year', 'Obser_Type']).count().unstack(1)
Date
Obser_Type A B C
Year
2001 1.0 NaN NaN
2002 1.0 1.0 NaN
2004 NaN NaN 1.0
2005 NaN 1.0 NaN
2006 1.0 NaN NaN
2007 NaN 1.0 1.0
# sum total number of observations per type over all years
#
grouped.loc['Total_Obs_per_Type',:] = grouped.sum(axis=0)
Date
Obser_Type A B C
Year
2001 1.0 NaN NaN
2002 1.0 1.0 NaN
2004 NaN NaN 1.0
2005 NaN 1.0 NaN
2006 1.0 NaN NaN
2007 NaN 1.0 1.0
Total_Obs_per_Type 3.0 3.0 2.0
# at this point the columns have a multiindex
#
grouped.columns
MultiIndex([('Date', 'A'),
('Date', 'B'),
('Date', 'C')],
names=[None, 'Obser_Type'])
# i only needed the second layer which looks like this
#
grouped.columns.get_level_values(1)
Index(['A', 'B', 'C'], dtype='object', name='Obser_Type')
# so i flattened the index
#
grouped.columns = grouped.columns.get_level_values(1)
# now i can easily address the columns
#
grouped.columns
Index(['A', 'B', 'C'], dtype='object', name='Obser_Type')
# create list of columns with observation types
# this refers to columns "A B C"
#
types_list = grouped.columns.values.tolist()
# create list to later access the columns with the cumulative sum of observations per type
# this refers to columns "A_cum B_cum C_cum"
#
types_cum_list = []
# calculate cumulative sum for the different kinds of observations
#
for columnName in types_list:
# create new columns with modified name and calculate for each type of observation the cumulative sum of observations
#
grouped[columnName+'_cum'] = grouped[columnName].cumsum()
# put the new column names in the list of columns with cumulative sum of observations per type
#
types_cum_list.append(columnName+'_cum')
# this gives
Obser_Type A B C A_cum B_cum C_cum
Year
2001 1.0 NaN NaN 1.0 NaN NaN
2002 1.0 1.0 NaN 2.0 1.0 NaN
2004 NaN NaN 1.0 NaN NaN 1.0
2005 NaN 1.0 NaN NaN 2.0 NaN
2006 1.0 NaN NaN 3.0 NaN NaN
2007 NaN 1.0 1.0 NaN 3.0 2.0
Total_Obs_per_Type 3.0 3.0 2.0 6.0 6.0 4.0
# create new column with total number of observations for all types of observation within a single year
#
grouped['All_Obs_Y'] = grouped.loc[:,types_list].sum(axis=1)
# this gives
Obser_Type A B C A_cum B_cum C_cum All_Obs_Y
Year
2001 1.0 NaN NaN 1.0 NaN NaN 1.0
2002 1.0 1.0 NaN 2.0 1.0 NaN 2.0
2004 NaN NaN 1.0 NaN NaN 1.0 1.0
2005 NaN 1.0 NaN NaN 2.0 NaN 1.0
2006 1.0 NaN NaN 3.0 NaN NaN 1.0
2007 NaN 1.0 1.0 NaN 3.0 2.0 2.0
Total_Obs_per_Type 3.0 3.0 2.0 6.0 6.0 4.0 8.0
# create new columns with cumulative sum of all kinds observations up to each year
#
grouped['All_Obs_Cum'] = grouped['All_Obs_Y'].cumsum()
# this gives
# sorry i could not work out the formatting and i am not allowed yet to include screenshots
Obser_Type A B C A_cum B_cum C_cum All_Obs_Y All_Obs_Cum
Year
2001 1.0 NaN NaN 1.0 NaN NaN 1.0 1.0
2002 1.0 1.0 NaN 2.0 1.0 NaN 2.0 3.0
2004 NaN NaN 1.0 NaN NaN 1.0 1.0 4.0
2005 NaN 1.0 NaN NaN 2.0 NaN 1.0 5.0
2006 1.0 NaN NaN 3.0 NaN NaN 1.0 6.0
2007 NaN 1.0 1.0 NaN 3.0 2.0 2.0 8.0
Total_Obs_per_Type 3.0 3.0 2.0 6.0 6.0 4.0 8.0 16.0
# create list of columns with the percentages each type of observation has within the observations of each year
# this refers to columns "A_%_Y B_%_Y C_Y_%"
#
types_percent_Y_list = []
# calculate the percentages each type of observation has within each year
#
for columnName in types_list:
# calculate percentages
#
grouped[columnName+'_%_Y'] = grouped[columnName] / grouped['All_Obs_Y']
# put the new columns names in list of columns with percentages each type of observation has within a year for later access
#
types_percent_Y_list.append(columnName+'_%_Y')
# this gives
Obser_Type A B C A_cum B_cum C_cum All_Obs_Y All_Obs_Cum A_%_Y B_%_Y C_%_Y
Year
2001 1.0 NaN NaN 1.0 NaN NaN 1.0 1.0 1.000 NaN NaN
2002 1.0 1.0 NaN 2.0 1.0 NaN 2.0 3.0 0.500 0.500 NaN
2004 NaN NaN 1.0 NaN NaN 1.0 1.0 4.0 NaN NaN 1.00
2005 NaN 1.0 NaN NaN 2.0 NaN 1.0 5.0 NaN 1.000 NaN
2006 1.0 NaN NaN 3.0 NaN NaN 1.0 6.0 1.000 NaN NaN
2007 NaN 1.0 1.0 NaN 3.0 2.0 2.0 8.0 NaN 0.500 0.50
Total_Obs_per_Type 3.0 3.0 2.0 6.0 6.0 4.0 8.0 16.0 0.375 0.375 0.25
# replace the NaNs in the types_cum columns, otherwise the calculation of the cumulative percentages in the next step would not work
#
# types_cum_list :
# if there is no observation for e.g. type B in the first year (2001) we put a count of 0 for that year,
# that is, in the first row.
# If there is no observation for type B in a later year (e.g. 2004) the cumulative count of Bs
# from the beginning up to that year does not change in that year, so we replace the NaN there with
# the last non-NaN value preceding it
#
# replace NaNs in first row by 0
#
for columnName in types_cum_list:
grouped.update(grouped.iloc[:1][columnName].fillna(value=0))
# replace NaNs in later rows with preceding non-NaN value
#
for columnName in types_cum_list:
grouped[columnName].fillna(method='ffill' , inplace=True)
# this gives
Obser_Type A B C A_cum B_cum C_cum All_Obs_Y All_Obs_Cum A_%_Y B_%_Y C_%_Y
Year
2001 1.0 NaN NaN 1.0 0.0 0.0 1.0 1.0 1.000 NaN NaN
2002 1.0 1.0 NaN 2.0 1.0 0.0 2.0 3.0 0.500 0.500 NaN
2004 NaN NaN 1.0 2.0 1.0 1.0 1.0 4.0 NaN NaN 1.00
2005 NaN 1.0 NaN 2.0 2.0 1.0 1.0 5.0 NaN 1.000 NaN
2006 1.0 NaN NaN 3.0 2.0 1.0 1.0 6.0 1.000 NaN NaN
2007 NaN 1.0 1.0 3.0 3.0 2.0 2.0 8.0 NaN 0.500 0.50
Total_Obs_per_Type 3.0 3.0 2.0 6.0 6.0 4.0 8.0 16.0 0.375 0.375 0.25
# create list of the columns with the cumulative percentages of the different observation types from the beginning up to that year
# this refers to columns "A_cum_% B_cum_% C_cum_%"
#
types_cum_percent_list = []
# calculate cumulative proportions of different types of observations from beginning up to each year
#
for columnName in types_cum_list:
# if we had not taken care of the NaNs in the types_cum columns this would produce incorrect numbers
#
grouped[columnName+'_%'] = grouped[columnName] / grouped['All_Obs_Cum']
# put the new columns in their respective list so we can access them conveniently later
#
types_cum_percent_list.append(columnName+'_%')
# this gives
Obser_Type A B C A_cum B_cum C_cum All_Obs_Y All_Obs_Cum A_%_Y B_%_Y C_%_Y A_cum_% B_cum_% C_cum_%
Year
2001 1.0 NaN NaN 1.0 0.0 0.0 1.0 1.0 1.000 NaN NaN 1.000000 0.000000 0.000000
2002 1.0 1.0 NaN 2.0 1.0 0.0 2.0 3.0 0.500 0.500 NaN 0.666667 0.333333 0.000000
2004 NaN NaN 1.0 2.0 1.0 1.0 1.0 4.0 NaN NaN 1.00 0.500000 0.250000 0.250000
2005 NaN 1.0 NaN 2.0 2.0 1.0 1.0 5.0 NaN 1.000 NaN 0.400000 0.400000 0.200000
2006 1.0 NaN NaN 3.0 2.0 1.0 1.0 6.0 1.000 NaN NaN 0.500000 0.333333 0.166667
2007 NaN 1.0 1.0 3.0 3.0 2.0 2.0 8.0 NaN 0.500 0.50 0.375000 0.375000 0.250000
Total_Obs_per_Type 3.0 3.0 2.0 6.0 6.0 4.0 8.0 16.0 0.375 0.375 0.25 0.375000 0.375000 0.250000
# to conclude i replace the remaining NaNs to make plotting easier
# replace NaNs in columns in types_list
#
# if there is no observation for a type of observation in a year we put a count of 0 for that year
#
for columnName in types_list:
grouped[columnName].fillna(value=0, inplace=True)
# replace NaNs in columns in types_percent_Y_list
#
# if there is no observation for a type of observation in a year we put a percentage of 0 for that year
#
for columnName in types_percent_Y_list:
grouped[columnName].fillna(value=0, inplace=True)
Obser_Type A B C A_cum B_cum C_cum All_Obs_Y All_Obs_Cum A_%_Y B_%_Y C_%_Y A_cum_% B_cum_% C_cum_%
Year
2001 1.0 0.0 0.0 1.0 0.0 0.0 1.0 1.0 1.000 0.000 0.00 1.000000 0.000000 0.000000
2002 1.0 1.0 0.0 2.0 1.0 0.0 2.0 3.0 0.500 0.500 0.00 0.666667 0.333333 0.000000
2004 0.0 0.0 1.0 2.0 1.0 1.0 1.0 4.0 0.000 0.000 1.00 0.500000 0.250000 0.250000
2005 0.0 1.0 0.0 2.0 2.0 1.0 1.0 5.0 0.000 1.000 0.00 0.400000 0.400000 0.200000
2006 1.0 0.0 0.0 3.0 2.0 1.0 1.0 6.0 1.000 0.000 0.00 0.500000 0.333333 0.166667
2007 0.0 1.0 1.0 3.0 3.0 2.0 2.0 8.0 0.000 0.500 0.50 0.375000 0.375000 0.250000
Total_Obs_per_Type 3.0 3.0 2.0 6.0 6.0 4.0 8.0 16.0 0.375 0.375 0.25 0.375000 0.375000 0.250000
This has the functionylity and flexibility i was looking for. But as i am still learning pandas suggestions for improvement are appreciated.
I have a panel data set/time series. I want to prepare the dataset for machine learning prediction next year's gcp. My data looks like this:
ID,year,age,area,debt_ratio,gcp
654001,2013,49,East,0.14,0
654001,2014,50,East,0.17,0
654001,2015,51,East,0.23,1
654001,2016,52,East,0.18,0
112089,2013,39,West,0.13,0
112089,2014,40,West,0.15,0
112089,2015,41,West,0.18,1
112089,2016,42,West,0.21,1
What I want is something like this:
ID,year,age,area,debt_ratio,gcp,gcp-1,gcp-2,gcp-3
654001,2013,49,East,0.14,0,NA,NA,NA
654001,2014,50,East,0.17,0,0,NA,NA
654001,2015,51,East,0.23,1,0,0,NA
654001,2016,52,East,0.18,0,1,0,0
112089,2013,39,West,0.13,0,NA,NA,NA
112089,2014,40,West,0.15,0,0,NA,NA
112089,2015,41,West,0.18,1,0,0,NA
112089,2016,42,West,0.21,1,1,0,0
I've tried Pandas melt function, but it didn't work out. I searched online and found this post that is exact what I want to do, but it is done in R:
https://stackoverflow.com/questions/19813077/prepare-time-series-for-machine-learning-long-to-wide-format
Does anybody know how to do this in Python Pandas? Any suggestion would be appreciated!
Use DataFrameGroupBy.shift in loop:
for i in range(1, 4):
df[f'gcp-{i}'] = df.groupby('ID')['gcp'].shift(i)
print (df)
ID year age area debt_ratio gcp gcp-1 gcp-2 gcp-3
0 654001 2013 49 East 0.14 0 NaN NaN NaN
1 654001 2014 50 East 0.17 0 0.0 NaN NaN
2 654001 2015 51 East 0.23 1 0.0 0.0 NaN
3 654001 2016 52 East 0.18 0 1.0 0.0 0.0
4 112089 2013 39 West 0.13 0 NaN NaN NaN
5 112089 2014 40 West 0.15 0 0.0 NaN NaN
6 112089 2015 41 West 0.18 1 0.0 0.0 NaN
7 112089 2016 42 West 0.21 1 1.0 0.0 0.0
More dynamic solution is get maximum number of groups and pass to range:
N = df['ID'].value_counts().max()
for i in range(1, N):
df[f'gcp-{i}'] = df.groupby('ID')['gcp'].shift(i)
print (df)
ID year age area debt_ratio gcp gcp-1 gcp-2 gcp-3
0 654001 2013 49 East 0.14 0 NaN NaN NaN
1 654001 2014 50 East 0.17 0 0.0 NaN NaN
2 654001 2015 51 East 0.23 1 0.0 0.0 NaN
3 654001 2016 52 East 0.18 0 1.0 0.0 0.0
4 112089 2013 39 West 0.13 0 NaN NaN NaN
5 112089 2014 40 West 0.15 0 0.0 NaN NaN
6 112089 2015 41 West 0.18 1 0.0 0.0 NaN
7 112089 2016 42 West 0.21 1 1.0 0.0 0.0
This is the dataframe:
bins year binA binB binC binD binE binF binG binH
0 1998 4.0 5.0 1.0 1.0 2.0 0.0 1.0 0.0
1 1999 4.0 2.0 1.0 0.0 0.0 4.0 1.0 2.0
2 2000 4.0 1.0 1.0 0.0 4.0 1.0 1.0 2.0
3 2001 2.0 1.0 4.0 1.0 1.0 0.0 2.0 3.0
My goal is to divide binA through binH by sum of binA:binH or for row 1998, divide by the sum of the row excluding the year number.
Sum of desired columns:
newdfdd.loc[:,'binA':'binH'].sum(axis=1)
To get the desired value this is what I have tried:
newdfdd[['binA','binB','binC','binD','binE',
'binF','binG' ,'binH']].div(newdfdd.loc[:,'binA':'binH'].sum(axis=1))
But, I get NaN and four extra columns as following:
0 1 2 3 binA binB binC binD binE binF binG binH
0 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
1 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
2 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
3 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
I want results in the following format:
bins year binA binB binC binD binE binF binG binH
0 1998 0.285 0.357 ... .... .... .... ... ...
1 1999 .. .. .. .. .. .. .. ..
.... means some number from calculation.
What do I need to edit in my code for the desired output?
In the div statement you need to provide the axis='index' and it should get the result you are looking for.
So your code above should look like:
newdfdd.update(newdfdd.loc[:,'binA':'binH'].div(newdfdd.loc[:,'binA':'binH'].sum(axis=1),
axis='index'))
This will compute your percent of row sum as desired and then update the values inplace in the newfdd dataframe.
Here is the entirety of my solution for clarity (I used df, and random variables, but the rest is the same):
df = pd.DataFrame({'bins':[0,1,2,3],
'year':[1998,1999,2000,2001],
'binA':np.random.randint(1,10,4),
'binB':np.random.randint(1,10,4),
'binC':np.random.randint(1,10,4),
'binD':np.random.randint(1,10,4),
'binE':np.random.randint(1,10,4),
'binF':np.random.randint(1,10,4),
'binG':np.random.randint(1,10,4),
'binH':np.random.randint(1,10,4)})
#reodering columns to match your dataframe layout
df = df[['bins','year','binA','binB','binC','binD','binE',
'binF','binG' ,'binH']]
df.update(df.loc[:,'binA':'binH'].div(df.loc[:,'binA':'binH'].sum(axis=1),axis='index'))
print(df)
bins year binA binB binC binD binE binF binG binH
0 0 1998 0.222222 0.037037 0.148148 0.185185 0.037037 0.111111 0.037037 0.222222
1 1 1999 0.264706 0.058824 0.205882 0.058824 0.029412 0.147059 0.176471 0.058824
2 2 2000 0.166667 0.041667 0.145833 0.020833 0.166667 0.166667 0.145833 0.145833
3 3 2001 0.062500 0.187500 0.020833 0.145833 0.083333 0.166667 0.166667 0.166667
I think this is the result you are looking for:
df['rowSum'] = df[df.columns[2:]].apply(sum, axis=1)
df[df.columns[2:]].apply(lambda x: (x / x['rowSum']), axis=1).drop(columns=['rowSum'])
binA binB binC binD binE binF binG binH
0 0.285714 0.357143 0.071429 0.071429 0.142857 0.000000 0.071429 0.000000
1 0.285714 0.142857 0.071429 0.000000 0.000000 0.285714 0.071429 0.142857
2 0.285714 0.071429 0.071429 0.000000 0.285714 0.071429 0.071429 0.142857
3 0.142857 0.071429 0.285714 0.071429 0.071429 0.000000 0.142857 0.214286