How to sum conditionally in pandas - pandas

Question

I am trying to populate a DataFrame using the result of a calculation performed on a different DataFrame.
These calculations should be run on a series, when conditions are met in two separate series.
Here is what I have tried.
I have built a dataframe, rswcapacity on which calculations should be run, then created another dataframe annualcapacity where I would like the conditional calculations to be stored.
#First DataFrame
d = {'technology': ['EAF', 'EAF', 'EAF', 'BOF', 'BOF', 'BOF'], 'equip_detail1': [150, 130, 100, 200, 200, 150], 'equip_number' : [1, 2, 3, 1, 2, 3], 'capacity_actual': [2400, 2080, 1600, 3200, 3200, 2400], 'start_year': [1992, 1993, 1994, 1989, 1990, 1991], 'closure_year': [ '', 2002, '', '', 2001, 2011] }
rswcapacity = pd.DataFrame(data = d)
rswcapacity['closure_year'].replace('', np.nan, inplace = True)
#Second DataFrame
annualcapacity = pd.DataFrame(columns=['years', 'capacity'])
annualcapacity ['years'] = [1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020]
#Neither of the attempts below yields the desired results:
for y in years:
annualcapacity['capacity'].append(rswcapacity['capacity_actual'].apply(lambda x : x['capacity_actual'].sum() (x['start_year'] >= y & (x['closure_year'] <= y | x['closure_year'].isnull()))).sum())
annualcapacity
#other attempt:
for y in years:
if (rswcapacity['start_year'] >= y).any() & ((rswcapacity['closure_year'].isnull()).any() | (rswcapacity['closure_year'] <= y).any()):
annualcapacity['capacity'].append(rswcapacity['capacity_actual'].sum())
annualcapacity
The result I would like to obtain is a sum performed for every year.
For instance:
1985 should return NaN as 1985 is smaller than any of the years in start_year 1992 should return 14880, as 1992 is larger than any start_year and smaller than any closure_year
2001 should return 7200, as it is larger than all start_year and larger of all closure_years.
Instead all three my attempts are only returning NaN across the list of years.
There is something wrong with my setting the conditions, but have not managed to work out what.
Any insight much appreciated!

Answer 1

You can do this as follows:
# start with an empty dataframe for the summed capacity
# with int32 as type of the year and float32 as type for the capacity
annualcapacity = pd.DataFrame({'years': pd.Series(dtype='int32'), 'capacity': pd.Series(dtype='float32')})
# use your list of years
years= [1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020]
for y in years:
# create a sum for each year
indexer= (rswcapacity['start_year'] <= y) & ((rswcapacity['closure_year'].isnull()) | (rswcapacity['closure_year'] >= y))
capa= rswcapacity.loc[indexer, 'capacity_actual'].sum()
# and append it to the result frame
annualcapacity= annualcapacity.append(dict(years=y, capacity=capa), ignore_index=True)
annualcapacity
The result looks like this:
years capacity
0 1980 0.0
1 1981 0.0
2 1982 0.0
3 1983 0.0
4 1984 0.0
5 1985 0.0
6 1986 0.0
7 1987 0.0
8 1988 0.0
9 1989 3200.0
10 1990 6400.0
11 1991 8800.0
12 1992 11200.0
13 1993 13280.0
14 1994 14880.0
15 1995 14880.0
16 1996 14880.0
17 1997 14880.0
18 1998 14880.0
19 1999 14880.0
20 2000 14880.0
21 2001 14880.0
22 2002 11680.0
23 2003 9600.0
24 2004 9600.0
25 2005 9600.0
26 2006 9600.0
27 2007 9600.0
28 2008 9600.0
29 2009 9600.0
30 2010 9600.0
31 2011 9600.0
32 2012 7200.0
33 2013 7200.0
34 2014 7200.0
35 2015 7200.0
36 2016 7200.0
37 2017 7200.0
38 2018 7200.0
39 2019 7200.0
40 2020 7200.0
Note: the sums are always numeric, so if there is no capacity for a year, the value is 0.0 instead of NaN. If you need NaN for some reason, you can replace it with the line below.
The second point is, that I switched your condition,
(rswcapacity['start_year'] >= y) & ((rswcapacity['closure_year'].isnull()) | (rswcapacity['closure_year'] <= y))
so >= became <= because I thought, you want to sum all capacities which were available for that year, right?
So if you need NaN entries instead of 0.0 if no capacity is available at all, you can do that as follows:
annualcapacity.loc[annualcapacity['capacity] == 0, 'capacity']= np.NaN
For this, you need to add import numpy as np in your header.