Why doesn't .loc reverse slice correctly? - pandas

From my understanding, there are two ways to subset a dataframe in pandas:
a) df['columns']['rows']
b) df.loc['rows', 'columns']
I was following a guided case study, where the instruction was to select the first and last n rows of a column in a dataframe. The solution used Method A, whereas I tried Method B.
My method wasn't working and I couldn't for the life of me figure out why.
I've created a simplified version of the dataframe...
male = [6, 14, 12, 13, 21, 14, 14, 14, 14, 18]
female = [9, 11, 6, 10, 11, 13, 12, 11, 9, 11]
df = pd.DataFrame({'Male': male,
'Female': female},
index = np.arange(1, 11))
df['Mean'] = df[['Male', 'Female']].mean(axis = 1).round(1)
df
Selecting the first two rows, works fine for method a and b
print('Method A: \n', df['Mean'][:2])
print('Method B: \n', df.loc[:2, 'Mean'])
Method A:
1 7.5
2 12.5
Method B:
1 7.5
2 12.5
But not for selecting the last 2 rows, it doesn't work the same. Method A returns the last two rows as it should.
Method B (.loc) doesn't, it returns the whole dataframe. Why is this and how do I fix it?
print('Method A: \n', df['Mean'][-2:])
print('Method B: \n', df.loc[-2:, 'Mean'])
Method A:
9 11.5
10 14.5
Method B:
1 7.5
2 12.5
3 9.0
4 11.5
5 16.0
6 13.5
7 13.0
8 12.5
9 11.5
10 14.5

You could use .index[-2:] to get the index of the lasts two rows which are 9 and 10 instead of only -2:. Here is some reproducible code:
male = [6, 14, 12, 13, 21, 14, 14, 14, 14, 18]
female = [9, 11, 6, 10, 11, 13, 12, 11, 9, 11]
df = pd.DataFrame({'Male': male,
'Female': female},
index = np.arange(1, 11))
df['Mean'] = df[['Male', 'Female']].mean(axis = 1).round(1)
print('Method B: \n', df.loc[df.index[-2:], 'Mean'])
Output:
Method B:
9 11.5
10 14.5
Name: Mean, dtype: float64
As you can see it returns the two last rows of your dataframe.

Also you can get with iloc and tail method, like that :
df['Mean'][-2:]
df['Mean'].iloc[-2:]
df['Mean'].tail(2)
We don't usually use loc for this. iloc or other methods are easier to use. But if you want to use it could be like this:
df.loc[df.index[-2:],'Mean']

Related

Iteration between rows and columsn of a DataFrame to calculate the mean

I have a dataframe which reads:
A 2007/Ago 2007/Set 2007/Out ... 2020/Jan 2020/Fev
row1 x number number number ... number number
row2 y number number number ... number number
row3 w number number number ... number number
...
row27 z number number number ... number number
I mean, there are numbers in each cell. I want to calculate the mean of the cells for which the columns starts with 2007, and then calculate the mean of the cells of which the columns starts with 2008, and then 2009, ..., and then 2020 and do this for each row.
What I tried to sketch is something like:
x = []
for i in df.row(i): #that is, for each row of the dataframe
if column.startswith('j'): #which starts with j=2008, 2009, 2010 etc
x += df[i][j] #the variable x gets the number on that row i,column j and sum
What I want in the end are various columsn with the results of the mean for each year, that is, I want
result1 result2 result3 ... resultn
mean colums mean colums mean columsn mean columsn
starts starts starts starts
with 2008 with 2009 with 2010 with 2020
That is, I want 13 new columns: one for each mean (years from 2008 to 2020).
I can't continue this loop and I do not know how much basic this is, but my questions are:
1- Are there any more optimal way of doing this? I mean, using pandas functions other than loops?
In my dataframe, each cell corresponds to the total cost of health expends in that month, and I want to take the mean of the cost of the entire year to compare it to the population of each city (which are thw rows). I am struggling with this for some time and I am not able to solve it. My level using pandas is very basic.
PS: sorry for the dataframe representation, I do not know how to properly write one in the stackoverflow's body question.
An option via melt + pivot_table with the aggfunc set to mean:
import pandas as pd
df = pd.DataFrame({
'A': {'row1': 'x', 'row2': 'y', 'row3': 'w', 'row27': 'z'},
'2007/Ago': {'row1': 1, 'row2': 2, 'row3': 3, 'row27': 4},
'2007/Set': {'row1': 5, 'row2': 6, 'row3': 7, 'row27': 8},
'2007/Out': {'row1': 9, 'row2': 10, 'row3': 11, 'row27': 12},
'2020/Jan': {'row1': 13, 'row2': 14, 'row3': 15, 'row27': 16},
'2020/Fev': {'row1': 17, 'row2': 18, 'row3': 19, 'row27': 20}
})
df = df.melt(id_vars='A', var_name='year')
# Rename month columns to their year value
df['year'] = df['year'].str.split('/').str[0]
# pivot to wide format based on the new year value
df = (
df.pivot_table(columns='year', index='A', aggfunc='mean')
.droplevel(0, 1)
.rename_axis(None)
.rename_axis(None, axis=1)
)
print(df)
df:
2007 2020
w 7 17
x 5 15
y 6 16
z 8 18
Suppose you have this dataframe:
A 2007/Ago 2007/Set 2007/Out 2020/Jan 2020/Fev
row1 x 1 5 9 13 17
row2 y 2 6 10 14 18
row3 w 3 7 11 15 19
row27 z 4 8 12 16 20
You can use .filter() and .mean(axis=1) to compute the values:
df["result"] = df.filter(regex=r"^\d{4}").mean(axis=1)
print(df)
Prints:
A 2007/Ago 2007/Set 2007/Out 2020/Jan 2020/Fev result
row1 x 1 5 9 13 17 9.0
row2 y 2 6 10 14 18 10.0
row3 w 3 7 11 15 19 11.0
row27 z 4 8 12 16 20 12.0
While re-working my other answer, I found this one-liner:
df.mean().groupby(lambda x: x[:4]).mean()
Explanation
Pandas' `mean' function calculates the mean per column:
# using the DataFrame from Henry's answer:
df = pd.DataFrame({
'A': {'row1': 'x', 'row2': 'y', 'row3': 'w', 'row27': 'z'},
'2007/Ago': {'row1': 1, 'row2': 2, 'row3': 3, 'row27': 4},
'2007/Set': {'row1': 5, 'row2': 6, 'row3': 7, 'row27': 8},
'2007/Out': {'row1': 9, 'row2': 10, 'row3': 11, 'row27': 12},
'2020/Jan': {'row1': 13, 'row2': 14, 'row3': 15, 'row27': 16},
'2020/Fev': {'row1': 17, 'row2': 18, 'row3': 19, 'row27': 20}
})
# calculate mean per column
col_means = df.mean()
# 2007/Ago 2.5
# 2007/Set 6.5
# 2007/Out 10.5
# 2020/Jan 14.5
# 2020/Fev 18.5
# dtype: float64
# group above columns by first 4 characters, i.e., the year
year_groups = col_means.groupby(lambda x: x[:4])
# calculate the mean per year group
year_groups.mean()
# 2007 6.5
# 2020 16.5
# dtype: float64
You could iterate over the years, select the subset of columns and just use pandas' mean() function to get the mean of that year:
means = {}
for year in range(2007, 2021):
# assuming df is your dataframe
sub_df = df.loc[:, df.columns.str.startswith(str(year))]
# first mean() aggregates per column, second mean() aggregates the whoƶe year
means[year] = sub_df.mean().mean()
This yields a dict with the years as key and the mean for that year as value. If there are no columns for one year, means[year] contains NaN.

How do I aggregate a pandas Dataframe while retaining all original data?

My goal is to aggregate a pandas DataFrame, grouping rows by an identity field. Notably, rather than just gathering summary statistics of the group, I want to retain all the information in the DataFrame in addition to summary statistics like mean, std, etc. I have performed this transformation via a lot of iteration, but I am looking for a cleaner/more pythonic approach. Notably, there may be more or less than 2 replicates per group, but all groups will always have the same number of replicates.
Example: I would llke to translate the below format
df = pd.DataFrame([
["group1", 4, 10],
["group1", 8, 20],
["group2", 6, 30],
["group2", 12, 40],
["group3", 1, 50],
["group3", 3, 60]],
columns=['group','timeA', 'timeB'])
print(df)
group timeA timeB
0 group1 4 10
1 group1 8 20
2 group2 6 30
3 group2 12 40
4 group3 1 50
5 group3 3 60
into a df of the following format:
target = pd.DataFrame([
["group1", 4, 8, 6, 10, 20, 15],
["group2", 6, 12, 9, 30, 45, 35],
["group3", 1, 3, 2, 50, 60, 55]
], columns = ["group", "timeA.1", "timeA.2", "timeA.mean", "timeB.1", "timeB.2", "timeB.mean"])
print(target)
group timeA.1 timeA.2 timeA.mean timeB.1 timeB.2 timeB.mean
0 group1 4 8 6 10 20 15
1 group2 6 12 9 30 45 35
2 group3 1 3 2 50 60 55
Finally, it doesn't really matter what the column names are, these ones are just to make the example more clear. Thanks!
EDIT: As suggested by a user in the comments, I tried the solution from the linked Q/A without success:
df.insert(0, 'count', df.groupby('group').cumcount())
df.pivot(*df)
TypeError: pivot() takes from 1 to 4 positional arguments but 5 were given
Try with pivot_table:
out = (df.assign(col=df.groupby('group').cumcount()+1)
.pivot_table(index='group', columns='col',
margins='mean', margins_name='mean')
.drop('mean')
)
out.columns = [f'{x}.{y}' for x,y in out.columns]
Output:
timeA.1 timeA.2 timeA.mean timeB.1 timeB.2 timeB.mean
group
group1 4.0 8.0 6.0 10 20 15
group2 6.0 12.0 9.0 30 40 35
group3 1.0 3.0 2.0 50 60 55

Reshaping column values into rows with Identifier column at the end

I have measurements for Power related to different sensors i.e A1_Pin, A2_Pin and so on. These measurements are recorded in file as columns. The data is uniquely recorded with timestamps.
df1 = pd.DataFrame({'DateTime': ['12/12/2019', '12/13/2019', '12/14/2019',
'12/15/2019', '12/16/2019'],
'A1_Pin': [2, 8, 8, 3, 9],
'A2_Pin': [1, 2, 3, 4, 5],
'A3_Pin': [85, 36, 78, 32, 75]})
I want to reform the table so that each row corresponds to one sensor. The last column indicates the sensor ID to which the row data belongs to.
The final table should look like:
df2 = pd.DataFrame({'DateTime': ['12/12/2019', '12/12/2019', '12/12/2019',
'12/13/2019', '12/13/2019','12/13/2019', '12/14/2019', '12/14/2019',
'12/14/2019', '12/15/2019','12/15/2019', '12/15/2019', '12/16/2019',
'12/16/2019', '12/16/2019'],
'Power': [2, 1, 85,8, 2, 36, 8,3,78, 3, 4, 32, 9, 5, 75],
'ModID': ['A1_PiN','A2_PiN','A3_PiN','A1_PiN','A2_PiN','A3_PiN',
'A1_PiN','A2_PiN','A3_PiN','A1_PiN','A2_PiN','A3_PiN',
'A1_PiN','A2_PiN','A3_PiN']})
I have tried Groupby, Melt, Reshape, Stack and loops but could not do that. If anyone could help? Thanks
When you tried stack, you were on one good track. you need to set_index first and reset_index after such as:
df2 = df1.set_index('DateTime').stack().reset_index(name='Power')\
.rename(columns={'level_1':'ModID'}) #to fit the names your expected output
And you get:
print (df2)
DateTime ModID Power
0 12/12/2019 A1_Pin 2
1 12/12/2019 A2_Pin 1
2 12/12/2019 A3_Pin 85
3 12/13/2019 A1_Pin 8
4 12/13/2019 A2_Pin 2
5 12/13/2019 A3_Pin 36
6 12/14/2019 A1_Pin 8
7 12/14/2019 A2_Pin 3
8 12/14/2019 A3_Pin 78
9 12/15/2019 A1_Pin 3
10 12/15/2019 A2_Pin 4
11 12/15/2019 A3_Pin 32
12 12/16/2019 A1_Pin 9
13 12/16/2019 A2_Pin 5
14 12/16/2019 A3_Pin 75
I'd try something like this:
df1.set_index('DateTime').unstack().reset_index()

how to convert a pandas column containing list into dataframe

I have a pandas dataframe.
One of its columns contains a list of 60 elements, constant across its rows.
How do I convert each of these lists into a row of a new dataframe?
Just to be clearer: say A is the original dataframe with n rows. One of its columns contains a list of 60 elements.
I need to create a new dataframe nx60.
My tentative:
def expand(x):
return(pd.DataFrame(np.array(x)).reshape(-1,len(x)))
df["col"].apply(lambda x: expand(x))]
it gives funny results....
The weird thing is that if i call the function "expand" on a single raw, it does exactly what I expect from it
expand(df["col"][0])
To ChootsMagoots: Thjis is the result when i try to apply your suggestion. It does not work.
Sample data
df = pd.DataFrame()
df['col'] = np.arange(4*5).reshape(4,5).tolist()
df
Output:
col
0 [0, 1, 2, 3, 4]
1 [5, 6, 7, 8, 9]
2 [10, 11, 12, 13, 14]
3 [15, 16, 17, 18, 19]
now exctract DataFrame from col
df.col.apply(pd.Series)
Output:
0 1 2 3 4
0 0 1 2 3 4
1 5 6 7 8 9
2 10 11 12 13 14
3 15 16 17 18 19
Try this:
new_df = pd.DataFrame(df["col"].tolist())
This is a little frankensteinish, but you could also try:
import numpy as np
np.savetxt('outfile.csv', np.array(df['col'].tolist()), delimiter=',')
new_df = pd.read_csv('outfile.csv')
You can try this as well:
newCol = pd.Series(yourList)
df['colD'] = newCol.values
The above code:
1. Creates a pandas series.
2. Maps the series value to columns in original dataframe.

Pandas DataFrame.update with MultiIndex label

Given a DataFrame A with MultiIndex and a DataFrame B with one-dimensional index, how to update column values of A with new values from B where the index of B should be matched with the second index label of A.
Test data:
begin = [10, 10, 12, 12, 14, 14]
end = [10, 11, 12, 13, 14, 15]
values = [1, 2, 3, 4, 5, 6]
values_updated = [10, 20, 3, 4, 50, 60]
multiindexed = pd.DataFrame({'begin': begin,
'end': end,
'value': values})
multiindexed.set_index(['begin', 'end'], inplace=True)
singleindexed = pd.DataFrame.from_dict(dict(zip([10, 11, 14, 15],
[10, 20, 50, 60])),
orient='index')
singleindexed.columns = ['value']
And the desired result should be
value
begin end
10 10 10
11 20
12 12 3
13 4
14 14 50
15 60
Now I was thinking about a variant of
multiindexed.update(singleindexed)
I searched the docs of DataFrame.update, but could not find anything w.r.t. index handling.
Am I missing an easier way to accomplish this?
You can use loc for selecting data in multiindexed and then set new values by values:
print singleindexed.index
Int64Index([10, 11, 14, 15], dtype='int64')
print singleindexed.values
[[10]
[20]
[50]
[60]]
idx = pd.IndexSlice
print multiindexed.loc[idx[:, singleindexed.index],:]
value
start end
10 10 1
11 2
14 14 5
15 6
multiindexed.loc[idx[:, singleindexed.index],:] = singleindexed.values
print multiindexed
value
start end
10 10 10
11 20
12 12 3
13 4
14 14 50
15 60
Using slicers in docs.