Pandas groupby-apply: cannot reindex from a duplicate axis - pandas

Question

I am performing a groupby operation on a DataFrame. On each of the group I have to rename two columns and drop one, so that each group will have the following form:
index(timestamp) | column-x | column-y
... | .... | .....
The index is a timestamp and it will be common to each group. 'column-x' and 'column-y' instead will be different to each group. My goal is then to join all groups on the index so that I have a unique DataFrame such as:
index(timestamp) | column-x1 | column-y1 | column-x2 | column-y2 | ...
... | ..... | ...... | ....... | ....... | ...
The function I apply to each group is (can I make inplace edit to the group while iterating?):
def process_ssp(df_ssp):
sensor_name = df_ssp.iloc[0]['subsystem-sensor-parameter'] # to be used as column name
df_ssp.rename(columns = {
'value_raw': '%s_raw' % sensor_name,
'value_hrf': '%s_hrf' % sensor_name,
}, inplace = True)
df_ssp.drop('subsystem-sensor-parameter', axis='columns', inplace=True) # since this is the column I am grouping on I guess this isn't the right thing to do?
return df_ssp
Then I call:
res = df_node.groupby('subsystem-sensor-parameter', as_index=False).apply(process_ssp)
Which produces the error:
ValueError: cannot reindex from a duplicate axis
EDIT:
Dataset sample https://drive.google.com/file/d/1RvPE1t3BmjeaqCNkVqGwmokCFQQp77n8/view?usp=sharing

Answer 1

You can first add column subsystem-sensor-parameter for MultiIndex, reshape by unstack, sorting MultiIndex in columns by second level and chane their positons. Last convert MultiIndex by flattening with map and join:
res = (df_node.set_index('subsystem-sensor-parameter', append=True)
.unstack()
.sort_index(axis=1, level=1)
.swaplevel(0,1, axis=1))
res.columns = res.columns.map('_'.join)

Answer 2

I'm able to successfully apply your code and produce the output you want by iterating over the groups rather than using apply:
import pandas as pd
df = pd.read_csv('/Users/jeffmayse/Downloads/sample.csv')
df.set_index('timestamp', inplace=True)
def process_ssp(df_ssp):
sensor_name = df_ssp.iloc[0]['subsystem-sensor-parameter'] # to be used as column name
df_ssp.rename(columns = {
'value_raw': '%s_raw' % sensor_name,
'value_hrf': '%s_hrf' % sensor_name,
}, inplace = True)
df_ssp.drop('subsystem-sensor-parameter', axis='columns', inplace=True) # since this is the column I am grouping on I guess this isn't the right thing to do?
return df_ssp
groups = df.groupby('subsystem-sensor-parameter')
out = []
for name, group in groups:
try:
out.append(process_ssp(group))
except:
print(name)
pd.concat(out).shape
Out[7]: (16131, 114)
And in fact, the issue is in the apply method, as your function is not needed to produce the error:
df.groupby('subsystem-sensor-parameter', as_index=False).apply(lambda x: x)
evaluates to ValueError: cannot reindex from a duplicate axis as well.
However, this statement evaluates as we'd expect:
df.reset_index(inplace=True)
df.groupby('subsystem-sensor-parameter', as_index=False).apply(process_ssp)
Out[22]:
nc-devices-alphasense_hrf ... wagman-uptime-uptime_raw
0 0 ... NaN
1 NaN ... NaN
2 NaN ... NaN
3 NaN ... NaN
...
The issue is that you have a DatetimeIndex with duplicate values. .apply is attempting to combine the result sets back together, but is not sure how to combine an index with duplicate values. At least, I believe that's it. Reset your index and try again.
Edit: to expand, you see this error commonly when trying to reindex a DatetimeIndex i.e., you have an hourly index and want to convert it to a second resolution index, or usually fill in missing hours. you use reindex, but it will fail if your index has duplicate values. I'd guess that is what is happening here: the dataframes produced by the function being applied have duplicate index values and the error comes from trying to produce the output via calling reindex on a DatetimeIndex with duplicates. Resetting the index works because your index is now all unique, and the timestamp column is not important to this operation.