Custom Class raises error when trying to call fit_transform - pandas

Question

I have created custom classes that want to use with scikit-learn Pipelines and Feature-Unions.
Each class takes as input a data frame with 2 columns: 0, 1
Each row in the first column is a numpy array, and each row in the second row is a number.
Given these two inputs, I have created a static function that calculates for each row of the data-frame 40 values.
So if the original data-frame is of shape 100 x 2 the output should be 100 x 40.
I've used the BaseEstimator and TransformerMixin modules to handle the fit_transform methods easily.
When I call the fit_transform method I get the following error:
ValueError: Shape of passed values is (288, 40), indices imply (288, 2)
class MelFrequencyCepstralCoefficientsCalculator(BaseEstimator, TransformerMixin):
"""outputs Mel-frequency cepstral coefficients"""
def __init__(self):
"""
"""
#staticmethod
def calculate_coef(file_array: np.array,
sample_rate):
"""
:param file_array:
:param sample_rate:
:return:
"""
# mfcc: Mel-frequency cepstral coefficients to each row
# this output is an array of shape (40, )
mfccs = np.mean(librosa.feature.mfcc(y=file_array,
sr=sample_rate,
n_mfcc=40).T, axis=0)
return mfccs
def transform(self, X, y=None) -> np.array:
print('Calculating Mel-frequency cepstral coefficients')
res = X.progress_apply(lambda row: self.calculate_coef(row[0], row[1]), axis=1).values
print('Output shape: {}'.format(res.shape))
return res
def fit(self, X, y=None):
"""Returns `self` unless something different happens in train and test"""
return self
mfccs = MelFrequencyCepstralCoefficientsCalculator()
x_val = mfccs.fit_transform(x_val_raw)
Shouldn't I be able to have a custom class that creates more than one features in a sklearn Pipeline?
PS: The progress_appy is valid. I'm using tqdm module that extends apply in order to have a progress bar for each calculation
The workaround that is much slower but works is the following:
def transform(self, X: pd.DataFrame, y=None) -> np.array:
"""
:param X:
:param y:
:return:
"""
print('Calculating Mel-frequency cepstral coefficients')
# res = X.apply(lambda row: self.calculate_coef(row[0], row[1]), axis=1)
features = np.empty((0, 40))
rows = X.iterrows()
for row in tqdm(rows):
ext_features = self.calculate_coef(row[1][0], row[1][1])
features = np.vstack([features, ext_features])
print('Output shape: {}'.format(features.shape))
return features