Pandas masked dataframe assign 2D array - pandas

Question

I'm trying to solve a riddle with dataframe and masked array.
Context
I'm trying to make some methods to help me with some machine learning stuff. My goals is from a simple dataframe to add new information to it. I want these functions as simple to use.
Problem
This is a function that take what I called a model, and can make a transform from data. Then, I want to set these new data on new or existing column. I'm not using apply function because, if I do it, I'm losing the ability to extract several data at once with parrallism (that's why I'm extracting first then applying the 'features'.
Here is the riddle, I want to work with a mask because without this mask I cannot modify the original dataframe. If you remove this mask,if dataframe and features have the same number of lines everything is fine. Now, if I add this mask, it seems to be taken row by row and dimension mismatch....
EDIT: I forgot to tell that 'features' correspond to a 2d numpy array.
Error
ValueError: Must have equal len keys and value when setting with an ndarray
Code
def transform(dataframe, tags, out, model, mask=None):
# Check mandatory fields
mandatory = ['datum']
if not isinstance(tags, dict) or not all(elem in mandatory for elem in tags.keys()):
raise Exception(f'Not a dict or missing tag: {mandatory}.')
# Mask creation (see pandas view / copy mechanism)
if mask is None:
mask = [True] * len(dataframe.index)
features = model.transform(dataframe.loc[mask, tags['datum']].to_numpy())
dataframe.loc[mask, out] = features.tolist()
return dataframe
Example
Data can be something like this:
Row ; Data ; Label
0 ; [61953.017837947686, 9.505037089204054, 74.585... ] ;0
1 ; [80832.69302693632, 9.524642547991316, 83.9228... ] ;1
A model could be a PCA from sklearn.
The method could be something like this:
transform(dataframe, {'datum': 'Data'}, 'PCA', PCA(), mask=dataframe[dataframe['Label']==1)
Then output would be:
Row ; Data ; Label ; PCA
0 ; [61953.017837947686, 9.505037089204054, 74.585... ] ;0 ; [74.585... ]
1 ; [80832.69302693632, 9.524642547991316, 83.9228... ] ;1 ; [92.578... ]
Current solution
mask = inputs[condition]
inputs['PCA'] = np.nan
inputs[mask] = Classification.transform(inputs[mask], {'datum': 'Data'}, wavelet, 'PCA')
def transform(dataframe, tags, model, out):
# Check mandatory fields
mandatory = ['datum']
if not isinstance(tags, dict) or not all(elem in mandatory for elem in tags.keys()):
raise Exception(f'Not a dict or missing tag: {mandatory}.')
features = model.transform(dataframe[tags['datum']].to_numpy())
dataframe[out] = features.tolist()
return dataframe
Thanks for answers!
Best regards

Answer 1

Ok, So I find a solution that fit my needs.
Second idea was according to me a bad idea, as it seems not easy to pass view as argument and edit it with a convenient way.
So, I change my first proposition. Here is the final version:
def transform(dataframe, tags, model, out, mask=None):
# Check mandatory fields
mandatory = ['datum']
if not isinstance(tags, dict) or not all(elem in mandatory for elem in tags.keys()):
raise Exception(f'Not a dict or missing tag: {mandatory}.')
# Mask creation (see pandas view / copy mechanism)
if mask is None:
mask = [True] * len(dataframe.index)
# dataframe[out] = np.nan
features = model.transform(dataframe.loc[mask, tags['datum']].to_numpy())
dataframe.loc[mask, out] = pd.Series(features.tolist())
return dataframe
Instead of doing dataframe.loc[mask, out] = features.tolist(), I switch to : dataframe.loc[mask, out] = pd.Series(features.tolist()). By going through a Series it fit without any problems.
Thanks Ken Syme to be involved in my problem.