Numpy or Pandas for multiple dataframes of 2darray datasets - pandas

Question

I hope I used the correct synonyms in the title, which describes my problem.
My data has the following structure
D = {E_1, E_2...,E_n} with E_i = {M_{i,1}, M_{i,2},...M_{i,m}} and each M_{i,j} is a 6x2 Matrix.
I used a numpy array with dimension n x m x 6 x 2 to save the data. This was okay if every dataset E_i has the same amount of matrices.
But this solution is not working anymore, since I now work with datasets E_i which have a different number of Matrices i.e. E_i has m_i matrices.
Is there maybe a way in Pandas to resolve my problem? At the end I need to enter each matrix to operate with it as a numpy array i.e. multiplication, inverse, determinant….

Answer 1

You could try to use a multiindex in pandas in order to do this. This allows you to select the dataframe by level. A simple example of how you could achieve something like that:
D = np.repeat([0, 1], 12)
E = np.repeat([0, 1, 0, 1], 6)
print(D, E)
index_cols = pd.MultiIndex.from_arrays(
[D, E],
names=["D_idx", "E_idx"])
M = np.ones([24,2])
df = pd.DataFrame(M,
index=index_cols,
columns=["left", "right"])
print(df)
This gives you the dataframe:
left right
D_idx E_idx
0 0 1.0 1.0
0 1.0 1.0
0 1.0 1.0
0 1.0 1.0
0 1.0 1.0
0 1.0 1.0
1 1.0 1.0
1 1.0 1.0
1 1.0 1.0
1 1.0 1.0
1 1.0 1.0
1 1.0 1.0
1 0 1.0 1.0
0 1.0 1.0
0 1.0 1.0
0 1.0 1.0
0 1.0 1.0
0 1.0 1.0
1 1.0 1.0
1 1.0 1.0
1 1.0 1.0
1 1.0 1.0
1 1.0 1.0
1 1.0 1.0
You can then slice the dataframe based on levels, i.e. if you want to retrieve all elements in set D_1 you can select: df.loc[[(0, 0), (0, 1)], :]
You can generate selectors like this using list(zip(d_idx, e_idx)) in order to select specific rows.
You can find more about slicing and selecting the dataframe here:
https://pandas.pydata.org/pandas-docs/stable/user_guide/advanced.html