Speeding up conversion of pandas to numpy - pandas

Question

I've got a Pandas df of approximately 2.5m rows, with a multi-index of the form:
('assetCode', 'date') and approximately 60 columns.
I'm trying to convert this to a 3D numpy matrix:
assetCodes = X_calculated.index.get_level_values(0).unique().sort_values().to_numpy()
dates = X_calculated.index.get_level_values(1).unique().sort_values().to_numpy()
columns = X_calculated.columns.to_numpy()
myData = np.empty((assetCodes.size, dates.size, columns.size))
def updateMatrix(row):
idx = row.name
assetLabel = np.searchsorted(assetCodes, idx[0])
dateLabel = np.where(dates == idx[1])
myData[assetLabel][dateLabel] = row.to_numpy()
X_calculated.apply(updateMatrix, axis=1)
This operation takes a very long time. Is there a quicker way?

Answer 1

I think if you already have all the combinations of assetCode and date in your dataframe, you can do it whit reshape:
# example data
X_calculated = pd.DataFrame(np.arange(36).reshape(9, -1),
index=pd.MultiIndex.from_product([range(101,104),
range(111,114)],
names=('assetCode','date')),
columns=list('abcd'))
# get dimensions
nb_asset = X_calculated.index.get_level_values(0).nunique()
nb_dates = X_calculated.index.get_level_values(1).nunique()
nb_cols = len(X_calculated.columns)
# create myData
myData = X_calculated.sort_index().to_numpy().reshape(nb_asset, nb_dates, nb_cols)
print (myData) #same result than with your code
[[[ 0 1 2 3]
[ 4 5 6 7]
[ 8 9 10 11]]
[[12 13 14 15]
[16 17 18 19]
[20 21 22 23]]
[[24 25 26 27]
[28 29 30 31]
[32 33 34 35]]]
If you have missing combinations, you can use reindex before with pd.MultiIndex.from_product with unique value in both index levels. No need to sort_index anymore I think as the new multiIndex is generated sorted
assetCodes = X_calculated.index.get_level_values(0).unique().sort_values()
dates = X_calculated.index.get_level_values(1).unique().sort_values()
myData = (X_calculated.reindex(pd.MultiIndex.from_product([assetCodes, dates]))
.to_numpy()
.reshape(len(assetCodes), len(dates), len(X_calculated.columns))
)