Need help unpacking a nested array into a pandas dataframe - pandas

Question

I am running some code that generates an array with the following shape (18433, 17, 600 to 885 in length). I need to unpack that into a pandas dataframe with 17 columns, and rows containing data for 18433 entities that each have 600 to 885 time series entries. The code that generates the array is shown below. I am a relative python newbie and have reached the extent of my skills. I tried unpacking using a for loop, but it takes forever. Are there any libraries or methods that are more efficient?
# Generate full monthly cash flow arrays
# define constant input parameters
eloss = 0
weight = 1.0
prod_wt = 1.0
inv_wt = 1.0
stx_oil = 0.0795
stx_gas = 0.0795
stx_ngl = 0.0795
adval = 0
aban = 150000
# Create function for slicing the volume array and calculating the monthly cash flow
def econ_ncf_iter(r):
econ_ncf_iter = econ_cf(index = r, uid = prop_list.loc[r, 'PROPNUM'], wi = prop_list.loc[r, 'WI'],
nri = prop_list.loc[r, 'NRI'], roy = prop_list.loc[r, 'Royalty'], eloss = eloss,
weight = weight, prod_wt = prod_wt, inv_wt = inv_wt,
shrink = np.round(prop_list.loc[r, 'SHRINK'] / 100, 6),
btu = np.round(prop_list.loc[r, 'BTU'] / 1000, 6),
ngl_yield = np.round(prop_list.loc[r, 'NGL/GAS'], 6),
pri_oil = np.extract(oilprice[r][0] == prop_list.loc[r, 'PROPNUM'], oilprice[r][1]),
pri_gas = np.extract(gasprice[r][0] == prop_list.loc[r, 'PROPNUM'], gasprice[r][1]),
paj_oil = prop_list.loc[r, 'PAJ_OIL'],
paj_gas = np.extract(gasdiff[r][0] == prop_list.loc[r, 'PROPNUM'], gasdiff[r][1]),
paj_ngl = prop_list.loc[r, 'PAJ_NGL'], stx_oil = stx_oil, stx_gas = stx_gas, stx_ngl = stx_ngl,
adval = adval, opc_fix = np.round(prop_list.loc[r, 'OPC/T'], 2),
opc_oil = np.round(prop_list.loc[r, 'OIL_OPEX'], 2),
opc_gas = np.round(prop_list.loc[r, 'GAS_OPEX'], 2),
capex = np.round(prop_list.loc[r, 'CAPITAL'] * 1000, 2), aban = aban)
return econ_ncf_iter
# generate net cash flow array
econ_ncf = lambda r: econ_ncf_iter(r)
vecon_ncf = np.vectorize(econ_ncf_iter, otypes = [object])
ncf_arr_packed = vecon_ncf(R)

Answer 1

I figured it out and it was pretty easy
'''
ncf_pd_dflist = []
columns = ['UID', 'Month', 'Grs Oil', 'Grs Gas', 'Net Oil', 'Net Gas', 'Net NGL', 'Oil Revenue', 'Gas Revenue',
'NGL Revenue', 'Total Revenue', 'Total Tax', 'OPEX', 'Operating Income', 'Cumulative Op CF', 'Net Cashflow',
'Cumulative Net CF']
pbar = tqdm(len(R))
for r in R:
ncf_pd_dflist.append(pd.DataFrame(np.transpose(ncf_arr_packed[r])))
pbar.update()
ncf_pd = pd.concat(ncf_pd_dflist)
ncf_pd.columns = columns
pbar.close()
'''
Simple code to loop though the array and create a list of pandas dataframes. After the loop finishes I concatenate the dataframe lists into a single dataframe. This took about 5 seconds to complete.

Answer 2

Although you already figured out a solution, here's a general alternative without an explicit loop. It takes some simple steps:
If the desired horizontal axis (your middle axis) is not the last, swap them.
Reshape to a 2D array of the horizontal rows.
Make the DataFrame with a MultiIndex from the cartesian product of the other axes.
Assuming the array is arr:
x, y, z = arr.shape
df = pd.DataFrame(arr.swapaxes(1, 2).reshape(x*z, -1),
pd.MultiIndex.from_product([np.arange(x), np.arange(z)]))