Paired difference of columns in dataframe to generate dataframe with 1.3 million columns - pandas

Question

I have a dataframe with 1600 columns.
The dataframe df looks like where the column names are 1, 3 , 2:
Row Labels 1 3 2
41730Type1 9 6 5
41730Type2 14 12 20
41731Type1 2 15 5
41731Type2 3 20 12
41732Type1 8 10 5
41732Type2 8 18 16
I need to create the following dataframe df2 pythonically:
Row Labels (1, 2) (1, 3) (2, 3)
41730Type1 -4 -3 1
41730Type2 6 -2 -8
41731Type1 3 13 10
41731Type2 9 17 8
41732Type1 -3 2 5
41732Type2 8 10 2
where e.g. column (1, 2) is created by df[2] - df[1]
The column names for df2 are created by pairing the column headers of df1 such that second element of each name is greater than first e.g. (1, 2), (1, 3), (2, 3)
The second challenge is can pandas dataframe support 1.3 million columns?

Answer 1

We can do combinations for the column , then create the dict and concat it back
import itertools
l=itertools.combinations(df.columns,2)
d={'{0[0]}|{0[1]}'.format(x) : df[x[0]]-df[x[1]] for x in [*l] }
newdf=pd.concat(d,axis=1)
1|3 1|2 3|2
RowLabels
41730Type1 3 4 1
41730Type2 2 -6 -8
41731Type1 -13 -3 10
41731Type2 -17 -9 8
41732Type1 -2 3 5
41732Type2 -10 -8 2

Answer 2

itertools combinations seems the obvious choice, same as #YOBEN_S, a different route to the solution, using numpy arrays and dictionary
from itertools import combinations
new_data = combinations(df.to_numpy().T,2)
new_cols = combinations(df.columns, 2)
result = {key : np.subtract(arr1,arr2)
if key[0] > key[1]
else np.subtract(arr2,arr1)
for (arr1, arr2), key
in zip(new_data,new_cols)}
outcome = pd.DataFrame.from_dict(result,orient='index').sort_index().T
outcome
(1, 2) (1, 3) (3, 2)
0 -4 -3 1
1 6 -2 -8
2 3 13 10
3 9 17 8
4 -3 2 5
5 8 10 2