improve performance of double loop in pandas - pandas

Question

I have a dataframe consisting of numeric and categorical fields:
import pandas as pd
df2=pd.DataFrame({'col1':[1,2,3,4],'col2':[5,6,7,8], 'col3':['cat','cat','dog','bird']})
df2
And am calculating how similar each row is with the following code:
#calculate distance matrix comparing how similar two rows are
vals=[]
for i in range(len(df2)):
for j in range(len(df2)):
if(j<=i): continue
a=df2.iloc[i,:]
b=df2.iloc[j,:]
d0=(a[0]-b[0])**2
d1=(a[1]-b[1])**2
d2=np.where(a[2]==b[2],0,10)**2
row_values=(i,j, (d0 + d1 +d2)**0.5)
vals.append(row_values)
new_df = pd.DataFrame(vals, columns =['Row1','Row2','Difference'])
new_df
this works fine for a small dataframe, but when I implement it similarly to a dataframe that has 10k rows and 10 columns being used, it takes a very loooong time to compute.
Are there any suggestions on how to improve the processing power of this code?
I start with:
col1 col2 col3
0 1 5 cat
1 2 6 cat
2 3 7 dog
3 4 8 bird
and end up with:
Row1 Row2 Difference
0 0 1 1.414214
1 0 2 10.392305
2 0 3 10.862780
3 1 2 10.099505
4 1 3 10.392305
5 2 3 10.099505
I am calculating the distance between each row of data.

Answer 1

This is a distance matrix problem, so we can use distance_matrix and broadcasting. But note that this only works when your data is not too large.
from scipy.spatial import distance_matrix
# normal distance:
d01 = distance_matrix(df2[['col1','col2']].values, df2[['col1','col2']].values)**2
# category distance
d2 = x = df2['col3'].values[:,None] != df2['col3'].values
# the matrix
dist_mat = np.sqrt(d1 + x*100)
# we only care for the distance with row != col
np.triu(dist_mat)
Output:
array([[ 0. , 1.41421356, 10.39230485, 10.86278049],
[ 0. , 0. , 10.09950494, 10.39230485],
[ 0. , 0. , 0. , 10.09950494],
[ 0. , 0. , 0. , 0. ]])