Separating tags in dirty data in pandas dataframe - pandas

Question

I have a dataframe similar to the following:
Column1 Column2 Tags Column3
str1 str2 owner:u1,env:prod1 str3
str2 str4 env:prod2,env:prod2 str6
str1 str3 str7
str3 str4 dwdws:qsded,ewe:22w str8
I cant filter the data based on Tags and clear out that is not in proper tags format since I need the whole data set. Please Note:
Third row, Tags column is already filtered as empty string
There may be repetitions in the key:value pairs.
I need to have Tags that I am interested in as a separate columns something like this:
Column1 Column2 Tags Column3 Owner env
str1 str2 owner:u1,env:prod1 str3 u1 prod1
str2 str4 env:prod2,env:prod2 str6 prod2
str1 str3 str7
str3 str4 dwdws:qsded,ewe:22w str8
I tried along the lines as:
Data['owner']=Data['Tags'].str.slice(Data.Tags.str.find('owner:'),Data.Tags.str.find('owner:')+<length until comma after owner is reached>)
I get all NaN values in the column. I am hoping there is a one or two liner to filter this out.
Thanks in advance

Answer 1

A generic method would be to extractall the key:value pairs, then to pivot:
out = (df.join(df['Tags'].str.extractall('([^:,]+):([^:,]+)')
.droplevel('match').pivot(columns=0, values=1))
)
Output:
Column1 Column2 Tags Column3 dwdws env ewe owner
0 str1 str2 owner:u1,env:prod str3 NaN prod NaN u1
1 str2 str4 env:prod str6 NaN prod NaN NaN
2 str1 str3 NaN str7 NaN NaN NaN NaN
3 str3 str4 dwdws:qsded,ewe:22w str8 qsded NaN 22w NaN
If you want to restrict the tags, adapt the first part of the regex:
out = (df.join(df['Tags'].str.extractall('(owner|env):([^:,]+)')
.droplevel('match').pivot(columns=0, values=1))
)
Output:
Column1 Column2 Tags Column3 env owner
0 str1 str2 owner:u1,env:prod str3 prod u1
1 str2 str4 env:prod str6 prod NaN
2 str1 str3 NaN str7 NaN NaN
3 str3 str4 dwdws:qsded,ewe:22w str8 NaN NaN
handling duplicated keys
out = (df.join(df['Tags'].str.extractall('(owner|env):([^:,]+)')
.droplevel('match').reset_index()
.pivot_table(index='index', columns=0, values=1, aggfunc='first')
)
)