Dataframe rows to json - pandas

Question

I have a dataframe with many rows and columns of the form (this is an oversimplified dataframe):
id dur proto service state attack_cat label
0 1 0.121478 tcp dns FIN Normal 0
1 2 4.287901 udp ftp INT Exploits 1
I would like to write all the rows of this dataframe as json items, like for example, for the first row:
{"type": "event",
"subtype": "",
"datatype": "Instance",
"domain": "Cyber",
"created": str(datetime.datetime.now()),
"details": {id: 1,
dur: 0.121478,
proto: tcp,
service: dns,
state: FIN,
attack_cat: Normal,
label:0}
}
I tried to do something like:
{"type": "event",
"subtype": "",
"datatype": "Instance",
"domain": "Cyber",
"created": str(datetime.datetime.now()),
"details": dataframe.loc[i].to_dict()
}
and do a for loop through all the rows, but it gives me the error
TypeError: unhashable type: 'dict'

Answer 1

There are actually two problems, as far as I can see.
Generally, a python dictionary can be the value of a dictionary. For example,
dict = {
"first_name": "robert",
"last_name": "ren",
"hist": {"today": 1, "yesterday": 2}
}
The value of hist is a dict. This means, that from this perspective your code seems to be ok. However, a dictionary can not be the key, nor can you nest one dict in another one. For example,
{{}}
gives you the following error
TypeError: unhashable type: 'dict'
If you want to solve this problem, I recommend that you print out each line as you create the dicts to see where it breaks. That said, I guess that "details": dataframe.loc[i].to_dict() causes you the trouble.
Second, your first dict is invalid as the keys are not strings. This should be
{
"type": "event",
"subtype": "",
"datatype": "Instance",
"domain": "Cyber",
"created": str(datetime.datetime.now()),
"details": {
"id": 1,
"dur": 0.121478,
"proto": tcp,
"service": dns,
"state": FIN,
"attack_cat": Normal,
"label": 0
}
}
Assuming that the variables are defined variables.
EDIT
The following code works on my machine.
import pandas as pd
df = pd.DataFrame({
"id": [1],
"dur": [0.12],
"proto": ["tcp"],
"service": ["dns"],
"state": ["FIN"],
"attack_cat": ["Normal"],
"label": [0]
})
row_list_as_dict = []
for idx, row in df.iterrows():
row_list_as_dict.append({
"type": "event",
"subtype": "",
"datatype": "Instance",
"domain": "Cyber",
"details": {
'id': row["id"],
'dur': row["dur"],
'proto': row["proto"],
'service': row["service"],
'state': row["state"],
'attack_cat': row["attack_cat"],
'label': row["label"]
}
})
row_list_as_dict