List of dictionarys to dataframe - pandas

Question

im trying to make a dataframe out of a list of dictionaries. I am quite new at this whole programming thing, and google just makes me more confused. That is why i am turning to you guys hoping for some assistance.
The first two list values (YV01', '3nP3RFgGnBrOfILK4DF2Tp) i would like to have under columns called: Name and GlobalId. I would lie to drop Pset_wallcommon, AC_Pset_RenovationAndPhasing, and BaseQuantities. And use the rest of the keys(if that what they are called) as column names.
It would be great if someone could give me the right push :)
For the record: Im am parsing an Ifc file with the IfcOpenshell package
The data:
['YV01', '3nP3RFgGnBrOfILK4DF2Tp', {'Pset_WallCommon': {'Combustible': False, 'Compartmentation': False, 'ExtendToStructure': False, 'SurfaceSpreadOfFlame': '', 'ThermalTransmittance': 0.0, 'Reference': '', 'AcousticRating': '', 'FireRating': '', 'LoadBearing': False, 'IsExternal': False}, 'AC_Pset_RenovationAndPhasing': {'Renovation Status': 'New'}, 'BaseQuantities': {'Length': 13786.7314346, 'Height': 2700.0, 'Width': 276.0, 'GrossFootprintArea': 3.88131387595, 'NetFootprintArea': 3.88131387595, 'GrossSideArea': 37.9693748734, 'NetSideArea': 37.9693748734, 'GrossVolume': 10.4795474651, 'NetVolume': 10.4795474651}}, 'YV01', '1M4JyBJhXD5xt8fBFUcjUU', {'Pset_WallCommon': {'Combustible': False, 'Compartmentation': False, 'ExtendToStructure': False, 'SurfaceSpreadOfFlame': '', 'ThermalTransmittance': 0.0, 'Reference': '', 'AcousticRating': '', 'FireRating': '', 'LoadBearing': False, 'IsExternal': False}, 'AC_Pset_RenovationAndPhasing': {'Renovation Status': 'New'}, 'BaseQuantities': {'Length': 6166.67382573, 'Height': 2700.0, 'Width': 276.0, 'GrossFootprintArea': 1.6258259759, 'NetFootprintArea': 1.6258259759, 'GrossSideArea': 15.9048193295, 'NetSideArea': 15.9048193295, 'GrossVolume': 4.38973013494, 'NetVolume': 4.38973013494}}
all_walls = ifc_file.by_type('IfcWall')
wallList = []
for wall in all_walls:
propertySets = (ifcopenshell.util.element.get_psets(wall))
wallList.append(wall.Name)
wallList.append(wall.GlobalId)
wallList.append(propertySets)
print(wallList)
wall_table = pd.DataFrame.from_records(wallList)
print(wall_table)
I have tried these basic pd.DataFrame.from_dict/records/arrays(data)
but the output looks like this
enter image description here
UPDATE: Thank you so much for your help, i am learning alot from this!
So i made a dictionary out of the wallList, and flattened the dict. like this:
#list of walls
for wall in all_walls:
propertySets = (ifcopenshell.util.element.get_psets(wall))
wallList.append(wall.Name)
wallList.append(wall.GlobalId)
wallList.append(propertySets)
#dict from list
wall_dict = {i: wallList[i] for i in range(0, len(wallList))}
new_dict = {}
#flattening dict
for key, value in wall_dict.items():
if isinstance(value, dict):
for key in value.keys():
for key2 in value[key].keys():
new_dict[key + '_' + key2] = value[key][key2]
else:
new_dict[key] = value
wall_table = pd.DataFrame.from_dict(new_dict, orient='index')
print(wall_table)
It seems to work pretty good, the only problem is that the dataframe contains all walls, but only propertyset data from the first in the list. I cant seem to understand how the dict flattening loop works. I would also like the index names (Pset_WallCommon_Combustible, and so on) to be the columns in my dataframe. Is that possible?
enter image description here

Answer 1

EDIT : Simply flattening a list as i did goes nowhere. Actually, i think you should drop this list thing altogether and try to load the Dataframe from a dictionnary. We'd need to see what does all_walls look like to help you for that, tho.
Have you tried directly loading the all_walls dictionary into a dataframe : df = pd.Dataframe.from_dict(all_walls) ?
I think if that doesnt work, flattening the dictionnaries in a fashion similar to the following should do the trick.
new_dict = {}
for key, value in all_walls.items():
if isinstance(value, dict):
for key in value.keys():
for key2 in value[key].keys():
new_dict[key + '_' + key2] = value[key][key2]
else:
new_dict[key] = value