I have a Python dictionary in the below structure. I am trying to extract certain elements from the Dictionary and convert them to a Dataframe.
When I try to perform pd.Dataframe(df) I get summary of the 2 groups data and PageCount whereas I only want the elements within Output in the Dataframe
{'code': 200,
'data': {'Output': [
{'id': 58,
'title': 'title1'},
{'id': 59,
'title': 'title2'}],
'PageCount': {'count': 196,
'page': 1,
'perPage': 10,
'totalPages': 20}},
'request_id': 'fggfgggdgd'}
Expected output:
id, title
58, title1
59, title2
You can do,
df = pd.io.json.json_normalize(dct["data"]["Output"])
You can also use;
l=[v['Output'] for k,v in d.items() if isinstance(v,dict) & ('Output' in str(v))]
pd.DataFrame(l[0])
id title
0 58 title1
1 59 title2
Related
i have several dataframes, with all the same columns, within one list that i would like to have within one dataframe.
For instance, i have these three dataframes here:
df1 = pd.DataFrame(np.array([[10, 20, 30], [40, 50, 60], [70, 80, 90]]),
columns=['a', 'b', 'c'])
df2 = pd.DataFrame(np.array([[11, 22, 33], [44, 55, 66], [77, 88, 99]]),
columns=['a', 'b', 'c'])
df3 = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]),
columns=['a', 'b', 'c'])
within one list:
dfList = [df1,df2,df3]
I know i can use the following which provides me with exactly what I'm looking for:
df_merge = pd.concat([dfList[0],dfList[1],dfList[2]])
However, my in my actual data i have 100s of dataframes within a list, so I'm trying to find a way to loop through and concat:
dfList_all = pd.DataFrame()
for i in range(len(dfList)):
dfList_all = pd.concat(dfList[i])
I tried the following above, but it provides me with the following error:
TypeError: first argument must be an iterable of pandas objects, you passed an object of type "DataFrame"
Any ideas would be wonderful. Thanks
I have the dataframe like
ID Series
1102 [('taxi instructions', 13, 30, 'NP'), ('consistent basis', 31, 47, 'NP'), ('the atc taxi clearance', 89, 111, 'NP')]
1500 [('forgot data pages info', 0, 22, 'NP')]
649 [('hud', 0, 3, 'NP'), ('correctly fotr approach', 12, 35, 'NP')]
I am trying to parse the text in column named Series to different columns named Series1 Series2 etc upto the highest number of texts parsed.
df_parsed = df['Series'].str[1:-1].str.split(', ', expand = True)
something like this:
ID Series Series1 Series2 Series3
1102 [('taxi instructions', 13, 30, 'NP'), ('consistent basis', 31, 47, 'NP'), ('the atc taxi clearance', 89, 111, 'NP')] taxi instructions consistent basis the atc taxi clearance
1500 [('forgot data pages info', 0, 22, 'NP')] forgot data pages info
649 [('hud', 0, 3, 'NP'), ('correctly fotr approach', 12, 35, 'NP')] hud correctly fotr approach
The format of your final result is not easy to understand, but maybe you can follow the concept to create your new columns:
def process(ls):
return ' '.join([x[0] for x in ls])
df['Series_new'] = df['Series'].apply(lambda x: process(x))
And if you want to create N new columns (N = max_len(Series_list)), I think you can calculate N first. Then, follow the concept above and fill in NaN properly to create N new columns.
To convert list of dictionary already present in the dataset to a dataframe.
The dataset looks something like this.
[{'id': 35, 'name': 'Comedy'}]
How do I convert this list of dictionary to dataframe?
Thank you for your time!
I want to retrieve:
Comedy
from the list of dictionary.
Use:
df = pd.DataFrame({'col':[[{'id': 35, 'name': 'Comedy'}],[{'id': 35, 'name': 'Western'}]]})
print (df)
col
0 [{'id': 35, 'name': 'Comedy'}]
1 [{'id': 35, 'name': 'Western'}]
df['new'] = df['col'].apply(lambda x: x[0].get('name'))
print (df)
col new
0 [{'id': 35, 'name': 'Comedy'}] Comedy
1 [{'id': 35, 'name': 'Western'}] Western
If possible multiple dicts in list:
df = pd.DataFrame({'col':[[{'id': 35, 'name': 'Comedy'}, {'id':4, 'name':'Horror'}],
[{'id': 35, 'name': 'Western'}]]})
print (df)
col
0 [{'id': 35, 'name': 'Comedy'}, {'id': 4, 'name...
1 [{'id': 35, 'name': 'Western'}]
df['new'] = df['col'].apply(lambda x: [y.get('name') for y in x])
print (df)
col new
0 [{'id': 35, 'name': 'Comedy'}, {'id': 4, 'name... [Comedy, Horror]
1 [{'id': 35, 'name': 'Western'}] [Western]
And if want extract all values:
df1 = pd.concat([pd.DataFrame(x) for x in df['col']], ignore_index=True)
print (df1)
id name
0 35 Comedy
1 4 Horror
2 35 Western
This is first time trying this. I actually have a dict of lists I am generating in a program, but since this is my first time ever trying this, I am using a dummy dict just for testing.
I am following this:
python Making heatmap from DataFrame
but I am failing with the following:
Traceback (most recent call last):
File "C:/Users/Mark/PycharmProjects/main/main.py", line 20, in <module>
sns.heatmap(df, cmap='RdYlGn_r', linewidths=0.5, annot=True)
File "C:\Users\Mark\AppData\Roaming\Python\Python36\site-packages\seaborn\matrix.py", line 517, in heatmap
yticklabels, mask)
File "C:\Users\Mark\AppData\Roaming\Python\Python36\site-packages\seaborn\matrix.py", line 168, in __init__
cmap, center, robust)
File "C:\Users\Mark\AppData\Roaming\Python\Python36\site-packages\seaborn\matrix.py", line 205, in _determine_cmap_params
calc_data = plot_data.data[~np.isnan(plot_data.data)]
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
My code:
import pandas as pd
import seaborn as sns
Index = ['key1', 'key2', 'key3', 'key4', 'key5']
Cols = ['A', 'B', 'C', 'D']
testdict = {
"key1": [1, 2, 3, 4],
"key2": [5, 6, 7, 8],
"key3": [9, 10, 11, 12],
"key4": [13, 14, 15, 16],
"key5": [17, 18, 19, 20]
}
df = pd.DataFrame(testdict, index=Index, columns=Cols)
df = df.transpose()
sns.heatmap(df, cmap='RdYlGn_r', linewidths=0.5, annot=True)
You need to switch your column and index labels
Cols = ['key1', 'key2', 'key3', 'key4', 'key5']
Index = ['A', 'B', 'C', 'D']
I have a new dataframe, how to append it to an existed csv?
I tried the following code:
f = open('test.csv', 'w')
df.to_csv(f, sep='\t')
f.close()
But it doesn't append anything to test.csv. The csv is big, I only want to use append, rather than read the whole csv as dataframe and concatenate it to and write it to a new csv. Is there any good method to solve the problem? Thanks.
Try this:
df.to_csv('test.csv', sep='\t', header=None, mode='a')
# NOTE: -----> ^^^^^^^^
TL:DR Answer from MaxU is correct.
df.to_csv('old_file.csv', header=None, mode='a')
I had the same problem, wishing to append to DataFrame and save to a CSV inside a loop. It seems to be a common pattern.
My criteria was:
Write back to the same file
Don't write data more than necessary.
Keep appending new data to the dataframe during the loop.
Save on each iteration (in case long running loop crashes)
Don't store index in the CSV file.
Note the different values of mode and header. In a complete write, mode='w' and header=True, but in an append, mode='a' and header='False'.
import pandas as pd
# Create a CSV test file with 3 rows
data = [['tom', 10], ['nick', 15], ['juli', 14]]
test_df = pd.DataFrame(data, columns = ['Name', 'Age'])
test_df.to_csv('test.csv', mode='w', header=True, index=False)
# Read CSV into a new frame
df = pd.read_csv('test.csv')
print(df)
# MAIN LOOP
# Create new data in a new DataFrame
for i in range(0, 2):
newdata = [['jack', i], ['jill', i]]
new_df = pd.DataFrame(newdata, columns = ['Name', 'Age'])
# Write the new data to the CSV file in append mode
new_df.to_csv('test.csv', mode='a', header=False, index=False)
print('check test.csv')
# Combine the new data into the frame ready for the next loop.
test_df = pd.concat([test_df, new_df], ignore_index=True)
# At completion, it shouldn't be necessary, but to write the complete data
test_df.to_csv('completed.csv', mode='w', header=True, index=False)
# completed.csv and test.csv should be identical.
try the following code, it will generate an old file(10 rows) and new file(2 rows) in your local folder. After I append, the new content all mix up:
import pandas as pd
import os
dir_path = os.path.dirname(os.path.realpath("__file__"))
print(dir_path)
raw_data = {'HOUR': [4, 9, 12, 7, 3, 15, 2, 16, 3, 21],
'LOCATION': ['CA', 'HI', 'CA', 'IN', 'MA', 'OH', 'OH', 'MN', 'NV', 'NJ'],
'TYPE': ['OLD', 'OLD', 'OLD', 'OLD', 'OLD', 'OLD', 'OLD', 'OLD', 'OLD', 'OLD'],
'PRICE': [4, 24, 31, 2, 3, 25, 94, 57, 62, 70]}
old_file = pd.DataFrame(raw_data, columns = ['HOUR', 'LOCATION', 'TYPE', 'PRICE'])
old_file.to_csv(dir_path+"/old_file.csv",index=False)
raw_data = {'HOUR': [2, 22],
'LOCATION': ['CA', 'MN'],
'TYPE': ['NEW', 'NEW'],
'PRICE': [80, 90]}
new_file = pd.DataFrame(raw_data, columns = ['HOUR', 'LOCATION', 'TYPE', 'PRICE'])
new_file.to_csv(dir_path+"/new_file.csv",index=False)
new_file=dir_path+"/new_file.csv"
df=pd.read_csv(new_file)
df.to_csv('old_file.csv', sep='\t', header=None, mode='a')
it will come to:
HOUR LOCATION TYPE PRICE
4 CA OLD 4
9 HI OLD 24
12 CA OLD 31
7 IN OLD 2
3 MA OLD 3
15 OH OLD 25
2 OH OLD 94
16 MN OLD 57
3 NV OLD 62
21 NJ OLD 70
02CANEW80
122MNNEW90
To append a pandas dataframe in a csv file, you can also try it.
df = pd.DataFrame({'Time':x, 'Value':y})
with open('CSVFileName.csv', 'a+', newline='') as f:
df.to_csv(f, index=False, encoding='utf-8', mode='a')
f.close()