I am wondering if anyone can help. I have a number of dataframes stored in a dictionary. I simply want to access each of these dataframes and count the values in a column in the column I have 10 letters. In the first dataframe there are 5bs and 5 as. For example the output from the count I would expect to be is a = 5 and b =5. However for each dataframe this count would be different hence I would like to store the output of these counts either into another dictionary or a separate variable.
The dictionary is called Dict and the column name in all the dataframes is called letters. I have tried to do this by accessing the keys in the dictionary but can not get it to work. A section of what I have tried is shown below.
import pandas as pd
for key in Dict:
Count=pd.value_counts(key['letters'])
Count here would ideally change with each new count output to store into a new variable
A simplified example (the actual dataframe sizes are max 5000,63) of the one of the 14 dataframes in the dictionary would be
`d = {'col1': [1, 2,3,4,5,6,7,8,9,10], 'letters': ['a','a','a','b','b','a','b','a','b','b']}
df = pd.DataFrame(data=d)`
The other dataframes are names df2,df3,df4 etc
I hope that makes sense. Any help would be much appreciated.
Thanks
If you want to access both key and values when iterating over a dictionary, you should use the items function.
You could use another dictionary to store the results:
letter_counts = {}
for key, value in Dict.items():
letter_counts[key] = value["letters"].value_counts()
You could also use dictionary comprehension to do this in 1 line:
letter_counts = {key: value["letters"].value_counts() for key, value in Dict.items()}
The easiest thing is probably dictionary comprehension:
d = {'col1': [1, 2,3,4,5,6,7,8,9,10], 'letters': ['a','a','a','b','b','a','b','a','b','b']}
d2 = {'col1': [1, 2,3,4,5,6,7,8,9,10,11], 'letters': ['a','a','a','b','b','a','b','a','b','b','a']}
df = pd.DataFrame(data=d)
df2 = pd.DataFrame(d2)
df_dict = {'d': df, 'd2': df2}
new_dict = {k: v['letters'].count() for k,v in df_dict.items()}
# out
{'d': 10, 'd2': 11}
Related
I have a dataframe with a very large number of columns of different types. I want to encode the categorical variables in my dataframe using get_dummies(). The question is: is there a way to get the column headers of the encoded categorical columns created by get_dummies()?
The hard way to do this would be to extract a list of all categorical variables in the dataframe, then append the different text labels associated to each categorical variable to the corresponding column headers. I wonder if there is an easier way to achieve the same end.
I think the way that should work with all the different uses of get_dummies would be:
#example data
import pandas as pd
df = pd.DataFrame({'P': ['p', 'q', 'p'], 'Q': ['q', 'p', 'r'],
'R': [2, 3, 4]})
dummies = pd.get_dummies(df)
#get column names that were not in the original dataframe
new_cols = dummies.columns[~dummies.columns.isin(df.columns)]
new_cols gives:
Index(['P_p', 'P_q', 'Q_p', 'Q_q', 'Q_r'], dtype='object')
I think the first column is the only column preserved when using get_dummies, so you could also just take the column names after the first column:
dummies.columns[1:]
which on this test data gives the same result:
Index(['P_p', 'P_q', 'Q_p', 'Q_q', 'Q_r'], dtype='object')
import pandas as pd
pd.DataFrame({"a":["a","b","c"],"d":[1,2,3]})
Given an array ["a","b","c","c"], I want it to use to map col "a", and get output [1,2,3,3] which is from column "d". Is there a short way to do this without iterating the rows?
Use Series.reindex with index by a converted to index by DataFrame.set_index:
a = ["a","b","c","c"]
L = df.set_index('a').reindex(a)['d'].tolist()
print (L)
[1, 2, 3, 3]
I have a 2 dataframes each with 2 columns (named the same in both df's) and I want to add them together to make a third column.
df1['C']=df1[['A','B']].sum(axis=1)
df1['D']=df1[['E','G']].sum(axis=1)
df2['C']=df2[['A','B']].sum(axis=1)
df2['D']=df2[['E','G']].sum(axis=1)
However in reality its more complicated than this. So can I put these in a dictionary and loop?
I'm still figuring out how to structure dictionarys for this type of problem, so any advice would be great.
Here's what I'm trying to do:
all_dfs=[df1,df2]
for df in all_dfs:
dict={Out=['C'], in=['A','B]
Out2=['D'], in2=['E','G]
}
for i in dict:
df[i]=df[['i[1....
I'm a bit lost in how to build this last bit
First change dictionary name because dict is python code word, then change it by key with output column and value by list of input columns and last loop by items() method:
d= {'C':['A','B'],'D': ['E','G']}
for k, v in d.items():
#checking key and value of dict
print (k)
print (v)
df[k]=df[v].sum(axis=1)
EDIT:
Here is simplier working with dictionary of DataFrames, use sum and last create anoter dictionary of DataFrames:
all_dfs= {'first': df1, 'second':df2}
out = {}
for name, df in all_dfs.items():
d= {'C':['A','B'],'D': ['E','G']}
for k, v in d.items():
df[k]=df[v].sum(axis=1)
#fill empty dict by name
out[name] = df
print (out)
print (out['first'])
print (out['second'])
I'm trying to save a named tuple n=NamedTuple(value1='x'=, value2='y') in a row of a pandas dataframe.
The problem is that the named tuple is showing a length of 2 because it has 2 parameters in my case (value1 and value2), so it doesn't fit it into a single cell of the dataframe.
How can I achieve that the named tuple is written into every call of a row of a dataframe?
df['columnd1']=n
an example:
from collections import namedtuple
import pandas as pd
n = namedtuple("test", ['param1', 'param2'])
n1 = n(param1='1', param2='2')
df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
df['nt'] = n1
print(df)
I don't really understand what you're trying to do, but if you want to put that named tuple in every row of a new column (i.e. like a scalar) then you can't rely on broadcasting but should instead replicate it yourself:
df['nt'] = [n1 for _ in range(df.shape[0])]
I have a variable
var=[name1,name2]
I have a dataframe also in a list
df= [df1, df2]
How do i assign df1 to name1 and df2 to name2 and so on.
If I understand correctly, assuming the lengths of both lists are the same you just iterate over the indices of both lists and just assign them, example:
In [412]:
name1,name2 = None,None
var=[name1,name2]
df1, df2 = 1,2
df= [df1, df2]
for x in range(len(var)):
var[x] = df[x]
var
Out[412]:
[1, 2]
If your variable list is storing strings then I would not make variables from those strings (see How do I create a variable number of variables?) and instead create a dict:
In [414]:
var=['name1','name2']
df1, df2 = 1,2
df= [df1, df2]
d = dict(zip(var,df))
d
Out[414]:
{'name1': 1, 'name2': 2}
To answer your question, you can do this by:
for i in zip(var, df):
globals()[i[0]] = i[1]
And then access your variables.
But proceeding this way is bad. You're like launching a dog in your global environment. It's better to keep control about what you handle, keep your dataframe in a list or dictionary.