I need to select column names where the count is greater than 2. I have this dataset:
Index | col_1 | col_2 | col_3 | col_4
-------------------------------------
0 | 5 | NaN | 4 | 2
1 | 2 | 2 | NaN | 2
2 | NaN | 3 | NaN | 1
3 | 3 | NaN | NaN | 1
The expected result is a list: ['col_1', 'col_4']
When I use
df.count() > 2
I get
col_1 True
col_2 False
col_3 False
col_4 True
Length: 4, dtype: bool
This is the code for testing
import pandas as pd
import numpy as np
data = {'col_1': [5, 2, np.NaN, 3],
'col_2': [np.NaN, 2, 3, np.NaN],
'col_3': [4, np.NaN, np.NaN, np.NaN],
'col_4': [2, 2, 1,1]}
frame = pd.DataFrame(data)
frame.count() > 2
You can do it this way.
import pandas as pd
import numpy as np
data = {'col_1': [5, 2, np.NaN, 3],
'col_2': [np.NaN, 2, 3, np.NaN],
'col_3': [4, np.NaN, np.NaN, np.NaN],
'col_4': [2, 2, 1,1]}
frame = pd.DataFrame(data)
expected_list = []
for col in list(frame.columns):
if frame[col].count() > 2:
expected_list.append(col)
Use dict can easily solve this:
frame[[key for key, value in dict(frame.count() > 2).items() if value]]
Try:
(df.columns)[(df.count() > 2).values].to_list()
Related
As an example, how do I convert df to df1, by gathering rows into matrices based on shared values in a specific column tidx?
>>> df = pd.DataFrame({'col3':[[1,40],[2,50],[3,60],[4,70]], 'tidx':[21,22,23,21]})
>>> df['col3'] = df['col3'].apply(np.array)
>>> df
col3 tidx
0 [1, 40] 21
1 [2, 50] 22
2 [3, 60] 23
3 [4, 70] 21
>>> df1 = pd.DataFrame({'col3':[[[1,40],[4,70]],[[2,50]],[[3,60]]], 'tidx':[21,22,23]})
>>> df1['col3'] = df1['col3'].apply(np.array)
>>> df1
col3 tidx
0 [[1, 40], [4, 70]] 21
1 [[2, 50]] 22
2 [[3, 60]] 23
You can use .groupby and then apply list function as shown in example below.
df = pd.DataFrame({'col3':[[1,40],[2,50],[3,60],[4,70]], 'tidx':[21,22,23,21]})
df1 = df.groupby('tidx')['col3'].apply(list).reset_index()
Supposing that i have a simple pd.DataFrame like so:
d = {'col1': [1, 20], 'col2': [3, 40], 'col3': [5, 50]}
df = pd.DataFrame(data=d)
df
col1 col2 col4
0 1 3 5
1 20 40 60
is there a way to convert this to nasted pandas Dataframe (df_new) , so as when i call df_new.values[0] taking as ouptut:
array(
[0 1
1 3
2 5
Length: 3, dtype: int], dtype=object)
I still don't think I understand the exact requirement, but here is something:
One way of getting the desired output is this:
>>> pd.Series(df.T[0].values)
0 1
1 3
2 5
dtype: int64
If you want to have these as 2d arrays:
>>> np.array(pd.DataFrame(df.T[0].values).reset_index())
array([[0, 1],
[1, 3],
[2, 5]])
>>> np.array(pd.DataFrame(df.T[1].values).reset_index())
array([[ 0, 20],
[ 1, 40],
[ 2, 50]])
I have the following dataframe with timeseries data:
df = pd.DataFrame(columns = ['id', 'value'])
df['value'] =[9, 16, 10, 12, 11, 14]
df['id'] = [1, 1, 1, 2, 2, 2]
For each timeseries (defined by column 'id' I want to calculate the variance to find timeseries that do not change at all or only very little.
The final dataframe should look like this:
df_end = pd.DataFrame(columns = ['id','value', 'var'])
df_end['value'] =[9, 16, 10, 12, 11, 14]
df_end['id'] = [1, 1, 1, 2, 2, 2]
df_end['var'] = [21, 21, 21, 2.3, 2.3, 2.3]
I tried:
df.groupby(df['id']).var()
which gives me the values, but I couldn't put it into the df in the right form. I am sure, there is a handy function for this that I don't know about yet!
Thanks for helping out!
Use GroupBy.transform with specify column value:
df['var'] = df.groupby('id')['value'].transform('var')
print (df)
id value var
0 1 9 14.333333
1 1 16 14.333333
2 1 10 14.333333
3 2 12 2.333333
4 2 11 2.333333
5 2 14 2.333333
How to convert dictionary to dataframe with default index and column names
dictionary d = {0: [1, 'Sports', 222], 1: [2, 'Tools', 11], 2: [3, 'Clothing', 23]}
df
id type value
0 1 Sports 222
1 2 Tools 11
2 3 Clothing 23
Use DataFrame.from_dict with orient='index' parameter:
d = {0: [1, 'Sports', 222], 1: [2, 'Tools', 11], 2: [3, 'Clothing', 23]}
df = pd.DataFrame.from_dict(d, orient='index', columns=['id','type','value'])
print (df)
id type value
0 1 Sports 222
1 2 Tools 11
2 3 Clothing 23
I'm trying to group the following data:
>>> a=[{'A': 1, 'B': 2, 'C': 3, 'D':4, 'E':5, 'F':6},{'A': 2, 'B': 3, 'C': 4, 'D':5, 'E':6, 'F':7},{'A': 3, 'B': 4, 'C': 5, 'D':6, 'E':7, 'F':8}]
>>> df = pd.DataFrame(a)
>>> df
A B C D E F
0 1 2 3 4 5 6
1 2 3 4 5 6 7
2 3 4 5 6 7 8
With the Following Dictionary:
dict={'A':1,'B':1,'C':1,'D':2,'E':2,'F':2}
such that
df.groupby(dict).groups
Will output
{1:['A','B','C'],2:['D','E','F']}
Needed to add the axis argument to groupby:
>>> grouped = df.groupby(groupDict,axis=1)
>>> grouped.groups
{1: ['A', 'B', 'C'], 2: ['D', 'E', 'F']}