Suppose that I have the DataFrames
In [1]: a=pd.DataFrame([[1,2],[3,4],[5,6],[7,8]],
...: index=pd.MultiIndex.from_product([('A','B'),('d','e')]))
In [2]: a
Out[2]:
0 1
A d 1 2
e 3 4
B d 5 6
e 7 8
In [3]: b=pd.DataFrame([[9,10],[11,12]],index=('d','e'))
In [4]: b
Out[4]:
0 1
d 9 10
e 11 12
and I want to append b to a, with the subindex C, producing the
DataFrame
0 1
A d 1 2
e 3 4
B d 5 6
e 7 8
C d 9 10
e 11 12
I tried
In [5]: a.loc['C'] = b
but got
TypeError: 'int' object is not iterable
How do I do it?
Assign a new value to b , then set_index and swaplevel before append to a
a.append(b.assign(k='C').set_index('k',append=True).swaplevel(0,1))
Out[33]:
0 1
A d 1 2
e 3 4
B d 5 6
e 7 8
C d 9 10
e 11 12
First update b's index to match the same levels as a, then concat:
b.index = pd.MultiIndex.from_arrays([('C','C'), ('d','e')])
pd.concat([a,b]))])
If wanna step-by-step;
df2 = pd.concat([a,b], ignore_index=True)
df2['i0'] = a.index.get_level_values(0).tolist() + ['C']*len(b)
df2['i1'] = a.index.get_level_values(0).union(b.index)
df2.set_index(['i0', 'i1'])
Outputs
0 1
i0 i1
A A 1 2
A 3 4
B B 5 6
B 7 8
C d 9 10
e 11 12
Related
I would like to impute negative values of Dask Dataframe, with pandas i use this code:
df.loc[(df.column_name < 0),'column_name'] = 0
I think need dask.dataframe.Series.clip_lower:
ddf['B'] = ddf['B'].clip_lower(0)
Sample:
import pandas as pd
df = pd.DataFrame({'F':list('abcdef'),
'B':[-4,5,4,-5,5,4],
'A':list('aaabbb')})
print (df)
A B F
0 a -4 a
1 a 5 b
2 a 4 c
3 b -5 d
4 b 5 e
5 b 4 f
from dask import dataframe as dd
ddf = dd.from_pandas(df, npartitions=3)
#print (ddf)
ddf['B'] = ddf['B'].clip_lower(0)
print (ddf.compute())
A B F
0 a 0 a
1 a 5 b
2 a 4 c
3 b 0 d
4 b 5 e
5 b 4 f
For more general solution use dask.dataframe.Series.mask`:
ddf['B'] = ddf['B'].mask(ddf['B'] > 0, 3)
print (ddf.compute())
A B F
0 a -4 a
1 a 3 b
2 a 3 c
3 b -5 d
4 b 3 e
5 b 3 f
for my graduation project, I would like to remove duplicate rows and keep only a row where column b and c are equal for the value in column a. I tried a lot of things, groupby, Merge combinations and duplicates, but nothing worked out till now. Can you please help me? Many thanks!
input:
a b c
0 1 A B
1 1 A A
2 1 A C
3 2 B A
4 2 B B
result:
a b c
1 1 A A
4 2 B B
I believe you need:
print (df)
a b c
0 1 A B
1 1 A A
2 1 A C
3 2 B A
4 2 B B
5 3 C C
6 4 C NaN
7 4 C E
7 5 NaN E
Replace NaNs by forward and back filling:
df1 = df[['b','c']].bfill(axis=1).ffill(axis=1)
print (df1)
b c
0 A B
1 A A
2 A C
3 B A
4 B B
5 C C
6 C C
7 C E
7 E E
Check condition in df1 and because same index is possible filter df:
df = df[df1['b'] == df1['c']]
print (df)
a b c
1 1 A A
4 2 B B
5 3 C C
6 4 C NaN
7 5 NaN E
I have two DataFrame objects which I want to apply an element-wise multiplication on each row onto:
df_prob_wc.shape # (3505, 13)
df_prob_c.shape # (13, 1)
I thought I could do it with DataFrame.apply()
df_prob_wc.apply(lambda x: x.multiply(df_prob_c), axis=1)
which gives me:
TypeError: ("'int' object is not iterable", 'occurred at index $')
or with
df_prob_wc.apply(lambda x: x * df_prob_c, axis=1)
which gives me:
TypeError: 'int' object is not iterable
But it's not working.
However, I can do this:
df_prob_wc.apply(lambda x: x * np.asarray([1,2,3,4,5,6,7,8,9,10,11,12,13]), axis=1)
What am I doing wrong here?
It seems you need multiple by Series created with df_prob_c by iloc:
df_prob_wc = pd.DataFrame({'A':[1,2,3],
'B':[4,5,6],
'C':[7,8,9],
'D':[1,3,5],
'E':[5,3,6],
'F':[7,4,3]})
print (df_prob_wc)
A B C D E F
0 1 4 7 1 5 7
1 2 5 8 3 3 4
2 3 6 9 5 6 3
df_prob_c = pd.DataFrame([[4,5,6,1,2,3]])
#for align data same columns in both df
df_prob_c.index = df_prob_wc.columns
print (df_prob_c)
0
A 4
B 5
C 6
D 1
E 2
F 3
print (df_prob_wc.shape)
(3, 6)
print (df_prob_c.shape)
(6, 1)
print (df_prob_c.iloc[:,0])
A 4
B 5
C 6
D 1
E 2
F 3
Name: 0, dtype: int64
print (df_prob_wc.mul(df_prob_c.iloc[:,0], axis=1))
A B C D E F
0 4 20 42 1 10 21
1 8 25 48 3 6 12
2 12 30 54 5 12 9
Another solution is multiple by numpy array, only need [:,0] for select:
print (df_prob_wc.mul(df_prob_c.values[:,0], axis=1))
A B C D E F
0 4 20 42 1 10 21
1 8 25 48 3 6 12
2 12 30 54 5 12 9
And another solution with DataFrame.squeeze:
print (df_prob_wc.mul(df_prob_c.squeeze(), axis=1))
A B C D E F
0 4 20 42 1 10 21
1 8 25 48 3 6 12
2 12 30 54 5 12 9
I have a pandas dataframe df with columns [a, b, c, d, e, f]. I want to perform a group by on df. I can best describe what it's supposed to do in SQL:
SELECT a, b, min(c), min(d), max(e), sum(f)
FROM df
GROUP BY a, b
How do I do this group by using pandas on my dataframe df?
consider df:
a b c d e f
1 1 2 5 9 3
1 1 3 3 4 5
2 2 4 7 4 4
2 2 5 3 8 8
I expect the result to be:
a b c d e f
1 1 2 3 9 8
2 2 4 3 8 12
use agg
df = pd.DataFrame(
dict(
a=list('aaaabbbb'),
b=list('ccddccdd'),
c=np.arange(8),
d=np.arange(8),
e=np.arange(8),
f=np.arange(8),
)
)
funcs = dict(c='min', d='min', e='max', f='sum')
df.groupby(['a', 'b']).agg(funcs).reset_index()
a b c e f d
0 a c 0 1 1 0
1 a d 2 3 5 2
2 b c 4 5 9 4
3 b d 6 7 13 6
with your data
a b c e f d
0 1 1 2 9 8 3
1 2 2 4 8 12 3
I have a DataFrame with columns [A, B, C, D, E, F, G, H].
An index has been made with columns [D, G, H]:
>>> print(dgh_columns)
Index(['D', 'G', 'H'], dtype='object')
How can I retrieve the original DataFrame without the columns D, G, H ?
Is there an index subset operation?
Ideally, this would be:
df[df.index - dgh_columns]
But this doesn't seem to work
I think you can use Index.difference:
df[df.columns.difference(dgh_columns)]
Sample:
df = pd.DataFrame({'A':[1,2,3],
'B':[4,5,6],
'C':[7,8,9],
'D':[1,3,5],
'E':[7,8,9],
'F':[1,3,5],
'G':[5,3,6],
'H':[7,4,3]})
print (df)
A B C D E F G H
0 1 4 7 1 7 1 5 7
1 2 5 8 3 8 3 3 4
2 3 6 9 5 9 5 6 3
dgh_columns = pd.Index(['D', 'G', 'H'])
print (df[df.columns.difference(dgh_columns)])
A B C E F
0 1 4 7 7 1
1 2 5 8 8 3
2 3 6 9 9 5
Numpy solution with numpy.setxor1d or numpy.setdiff1d:
dgh_columns = pd.Index(['D', 'G', 'H'])
print (df[np.setxor1d(df.columns, dgh_columns)])
A B C E F
0 1 4 7 7 1
1 2 5 8 8 3
2 3 6 9 9 5
dgh_columns = pd.Index(['D', 'G', 'H'])
print (df[np.setdiff1d(df.columns, dgh_columns)])
A B C E F
0 1 4 7 7 1
1 2 5 8 8 3
2 3 6 9 9 5
use drop
df.drop(list('DGH'), axis=1)
df = pd.DataFrame({'A':[1,2,3],
'B':[4,5,6],
'C':[7,8,9],
'D':[1,3,5],
'E':[7,8,9],
'F':[1,3,5],
'G':[5,3,6],
'H':[7,4,3]})
df.drop(list('DGH'), 1)