Suppose I have the following df:
df = pd.DataFrame({
'col1':['x1','x2','x3'],
'col2':['y1','y2','y3'],
'col3':['z1','z2','z3'],
'col4':['a1','b2','c3']
})
and a list of elements:
l = ['x1','x2','y3']
I want to search elements of l in all the columns of my df, as it stands from my list x1 and x2 appear in col1 and y3 is in col2, so I did:
df.loc[df['col1'].apply(lambda x: True if any(i in x for i in l) else False)|
df['col2'].apply(lambda x: True if any(i in x for i in l) else False)]
which gives me
col1 col2 col3 col4
0 x1 y1 z1 a1
1 x2 y2 z2 b2
2 x3 y3 z3 c3
as desired but the above method needs me to make a | operator for each column. So I wonder how can I do this iteration over all columns efficiently without using | for every column?
A much, much more efficient way of doing this would be to use numpy broadcasting.
row_mask = (df.to_numpy() == l[:, None, None]).sum(axis=0).any(axis=1)
filtered = df[row_mask]
Output:
>>> filtered
col1 col2 col3 col4
0 x1 y1 z1 a1
1 x2 y2 z2 b2
2 x3 y3 z3 c3
First, I want to point out that I didn't found the answer for my question here in stackoverflow nor in pandas documentation, so, if the question had been asked before, I'd appreciate a link for that thread.
I want to join two DataFrames as follows.
df1 =
key x y z
0 x0 y0 z0
1 x1 y1 z1
...
10 x10 y10 z10
df2 =
key w v u
0 w0 v0 u0
0 w0 v0 u0
0 w0 v0 u0
1 w1 v1 u1
1 w1 v1 u1
2 w2 v2 u2
3 w3 v3 u3
...
10 w10 v10 u10
10 w10 v10 u10
desired_df_output =
key x y z w v u
0 x0 y0 z0 w0 v0 u0
1 x1 y1 z1 w1 v1 u1
...
10 x10 y10 z10 w10 v10 u10
I've tried this df1.join(df2, how='inner', on='key'), but I get this error: TypeError: object of type 'NoneType' has no len().
Thanks
It seems df2 has duplicates values, so if you drop them using drop_duplicates method and merge with df1 from the right side, you get the desired outcome.
out = df1.merge(df2.drop_duplicates(), on='key')
Output:
key x y z w v u
0 0 x0 y0 z0 w0 v0 u0
1 1 x1 y1 z1 w1 v1 u1
2 10 x10 y10 z10 w10 v10 u10
import pandas as pd
df1 = pd.DataFrame({'k':[0, 1, 2, 3],
'x':['x0', 'x1', 'x2', 'x3'],
'y':['y0', 'y1', 'y2', 'y3'],
'z':['z0', 'z1', 'z2', 'z3']
})
df1.set_index('k', inplace=True)
df2 = pd.DataFrame({'k':[0, 0, 0, 1, 1, 1],
'v':['v0', 'v0', 'v0','v1', 'v1', 'v1',],
'w':['w0', 'w0', 'w0','w1', 'w1', 'w1',],
'u':['u0', 'u0', 'u0','u1', 'u1', 'u1',]
})
df2.set_index('k', inplace=True)
df_merged = pd.merge(df1, df2.drop_duplicates(), how='inner', left_index=True, right_index=True)
df_merged
x y z v w u
k
0 x0 y0 z0 v0 w0 u0
1 x1 y1 z1 v1 w1 u1
I need some guidance on how to approach for this problem. I've simplified a real life example and if you can help me crack this by giving me some guidance, it'll be awesome.
I've been looking at public optimization algorithms here (https://www.cvxpy.org/) but I'm a noob and I'm not able to figure out which algorithm would help me (or if I really need it).
Problem:
x1 to x4 are items with certain properties (a,b,c,y,z)
I have certain needs:
Parameter My Needs
a 150
b 800
c 80
My goal is get all optimal coefficient sets for x1 to x4 (can be
fractions) so as to get as much of a, b and c as possible to satisfy
needs from the smallest possible y.
These conditions must always be met:
1)Individual values of z should stay within threshold (between maximum and minimum for x1, x2, x3 and x4)
2)And Total y should be maintained within limits (y <=1000 & y>=2000)
To illustrate an example:
x1
Each x1 has the following properties
a 20 Minimum z 10 Maximum z 50
b 200
c 0
y 300
z 20
x2
Each x2 has the following properties
a 30 Minimum z 60 Maximum z 160
b 5
c 20
y 50
z 40
x3
Each x3 has the following properties
a 20 Minimum z 100 Maximum z 200
b 200
c 15
y 200
z 40
x4
Each x4 has the following properties
a 5 Minimum z 100 Maximum z 300
b 30
c 20
y 500
z 200
One possible arrangement can be (not the optimal solution as I'm trying to keep y as low as possible but above 1000 but to illustrate output)
2x1+2x2+1x3+0.5x4
In this instance:
Coeff x1 2
Coeff x2 2
Coeff x3 3
Coeff x4 0.5
This set of coefficients yields
Optimal?
total y 1550 Yes
total a 162.5 Yes
total b 1025 Yes
total c 95 Yes
z of x1 40 Yes
z of x2 80 Yes
z of x3 120 Yes
z of x4 100 Yes
Lowest y? No
Can anyone help me out?
Thanks!
I have the next LP problem
Maximize
1000 x1 + 500 x2 - 500 x5 - 250 x6
Subject To
c1: x1 + x2 - x3 - x4 = 0
c2: - x3 + x5 = 0
c3: - x4 + x6 = 0
With these Bounds
0 <= x1 <= 10
0 <= x2 <= 15
0 <= x5 <= 15
0 <= x6 <= 5
By solving this problem with Cplex dual algorithm I get an optimal solution of 6250. But checking the reduced costs of the variables I get the next results
Variable value reduced cost
1 10.0 500.0
1 0.0 -0.0
2 5.0 -0.0
3 5.0 -0.0
4 5.0 -0.0
5 5.0 250.0
Is it possible to have a positive reduced cost on a positive valued variable? Because the reduced cost value indicates how much the objective function coefficient on the corresponding variable must be improved before the value of the variable will be positive in the optimal solution, what does a positive reduced cost means on a postive valued variable?
Variable 1 is listed twice in the solution?
Note that you need to distinguish between nonbasic at lower bound and nonbasic at upper bound. The reduced cost indicates how much the objective can change when the corresponding bound changes by one unit.
Note also that most textbooks focus on the special case x >= 0 while practical solvers support both lower and upper bounds: L <= x <= U.
I have a problem which may be defined as:
max 5 x11 + 6 x12 + 2 x21 + 3 x22 st
st.
x11,x12,x21,x22 binary
x11 + x12 = 1
x21 + x22 = 1
-25 x1 x2 >= 0
I want to check only if both x11 and x21 are 1.
How can I linearize this constraint? Or is it possible to linearize?
Thank you very much.
"I want to check only if both x11 and x21 are 1." How about:
x11 + x21 >= 1
instead of
-25 x1 x2 >= 0