PyQt QPushButton clicked connect lambda can't get correct value [duplicate] - pyqt5

This question already has answers here:
tkinter creating buttons in for loop passing command arguments
(3 answers)
Closed 2 years ago.
Problemset:
Context Menu should show filter variables dynamically and execute a function with parameters defined inside the callback.
Generic descriptions show properly, but function call is always executed with last set option.
What I have tried:
#!/usr/bin/env python
import Tkinter as tk
import ttk
from TkTreectrl import MultiListbox
class SomeClass(ttk.Frame):
def __init__(self, *args, **kwargs):
ttk.Frame.__init__(self, *args, **kwargs)
self.pack(expand=True, fill=tk.BOTH)
self.grid_rowconfigure(0, weight=1)
self.grid_columnconfigure(0, weight=1)
self.View=MultiListbox(self)
__columns=("Date","Time","Type","File","Line","-","Function","Message")
self.View.configure(columns=__columns, expandcolumns=(0,0,0,0,0,0,0,1))
self.View.bind("", self.cell_context)
self.View.grid(row=0, column=0, sticky=tk.NW+tk.SE)
self.__recordset = []
self.__recordset_filtered = False
#Some dummy values
self.__recordset.append(["Date", "Time", "INFO", "File", "12", "-", "Function", "Message Info"])
self.__recordset.append(["Date", "Time", "DEBUG", "File", "12", "-", "Function", "Message Info"])
self.__recordset.append(["Date", "Time", "WARNING", "File", "12", "-", "Function", "Message Info"])
self.__refresh()
def cleanView(self):
self.View.delete(0, tk.END)
def __refresh(self):
self.cleanView()
for row in self.__recordset:
self.View.insert(tk.END, *row)
def filter_records(self, column, value):
print("Filter Log Recordset by {column} and {value}".format(**locals()))
# Filter functionality works as expected
# [...]
def cell_context(self, event):
__cMenu=tk.Menu(self, tearoff=0)
if self.__recordset_filtered:
__cMenu.add_command(label="Show all", command=lambda: filter_records(0, ""))
else:
column=2
options=["INFO", "WARNING", "DEBUG"]
for i in range(len(options)):
option=options[i]
__cMenu.add_command(label="{}".format(option), command=lambda: self.filter_records(column, option))
# Also tried using for option in options here with same result as now
__cMenu.post(event.x_root, event.y_root)
if __name__=="__main__":
root=tk.Tk()
app=SomeClass(root)
root.mainloop()
The current output i get is:
Filter Log Recordset by 2 and DEBUG
No matter which of the three options i choose. I assume it has sth to do with the garbage collection that only the last option remains but i cannot figure out how to avoid this.
Any help is recommended.

Please read about minimal examples. Without reading your code, I believe you have run into a well known issue addressed in previous questions and answers that needs 2 lines to illustrate. Names in function bodies are evaluated when the function is executed.
funcs = [lambda: i for i in range(3)]
for f in funcs: print(f())
prints '2' 3 times because the 3 functions are identical and the 'i' in each is not evaluated until the call, when i == 2. However,
funcs = [lambda i=i:i for i in range(3)]
for f in funcs: print(f())
makes three different functions, each with a different captured value, so 0, 1, and 2 are printed. In your statement
__cMenu.add_command(label="{}".format(option),
command=lambda: self.filter_records(column, option))
add option=option before : to capture the different values of option. You might want to rewrite as
lambda opt=option: self.filter_records(column, opt)
to differentiate the loop variable from the function parameter. If column changed within the loop, it would need the same treatment.

Closures in Python capture variables, not values. For example consider:
def f():
x = 1
g = lambda : x
x = 2
return g()
What do you expect the result of calling f() to be? The correct answer is 2, because the lambda f captured the variable x, not its value 1 at the time of creation.
Now if for example we write:
L = [(lambda : i) for i in range(10)]
we created a list of 10 different lambdas, but all of them captured the same variable i, thus calling L[3]() the result will be 9 because the value of variable i at the end of the iteration was 9 (in Python a comprehension doesn't create a new binding for each iteration; it just keeps updating the same binding).
A "trick" that can be seen often in Python when capturing the value is the desired semantic is to use default arguments. In Python, differently from say C++, default value expressions are evaluated at function definition time (i.e. when the lambda is created) and not when the function is invoked. So in code like:
L = [(lambda j=i: j) for i in range(10)]
we're declaring a parameter j and setting as default the current value of i at the time the lambda was created. This means that when calling e.g. L[3]() the result will be 3 this time because of the default value of the "hidden" parameter (calling L[3](42) will return 42 of course).
More often you see the sightly more confusing form
lambda i=i: ...
where the "hidden" parameter has the same name as the variable of which we want to capture the value of.

I know I am late, but I found a messy workaround which gets the job done (tested in Python 3.7)
If you use a double lambda (like I said, very messy) you can preserve the value, like so:
Step 1: Create the nested lambda statement:
send_param = lambda val: lambda: print(val)
Step 2: Use the lambda statement:
send_param(i)
The send_param method returns the inner most lambda (lambda: print(val)) without executing the statement, until you call the result of send_param which takes no arguments, for example:
a = send_param(i)
a()
Only the second line will execute the print statement.

Related

Compact way to save JuMP optimization results in DataFrames

I would like to save all my variables and dual variables of my finished lp-optimization in an efficient manner. My current solution works, but is neither elegant nor suited for larger optimization programs with many variables and constraints because I define and push! every single variable into DataFrames separately. Is there a way to iterate through the variables using all_variables() and all_constraints() for the duals? While iterating, I would like to push the results into DataFrames with the variable index name as columns and save the DataFrame in a Dict().
A conceptual example would be for variables:
Result_vars = Dict()
for vari in all_variables(Model)
Resul_vars["vari"] = DataFrame(data=[indexval(vari),value(vari)],columns=[index(vari),"Value"])
end
An example of the appearance of the declared variable in JuMP and DataFrame:
#variable(Model, p[t=s_time,n=s_n,m=s_m], lower_bound=0,base_name="Expected production")
And Result_vars[p] shall approximately look like:
t,n,m,Value
1,1,1,50
2,1,1,60
3,1,1,145
Presumably, you could go something like:
x = all_variables(model)
DataFrame(
name = variable_name.(x),
Value = value.(x),
)
If you want some structure more complicated, you need to write custom code.
T, N, M, primal_solution = [], [], [], []
for t in s_time, n in s_n, m in s_m
push!(T, t)
push!(N, n)
push!(M, m)
push!(primal_solution, value(p[t, n, m]))
end
DataFrame(t = T, n = N, m = M, Value = primal_solution)
See here for constraints: https://jump.dev/JuMP.jl/stable/constraints/#Accessing-constraints-from-a-model-1. You want something like:
for (F, S) in list_of_constraint_types(model)
for con in all_constraints(model, F, S)
#show dual(con)
end
end
Thanks to Oscar, I have built a solution that could help to automatize the extraction of results.
The solution is build around a naming convention using base_name in the variable definition. One can copy paste the variable definition into base_name followed by :. E.g.:
#variable(Model, p[t=s_time,n=s_n,m=s_m], lower_bound=0,base_name="p[t=s_time,n=s_n,m=s_m]:")
The naming convention and syntax can be changed, comments can e.g. be added, or one can just not define a base_name. The following function divides the base_name into variable name, sets (if needed) and index:
function var_info(vars::VariableRef)
split_conv = [":","]","[",","]
x_str = name(vars)
if occursin(":",x_str)
x_str = replace(x_str, " " => "") #Deletes all spaces
x_name,x_index = split(x_str,split_conv[1]) #splits raw variable name+ sets and index
x_name = replace(x_name, split_conv[2] => "")
x_name,s_set = split(x_name,split_conv[3])#splits raw variable name and sets
x_set = split(s_set,split_conv[4])
x_index = replace(x_index, split_conv[2] => "")
x_index = replace(x_index, split_conv[3] => "")
x_index = split(x_index,split_conv[4])
return (x_name,x_set,x_index)
else
println("Var base_name not properly defined. Special Syntax required in form var[s=set]: ")
end
end
The next functions create the columns and the index values plus columns for the primal solution ("Value").
function create_columns(x)
col_ind=[String(var_info(x)[2][col]) for col in 1:size(var_info(x)[2])[1]]
cols = append!(["Value"],col_ind)
return cols
end
function create_index(x)
col_ind=[String(var_info(x)[3][ind]) for ind in 1:size(var_info(x)[3])[1]]
index = append!([string(value(x))],col_ind)
return index
end
function create_sol_matrix(varss,model)
nested_sol_array=[create_index(xx) for xx in all_variables(model) if varss[1]==var_info(xx)[1]]
sol_array=hcat(nested_sol_array...)
return sol_array
end
Finally, the last function creates the Dict which holds all results of the variables in DataFrames in the previously mentioned style:
function create_var_dict(model)
Variable_dict=Dict(vars[1]
=>DataFrame(Dict(vars[2][1][cols]
=>create_sol_matrix(vars,model)[cols,:] for cols in 1:size(vars[2][1])[1]))
for vars in unique([[String(var_info(x)[1]),[create_columns(x)]] for x in all_variables(model)]))
return Variable_dict
end
When those functions are added to your script, you can simply retrieve all the solutions of the variables after the optimization by calling create_var_dict():
var_dict = create_var_dict(model)
Be aware: they are nested functions. When you change the naming convention, you might have to update the other functions as well. If you add more comments you have to avoid using [, ], and ,.
This solution is obviously far from optimal. I believe there could be a more efficient solution falling back to MOI.

Airflow: Best way to pass BigQuery result as XCom [duplicate]

I'm using Airflow 1.8.1 and I want to push the result of a sql request from PostgreOperator.
Here's my tasks:
check_task = PostgresOperator(
task_id='check_task',
postgres_conn_id='conx',
sql="check_task.sql",
xcom_push=True,
dag=dag)
def py_is_first_execution(**kwargs):
value = kwargs['ti'].xcom_pull(task_ids='check_task')
print 'count ----> ', value
if value == 0:
return 'next_task'
else:
return 'end-flow'
check_branch = BranchPythonOperator(
task_id='is-first-execution',
python_callable=py_is_first_execution,
provide_context=True,
dag=dag)
and here is my sql script:
select count(1) from table
when i check the xcom value from check_task it retrieves none value.
If i'm correct, airflow automatically pushes to xcom when a query returns a value. However, when you look at the code of the postgresoperator you see that it has an execute method that calls the run method of the PostgresHook (extension of dbapi_hook). Both methods do not return anything, as such it pushes nothing to xcom.
What we did to fix this is create a CustomPostgresSelectOperator, a copy of the PostgresOperator, but instead of 'hook.run(..)' do 'return hook.get_records(..)'.
Hope that helps you.
Finally, I created a new Sensor ExecuteSqlOperator in the plugin manager under $AIRFLOW_HOME/plugins.
I used CheckOperator as an example and I modified the returned value: the basic running of this operator was exactly the reverse of what I needed.
Here's the of the default ExecuteSqlOperator:
CheckOperator
and here is my customized SqlSensor: ReverseSqlSensor
class SqlExecuteOperator(BaseOperator):
"""
Performs checks against a db. The ``CheckOperator`` expects
a sql query that will return a single row.
Note that this is an abstract class and get_db_hook
needs to be defined. Whereas a get_db_hook is hook that gets a
single record from an external source.
:param sql: the sql to be executed
:type sql: string
"""
template_fields = ('sql',)
template_ext = ('.hql', '.sql',)
ui_color = '#fff7e6'
#apply_defaults
def __init__(
self, sql,
conn_id=None,
*args, **kwargs):
super(SqlExecuteOperator, self).__init__(*args, **kwargs)
self.conn_id = conn_id
self.sql = sql
def execute(self, context=None):
logging.info('Executing SQL statement: ' + self.sql)
records = self.get_db_hook().get_first(self.sql)
logging.info("Record: " + str(records))
records_int = int(records[0])
print (records_int)
return records_int
def get_db_hook(self):
return BaseHook.get_hook(conn_id=self.conn_id)

generating DataFrames in for loop in Scala Spark cause out of memory

I'm generating small dataFrames in for loop. At each round of for loop, I pass the generated dataFrame to a function which returns double. This simple process (which I thought could be easily taken care of by garbage collector) blow up my memory. When I look at Spark UI at each round of for loop it adds a new "SQL{1-500}" (my loop runs 500 times). My question is how to drop this sql object before generating a new one?
my code is something like this:
Seq.fill(500){
val data = (1 to 1000).map(_=>Random.nextInt(1000))
val dataframe = createDataFrame(data)
myFunction(dataframe)
dataframe.unpersist()
}
def myFunction(df: DataFrame)={
df.count()
}
I tried to solve this problem by dataframe.unpersist() and sqlContext.clearCache() but neither of them worked.
You have two places where I suspect something fishy is happening:
in the definition of myFunction : you really need to put the = before the body of the definition. I had typos like that compile, but produce really weird errors (note I changed your myFunction for debugging purposes)
it is better to fill your Seq with something you know and then apply foreach or some such
(You also need to replace random.nexInt with Random.nextInt, and also, you can only create a DataFrame from a Seq of a type that is a subtype of Product, such as tuple, and need to use sqlContext to use createDataFrame)
This code works with no memory issues:
Seq.fill(500)(0).foreach{ i =>
val data = {1 to 1000}.map(_.toDouble).toList.zipWithIndex
val dataframe = sqlContext.createDataFrame(data)
myFunction(dataframe)
}
def myFunction(df: DataFrame) = {
println(df.count())
}
Edit: parallelizing the computation (across 10 cores) and returning the RDD of counts:
sc.parallelize(Seq.fill(500)(0), 10).map{ i =>
val data = {1 to 1000}.map(_.toDouble).toList.zipWithIndex
val dataframe = sqlContext.createDataFrame(data)
myFunction(dataframe)
}
def myFunction(df: DataFrame) = {
df.count()
}
Edit 2: the difference between declaring function myFunction with = and without = is that the first is (a usual) function definition, while the other is procedure definition and is only used for methods that return Unit. See explanation. Here is this point illustrated in Spark-shell:
scala> def myf(df:DataFrame) = df.count()
myf: (df: org.apache.spark.sql.DataFrame)Long
scala> def myf2(df:DataFrame) { df.count() }
myf2: (df: org.apache.spark.sql.DataFrame)Unit

Import a dictionary into the current scope as variables

I have a .mat file in which I put data previously processed. When I perform
dict = scipy.io.loadmat('training_data.mat')
I get back a dict that is like this
{'encoders' : ......, 'decoders' : ........, 'stuff' : .....}
I want to selectively import the encoders and decoders variables into my current scope. The effect is the same as:
encoders = dict['encoders']
decoders = dict['decoders']
How do I cleanly do this without typing 10-15 lines?
You could import a dictionary d into the global scope using
globals().update(d)
The same thing is impossible for local scopes, since modifying the dictionary returned by locals() results in undefined behaviour.
A slightly hacky trick you could use in this situation is to import the names into the dictionary of an on-the-fly created type:
d = {"encoders": 1, "decoders": 2}
t = type("", (), d)
print t.encoders
print t.decoders
This will at least be slightly more convenient than using d["decoders"] etc.
Alternatively, you could use exec statements to create your variables:
d = {"encoders": 1, "decoders": 2}
for k, v in d.iteritems():
exec k + " = v"
This could also be done selectively.

overriding a method in R, using NextMethod

how dows this work in R...
I am using a package (zoo 1.6-4) that defines a S3 class for time series sets.
I am writing a derived class where I want to override a few methods and can't get past this one:[.zoo!
in my derived class rows are indexed by timestamp, like in zoo, but differently from zoo, I allow only POSIXct values in the index. my users will be selecting columns all of the time, while slicing series only occasionally so I want to offer obj[name] instead of obj[, name].
my objects have class c("delftfews", "zoo").
but...
how do I override a method?
I tried this:
"[.delftfews" <- function(x, i, j, drop=TRUE, ...) {
if (missing(i)) return(NextMethod())
if (all(class(i) == "character") && missing(j)) {
return(NextMethod('[', x=x, i=1:NROW(x), j=i, drop=drop, ...))
}
NextMethod()
}
but I get this error: Error in rval[i, j, drop = drop., ...] : incorrect number of dimensions.
I have solved by editing the source from zoo: I removed those ..., but I don't get why that works. anybody can explain what is going on here?
The problem is that with the above definition of [.delftfews this code:
library(zoo)
z <- structure(zoo(cbind(a = 1:3, b = 4:6)), class = c("delftfews", "zoo"))
z["a"]
# generates this call: `[.zoo`(x = 1:6, i = 1:3, j = "a", drop = TRUE, z, "a")
Your code does work as is if you write the call like this:
z[j = "a"]
# generates this call: `[.zoo`(x = z, j = "a")
I think what you want is to change the relevant line in [.delftfews to this:
return(NextMethod(.Generic, object = x, i = 1:NROW(x), drop = drop))
# z["a"] now generates this call: `[.zoo`(x = z, i = 1:3, j = "a", drop = TRUE)
A point of clarification: allowing only POSIXct index values does not allow indexing columns by name only. I'm not sure how you arrived at that conclusion.
You're overriding zoo correctly, but I think you misunderstand NextMethod. The error is caused by if (missing(i)) return(NextMethod()), which calls [.zoo if i is missing, but [.zoo requires i because zoo's internal data structure is a matrix. Something like this should work:
if (missing(i)) i <- 1:NROW(x)
though I'm not sure if you have to explicitly pass this new i to NextMethod...
You may be interested in the xts package, if you haven't already taken a look at it.