i'm trying to create a udf which contains other function, then i'm registering the function in spark to use that in select query. I need to pass the df(temp table) values as input to the function which is registered.
When tried executing the function getting error as follows
Error: PicklingError: Could not serialize object: Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063
def funcmain(a,b,c,lvl,x):
lvl1=lvl
x1=x
while lvl1!=lv1:
x1=x
if x1!= something:
x1= something
func1(a,b,c)
return x1
udf_fun = spark.register.udf("funcmain",funcmain)
Related
I am trying to use the SearchGlobalRequest API method for global, full-text search on Telegram, but I am not sure what to use for some of the arguments, especially the offset_peer parameter. When I do this:
try:
result = client(SearchGlobalRequest(
q=search_term,
filter=None,
min_date=datetime.datetime.strptime(min_date, '%Y-%m-%d'),
max_date=datetime.datetime.strptime(max_date, '%Y-%m-%d'),
offset_rate=-1,
# offset_peer=None,
offset_id=-1,
limit=10
))
except Exception as e:
print(e)
I get __init__() missing 1 required positional argument: 'offset_peer'.
When I try to pass None as offset_peer, I get Cannot cast NoneType to any kind of Peer. I am not trying to search in any specific channel, I just want to specify the start and end date and find all (or rather as many as possible) matching results.
I am using Telethon version 1.24.0.
Next code works for me:
from telethon.tl.functions.messages import SearchGlobalRequest
from telethon.tl.types import InputMessagesFilterEmpty, InputPeerEmpty
results = client(SearchGlobalRequest(
q='your_query_here',
filter=InputMessagesFilterEmpty(),
min_date=None,
max_date=None,
offset_rate=0,
offset_peer=InputPeerEmpty(),
offset_id=0,
limit=100,
folder_id=None
))
print(results.stringify())
I am failing to use readNetFromDarknet function for the reasons I do not understand. I am trying to use yolo3-spp with this configuration file https://github.com/pjreddie/darknet/blob/master/cfg/yolov3-spp.cfg.
import cv2
net = cv2.dnn.readNetFromDarknet("../models/yolov3-spp.weights", "../models/yolov3-spp.cfg")
However, this gives me the following error:
error: OpenCV(4.5.4) /tmp/pip-req-build-3129w7z7/opencv/modules/dnn/src/darknet/darknet_io.cpp:660: error: (-215:Assertion failed) separator_index < line.size() in function 'ReadDarknetFromCfgStream
Interestingly, if I use readNet instead of readNetFromDarknet, it seems to work just fine. Should I stick to using readNet instead and why readNetFromDarknet is not actually working?
Your problem in order function arguments. readNetFromDarknet waits first config file and second - weights. A readNet function can swap arguments if they have a wrong order.
So right code:
net = cv2.dnn.readNetFromDarknet("../models/yolov3-spp.cfg", "../models/yolov3-spp.weights")
I'm using DolphinDB subscribeTable but the error occurs:
I’d like to know what is wrong with my code. I wanted to subscribe to stream tables in DolphinDB and the code is:
csEngine1=createCrossSectionalEngine(name=sTb_Cs + "_eng",
dummyTable=objByName(sTb_join12),
keyColumn=`symbol,
triggeringPattern="perRow",
useSystemTime=false, timeColumn=`TimeStamp)
subscribeTable(tableName=sTb_join12, actionName="do"+sTb_Cs, offset=-1, handler=append!{csEngine1}, msgAsTable=true, hash=5, reconnect=true)
The UDF handler of another subscription to sTb_join12 also queries csEngine1.
def append_plan (csEngine1, candidates2, strategy, msg){
subscribeTable(tableName=sTb_join12, actionName="do" +strategy, offset=-1, handler=append_plan {csEngine1, candidates2, strategy}, msgAsTable=true, hash=6, reconnect=true)
Please tell why the error occurs and how I can fix it.
The problem is that the parameter in your function (append_plan in this case) was not mutable. With an immutable parameter, the object (csEngine1) will be set to readOnly when calling the function. If another thread writes this object (as shown in your first subscription), it will report the error.
Solution #1
Pass the same hash value in the two subscribeTable functions that subscribe to sTb_join12. With the same hash value, the two subscription tasks will be processed in the same thread in turn, which avoids concurrency and error reporting.
Solution #2
Simplify the two subscribeTable functions into one statement. ‘append!’ the cross-section engine with udf, and then continue with the above process to ensure the append operation and select query to the table CSEngines1 are sequentially processed. You can refer to the following example:
tradesCrossEngine008=createCrossSectionalEngine(name="tradesCrossEngine008", dummyTable=objByName(sTc_join12_testSTR), keyColumn=`Symbol, triggeringPattern=`perRow, useSystemTime=false, timeColumn=`TimeStamp)
def append_plan(msg){
getStreamEngine("tradesCrossEngine008").append!(msg)
select Symbol, rank(left_v1+left_v2) as rmk from getStreamEngine("tradesCrossEngine008")
}
subscribeTable(tableName=sTc_join12_testSTR, actionName="tradesCrossEngine008", offset=0, handler=append_plan, msgAsTable=true, hash=9, reconnect=true)
I have been puzzled by how to define and use custom module in Julia.
For example, I defined a module named myMoldule to wrap a mutable struct Param and a function add in D:\\run\\defineModule.jl:
module myMoldule
export Param, add
mutable struct Param
x ::Int64
y ::Int64
end
function add(x::Int64, y::Int64)
sum ::Int64
sum = x + y
return sum
end
end
and used this module in D:\\run\\useModule.jl like:
include("D:\\run\\defineModule.jl")
using .myMoldule
function testModule()
param = Param(1, 2)
sum = add(param.x, param.y)
println(sum)
end
An error occurred when running testModule() as follows:
julia> testModule()
ERROR: UndefVarError: Param not defined
Stacktrace:
[1] testModule() at D:\run\useModule.jl:8
[2] top-level scope at none:1
Note that I used the absolute path in the include(...) to avoid using LOAD_PATH stuff, and added . before the module name (i.e., using .myMoldule).
What seems to be the problem?
P.S.: Julia version information:
julia> versioninfo()
Julia Version 1.5.2
Commit 539f3ce943 (2020-09-23 23:17 UTC)
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: Intel(R) Core(TM) i7-8700K CPU # 3.70GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
JULIA_DEPOT_PATH = C:\Users\f\.julia;C:\opt\JuliaPro-1.5.2-1\Julia-1.5.2\local\share\julia;C:\opt\JuliaPro-1.5.2-1\Julia-1.5.2\share\julia
JULIA_LOAD_PATH = #;#v#.#;#stdlib
JULIA_NUM_THREADS = 6
JULIA_PKG_SERVER = pkg.juliahub.com
Corrections to be made:
sum is a function in Base you should use a different name
no need to declare sum variable (and it should be named something like mysum)
Remove space before ::
Module names should start with a CapitalLetter
You have a typo in module name perhaps you are loading a different module than you think?
Once corrected your code works.
The issue happens because you are using .MyModule1, not in the MyModule2, and thus you import Param to the Main module but not to the MyModule2, thus Module2 does not see Param.
If you will put using ..MyModule1 (two dots instead of one as you there is one more level) into MyModule2 this issue will go.
However, your code still will not work, since julia's include function just runs all the content of the included file, thus even you included the same file, you will create different instances of the modules. This warning WARNING: replacing module... indicates that somewhere in your code you might use the different version of the module (in you case, Main.Module1 and Main.Module2.Module1).
The common practice in julia is to include all the files in one place (they should be included only once). For instance, you can put all the includes in the file useModule.jl:
include("./defineModule1.jl")
include("./defineModule2.jl")
using .MyModule1
using .MyModule2
function testModule()
param = Param(1, 2)
# call myAdd to get the sum of param.x and param.y
sumValue = myAdd(param)
println(sumValue)
# call mySubtract to get the difference of param.x and param.y
difValue = mySubtract(param)
println(difValue)
end
Do not include files in other places. e.g.
defineModule2.jl content:
module MyModule2
using ..MyModule1
export myAdd, mySubtract
function myAdd(param::Param)
return param.x + param.y
end
function mySubtract(param::Param)
return param.x - param.y
end
end # end of module
Questioner's note: The following new question voted downwards (perhaps) was extended form the original one and was well answered by Vitaliy Yakovchuk.
I fixed all the issues pointed out by Przemyslaw Szufel. In my case above, it's not the improper ways of naming that cause the issue.
Now, I have a better exmaple to clarify my issue.
Suppose that, to meet the needs, I have to seperate my julia source code into two modules, e.g., define of a mutable struct Param in defineModule1.jl and define of functions in defineModule2.jl. The code scripts are as follows:
"D:\\run\\defineModule1.jl":
module MyModule1
export Param
mutable struct Param
x::Int64
y::Int64
end
end # end of module
"D:\\run\\defineModule2.jl":
include("D:\\run\\defineModule1.jl"); using .MyModule1
module MyModule2
export myAdd, mySubtract
function myAdd(param::Param)
return param.x + param.y
end
function mySubtract(param::Param)
return param.x - param.y
end
end # end of module
Note that Param is not defined here, and to make Param available, a line include("D:\\run\\defineModule1.jl"); using .MyModule1 is added as the first line of this file.
"D:\\run\\useModule.jl":
include("D:\\run\\defineModule1.jl"); using .MyModule1
include("D:\\run\\defineModule2.jl"); using .MyModule2
function testModule()
param = Param(1, 2)
# call myAdd to get the sum of param.x and param.y
sumValue = myAdd(param)
println(sumValue)
# call mySubtract to get the difference of param.x and param.y
difValue = mySubtract(param)
println(difValue)
end
Note that both function myAdd(param) and mySubtract(param) in the script defineModule2.jl need the predefined mutable struct Param in defineModule1.jl.
This is what I got when I run D:\\run\\useModule.jl:
julia> include("D:\\run\\useModule.jl")
WARNING: replacing module MyModule1.
WARNING: replacing module MyModule1.
WARNING: replacing module MyModule2.
ERROR: LoadError: LoadError: UndefVarError: Param not defined
Stacktrace:
[1] top-level scope at D:\run\defineModule2.jl:7
[2] include(::String) at .\client.jl:457
[3] top-level scope at D:\run\useModule.jl:2
[4] include(::String) at .\client.jl:457
[5] top-level scope at none:1
in expression starting at D:\run\defineModule2.jl:7
in expression starting at D:\run\useModule.jl:2
I believed that, by using the following lines in the beginning of "D:\run\useModule.jl", the mutable struct Param should have be found:
include("D:\\run\\defineModule1.jl"); using .MyModule1
include("D:\\run\\defineModule2.jl"); using .MyModule2
...
Still, error LoadError: UndefVarError: Param not defined is reported.
So, Why can't Param be found by D:\\run\\useModule.jl?
I'm defining a server action in Odoo 10. Within this action I am trying to use the following code:
for data in datas:
try:
inventory_level = int(data[context['inventory_level_column']].strip())
except TypeError:
continue
However, I receive an error:
ValueError: <type 'exceptions.NameError'>: "name 'TypeError' is not defined"
Is it not possible to catch errors within the context of an Odoo server action? Why is TypeError not defined?
The code written on a server action is passed through to safe_eval method. There the __builtins__ are stripped and replaced (thus the exceptions.NameError class is removed.
You can check this behaviour on odoo/tools/safe_eval.py on the definition of safe_eval method. See globals_dict['__builtins__'] = _BUILTINS where _BUILTINS does not contain this exception.
Exception is the parent class of all exception classes, so if you want to catch exception then just specify top most parent class in except and in e you will get error message.
for data in datas:
try:
inventory_level = int(data[context['inventory_level_column']].strip())
except Exception ,e:
print e
pass