I have a UDF defined like so:
def my_function(input: Array[Byte])
and I want to call it in spark SQL, so i'm trying
SELECT my_function(binary(CONCAT(*))) FROM table;
but I don't think this is working. To my understanding, select * will return Array[Row], and then calling the native function binary will serialize that. Will that convert Array[Row] to Array[Byte]? Not sure how to call this udf via sql
We have to register the function and then we can use the UDF
ie
spark.udf.register(funname and definition )
you can explore more on link
Related
I'm trying to use jQuery with UDFs written in javascript, to be used with BigQuery. I uploaded the jQuery library to my cloud-storage, but when I try to upload it to my UDF I'm getting an error
TypeError: Cannot read property 'createElement' of undefined at gs://mybucket/jquery.min.js line 2, columns 7311-7312
Any help, please?
Thank you.
CREATE TEMP FUNCTION test()
RETURNS STRING
LANGUAGE js
OPTIONS (
library=["gs://mybucket/jquery.min.js"]
)
AS """
return "test";
""";
As you might know, some limitations apply to temporary and persistent user-defined functions
One of them - The DOM objects Window, Document, and Node, and functions that require them, are not supported.
This might be a reason!
As illustrated here, dumping the wasm byte code and copy past into the javascript seems difficult.
I guess you mean a better way than copying into JS - I haven't investigated that (yet), but this will make UDFs easier for others to use:
Move the .js out of the query into a file.
Create a persistent function.
Then people will be able to call it like this:
SELECT fhoffa.x.sample_wasm_udf([2,3,4])
To create this function I did:
CREATE OR REPLACE FUNCTION fhoffa.x.sample_wasm_udf(x ARRAY<INT64>)
RETURNS ARRAY<INT64>
LANGUAGE js AS '''
return main(x)
'''
OPTIONS (library="gs://fh-bigquery/js/wasm.udf.js");
For more on persistent functions, see:
https://medium.com/#hoffa/new-in-bigquery-persistent-udfs-c9ea4100fd83
In my project we have a requirement where we have to insert an array of struct as an input parameter.How i can achieve that.
I don't want to use the approach suggested here
https://dzone.com/articles/passing-java-arrays-in-oracle-stored-procedure-fro
I want to use the inbuilt objects given in mule.
we have to insert an array of struct as an input parameter
Where you want to insert this array ?
array of struct
can we consider the User defined data type as POJOs ( just java object?) ?
I want to use the inbuilt objects given in mule.
As I can see in the link, the example use Java + Mule component. Should I guess that you want to avoid Java code?
Please, elaborate your use case... hard to replicate
I have a function that I created in SQL, and I need to test to make sure that its functioning properly. I know I can use something like...
SELECT * FROM dbo.TestFunction
to execute it, but how do I pass a parameter to the function?
Just call it with parameter as follows:
SELECT * FROM dbo.TestFunction (your_parameters_separated_by_commas)
If it's one parameter only, you call it as dbo.TestFunction(#param1). If it has multiple parameters, change to dbo.TestFunction(#param1,#param2...).
I wanna format some fields in the output of my PostgreSQL 9.1 database. I thought of creating a type, so I could do the formatting in the output function, and checking for inconsistencies in the input function. I decided to use the procedural language PL/pgSQL. But I'm getting some errors:
CREATE OR REPLACE FUNCTION "CPF_in"(cstring)
"PL/pgSQL functions cannot accept type cstring"
(But that's how it is in the manual.) I can put "character varying" instead of cstring, or even leave the () empty. But when I'm going to create the desired type:
CREATE TYPE Tcpf (
INPUT = CPF_in(character varying),
OUTPUT = CPF_out
);
I got an error:
ERROR: syntax error at or near ")"
LINE 2: INPUT = CPF_in(character varying),
and if I try
CREATE TYPE Tcpf (
INPUT = CPF_in(),
OUTPUT = CPF_out
);
I get
ERROR: syntax error at or near ")"
LINE 2: INPUT = CPF_in(),
How is this supposed to be done? The manual only say cstring...
The cstring pseudo-type is used for programming in a low-level language like C, not in PL/pgSQL. You have to use a low-level language like C if you're creating a new base type.
You must register two or more functions (using CREATE FUNCTION) before
defining the type. The support functions input_function and
output_function are required . . . .
Generally these functions have to be coded in C or another low-level
language.
A simpler way to control the output format is to use a view. If your formatting is complex, write a function, and call that function from a view. You can revoke permissions on the base table if you need to force every client to use your formatting. You might need to create triggers to make your view fully updatable.
For controlling input, you can use a function. (CREATE FUNCTION...) You can write functions in PL/pgSQL. Again, consider revoking permissions on the table.