I'm trying to set up a time function with Graphite and Grafana, and I found the identity() function in Graphite's documentation.
But I couldn't enter the function into Grafana, how do I get it work?
Related
I am trying to create a function in Bigquery which pulls a Cloud Functions:
CREATE OR REPLACE FUNCTION `DATASET.XXXXX`(user_id int64, corp_id STRING) RETURNS STRING
REMOTE WITH CONNECTION `myPROJECTID.REGION.MY_CONNECTION`
OPTIONS (
endpoint = 'https://XXXX.cloudfunctions.net/XXXXX'
)
previously create a connection in the Bigquery shell, but I get the following error, does anyone know?
Keyword REMOTE is not supported at [2:1]
or
Not found: Connection my-connection
Your project must be allowlisted. It's a private preview (I asked 2 month ago, still nothing....)
I am using a Spark UDF to read some data from a GET endpoint and write them as a CSV file to a Azure BLOB location.
My GET endpoint takes 2 query parameters,param1 and param2.
So initially, I have a dataframe paramDF that has two columns param1 and param2.
param1 param2
12 25
45 95
Schema: paramDF:pyspark.sql.dataframe.DataFrame
param1:string
param2:string
Now I write a UDF that accept the two parameters, register it, and then invoke this UDF for each row in the dataframe.
UDF is as below:
def executeRestApi(param1,param2):
dlist=[]
try:
print(DataUrl.format(token=TOKEN, q1=param1,q2=param2))
response=requests.get(DataUrl.format(token=TOKEN, oid=param1,wid=param2))
if(response.status_code==200):
metrics=response.json()['data']['metrics']
dic={}
dic['metric1'] = metrics['metric1']
dic['metric2'] = metrics['metric2']
dlist.append(dic)
pandas.DataFrame(dlist).to_csv("../../dbfs/mnt/raw/Important/MetricData/listofmetrics.csv",header=True,index=False,mode='x')
return "Success"
except Exception as e:
return "Failure"
Register the UDF:
udf_executeRestApi = udf(executeRestApi, StringType())
Finally the call the UDF this way
paramDf.withColumn("result",udf_executeRestApi(col("param1"),col("param2"))
I dont see any errors while calling the UDF, in fact the UDF returns the value "Success" correctly.
Only thing is that the files are not written to Azure BLOB storage, no matter what I try.
UDFs' are primarily meant for custom functionality(and return a value).However ,in my case, I am trying to execute the GET API call and the write operation using the UDF(and that is my main intention here).
There is no issue with my pandas.DataFrame().tocsv(),as the same line, when tried separately,with a simple list is writing data to the BLOB correctly.
What could be going wrong here?
Note: Env is Spark on Databricks.
There isn't any problem with the indentation, even though it looks untidy here.
Try calling display on the dataframe
I am trying to generate a variable for a Lambda Function that is based on the setting from an API Gateway created at the same time using Terraform. I am using trimprefix and trimsuffix to modify the setting I get from the api gateway, which I then set as an Environment Variable to be used by the Lambda Function Code.
I had this working initially using output statements as I was originally using modules. I have since decided to move away from modules to simplify the code. However my real issue is how do I perform the trimprefix and trimsuffix actions when I am also using the count feature.
Here is my original code when I was still using modules, and it successfully created the final "invoke_url" after trimming "https://" from the beginning, and "/default" from the end.
## Obtain the rest_api_id
output "rest_api_id" {
value = aws_api_gateway_deployment.retaildiscount[count.index].rest_api_id
}
## Trim the https prefix from the invoke URL and store in var.invoke_url_tmp
output "invoke_url_tmp" {
value = trimprefix(aws_api_gateway_deployment.retaildiscount[count.index].invoke_url, "https://")
}
## Trim the /default suffix from var.invoke_url_tmp and output as var.invoke_url to be used
## by the retailorderprice function
output "invoke_url" {
value = trimsuffix(var.invoke_url_tmp, "/default")
}
I am now trying to do the same, but when using "count" to create multiple copies of the same Lambda Functions and API Gateways (this is to create multiple instances for a lab style workshop, each Function will have a unique name pulled from a aito.tfvars file
For the life of me I cannot work out how to generate the modified variable and link it back to the appropriate function
I have tried to implement the navigateToConversation Function into my Programm but it is telling me that the function does not exist.
I have allready succesfully signed in and can getConversations as well as addItems.
client.navigateToConversation('5af91241-dd21-42a4-ade4-370c5e32907f')
.then(() => console.log('Navigate to Conversation'))
The convId is a real one as i have previosly written to it using the SDK.
The Error produced by calling the Function
I am also using AngularJS.
The Circuit SDK has been updated on npm and unpkg. The function is now available.
https://unpkg.com/circuit-sdk
I am reading a databricks blog link
and I find a problem with the built-in function to_json.
In the codes blew within this tutorial, it returns error:
org.apache.spark.sql.AnalysisException: Undefined function: 'to_json'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.
Does this means that this usage in the tutorial is wrong? and no udf could be used in selectExpr. Could I do something like register this to_json function into default database?
val deviceAlertQuery = notifydevicesDS
.selectExpr("CAST(dcId AS STRING) AS key", "to_json(struct(*)) AS value")
.writeStream
.format("kafka")
.option("kafka.bootstrap.servers", "host1:port1,host2:port2")
.option("toipic", "device_alerts")
.start()
You need to improt the to_json function as
import org.apache.spark.sql.functions.to_json
This should work rather than the selectExpr
data.withColumn("key", $"dcId".cast("string"))
.select(to_json(struct(data.columns.head, data.columns.tail:_*)).as("value")).show()
You must also use the spark 2.x
I hope this helps to solve your problem.
based on information I get from mail list. this function are not added into SQL from spark 2.2.0. Here is the commit link:commit.
Hope this will help. THX Hyukjin Kwon and Burak Yavuz.