I guess defines allows passing parameters to the Hive query, but I haven't found any documentation or usage examples. How do I use these parameters in the query?
I am using defines parameter from powershell, but hopefully it will help you.
So first in PowerShell script I am writing something like following:
[HashTable] $defines
$defines = #{}
$defines.Add("someContainerUrl",${someContainerUrl})
Then I am using this new defines variable when I am calling the HDInsight:
$jobDef = New-AzureHDInsightHiveJobDefinition -Query $hiveQuery -JobName $jobName -Defines $defines -StatusFolder "/JobStatus/Hive"
$hiveJob = Start-AzureHDInsightJob –Cluster $clusterName –JobDefinition $jobDef
Then in my hive code, I can use this defines variable for example as following:
ADD FILES
${hiveconf:someContainerUrl}/SomeFile.exe;
I think this article might be of help to you as well:
http://blogs.msdn.com/b/bigdatasupport/archive/2014/02/13/how-to-pass-hadoop-configuration-values-for-a-job-via-hdinsight-powershell-and-net-sdk.aspx
Related
In the Polarion documentation:
https://almdemo.polarion.com/polarion/sdk/doc/javadoc/com/polarion/alm/tracker/model/IWorkItem.html#traverseLinkedWorkitems(java.util.Set,java.util.Set,java.util.Set,com.polarion.alm.tracker.model.IWorkItem.ITerminalCondition)
I have created empty sets using $objectFactory.newSet() to account for the first 3 parameters, and I have tried "null" for the conditional parameter, but nothing works.
This is an example of what I have tried:
#set($project = "Project X"
#set($workItem1 = 'ABC-123')
#set($emptySet = $objectFactory.newSet())
#set($ts1 = $trackerService.getWorkItem($project,$workItem1))
$ts1 ##output: PObject(WorkItem; subterra:data-service:objects:/default/Project X${WorkItem}ABC-123)
$ts1.traverseLinkedWorkItems($emptySet,$emptySet,$emptySet,'null')
The output is always $ts1.traverseLinkedWorkItems($emptySet,$emptySet,$emptySet,'null')
Is there no way to do this in Velocity? I have seen only one other post regarding this question:
https://community.sw.siemens.com/s/question/0D54O000075P0SCSA0/any-way-to-call-traverselinkedworkitems-from-a-velocity-script-block-widget
Have you tried $null as the last argument? As an undefined reference, it will translate to null.
But this solution will only work if Velocity is not running in strict mode.
I am looking for a way to use a regex as value in a named parameter in the Java SDK. According to the documentation, there is no datatype for that, and using a String parameter does not work.
Is there any way to use a regex as value in a named parameter?
QueryParameterValue Class has no datatype for that:
https://googleapis.dev/java/google-cloud-clients/0.91.0-alpha/com/google/cloud/bigquery/QueryParameterValue.html#of-T-com.google.cloud.bigquery.StandardSQLTypeName-
A regex in the query would e.g. look like this:
REGEXP_CONTAINS(some_attribute, r"^any\sregex\ssearchstring$")
and should be replaced by a named parameter like:
REGEXP_CONTAINS(some_attribute, #named_regex_parameter)
I tried different syntax in the query like
REGEXP_CONTAINS(some_attribute, r#named_regex_parameter)
etc. but none of them worked. The #named_regex_parameter is of type String. I tried to use values in the form of r"regex_expression" and just the regex_expression in the parameter value.
Seems like I need to build the query String without a named parameter for the regex part. Any hints to solve this with parameters would be really appreciated!
//Edit: added code example how the named parameters are used in the query config
QueryJobConfiguration queryConfig = QueryJobConfiguration.newBuilder(query)
.setDestinationTable(TableId.of(destinationDataset, destinationTable))
.setAllowLargeResults(true)
.setCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
.setTimePartitioning(partitioning)
.setWriteDisposition(WriteDisposition.WRITE_APPEND)
.setParameterMode("NAMED")
.addNamedParameter("regexExpressionParam", QueryParameterValue.string(someRegexExpressionStringVariable)) //this does not work
.addNamedParameter("someStringParam", QueryParameterValue.string(stringVariable))
.setPriority(Priority.BATCH)
.build();
The query should use the parameter #regexExpressionParam like so:
REGEXP_CONTAINS(theAttributeToQuery, #regexExpressionParam))
You need to pass the regular expression string without r'...'
I had a very similar problem with running parameterized queries on Python: it was something like this.
from google.cloud import bigquery
regex_input = "^begin_word.*end_here$"
# Construct a BigQuery client object.
client = bigquery.Client()
query = """
SELECT word, word_count
FROM `bigquery-public-data.samples.shakespeare`
WHERE REGEXP_CONTAINS(word, #regex)
ORDER BY word_count DESC;
"""
job_config = bigquery.QueryJobConfig(
query_parameters=[
bigquery.ScalarQueryParameter("regex", "STRING", f"r'{regex_input}'"),
]
)
query_job = client.query(query, job_config=job_config)
At first, I thought the input had to be wrapped by r'...'; just like how I normally write a regex on BQ explorer.
I tried to modify the string input to make it like a regular expression, which was this pard f"r'{regex_input}'" of the code.
but apparently BQ correctly escapes string without our help and I can just pass down the regex string like bigquery.ScalarQueryParameter("regex", "STRING", regex_input)
Is there any easy way to handle huge query param like below. Also I would like to know how can I do run time parameterisation for some values?
http://154.213.196.243:7941/v1/banking/Jumio/callback?callBackType=NetVerifyId&jumioIdScanReference=123abcde-1244-8571-3454-abcd12345567&merchantIdScanReference=66a9ff2e-d8ec-e811-a956-000d3ab3f117&verificationStatus=APPROVED_VERIFIED&idScanStatus=SUCCESS&id+ScanSource=API&idCheckDataPositions=OK&idCheckDocumentValidation=OK&idCheckHologram=OK&idCheckMRZcode=OK&idCheckMicroprint=OK&idCheckSecurityFeatures=OK&idCheckSignature=OK&transactionDate=2018-11-20T20%3A53%3A25.797Z&callbackDate=2018-11-20T20%3A53%3A25.797Z&idType=DRIVING_LICENSE&idCountry=GBR&idScanImage+=https%3A%2F%2Fnetverify.com%2Frecognition%2Fv1%2Fidscan%2F123abcde-1244-8571-3454-abcd12345567%2Ffront&idFirstName=ILARIA&idLastName=FURS&idDob=1976-12-23&idExpiry=2025-12-31&personalNumber=123456789&clientIp=xxx.xxx.xxx.xxx&idAddress=%7B%22country%22%3A%22USA%22%2C%20%22stateCode%22%3A%22US-OH%22%7D&idNumber=P12345&idStatus=TESTER961260SS9DL54&identityVerification=%7B%22similarity%22%3A%22MATCH%22%2C%22validity%22%3Atrue%7D HTTP/1.1
Yes. Read the docs: https://github.com/intuit/karate#param
For example:
* param callBackType = 'NetVerifyId'
and so on. And look at params where you can set all keys up as one single JSON and also do parameterization if needed, there are multiple possibilities: https://github.com/intuit/karate#params
See this example as well: dynamic-params.feature
I am trying to avoid a sql injection. This topic has been dealt with in Java (How to prevent query injection on Google Big Query) and Php.
How is this accomplished in App Scripts? I did not find how to add a parameter to a SQL statement. Here is what I had hoped to do:
var sql = 'SELECT [row],etext,ftext FROM [hcd.hdctext] WHERE (REGEXP_MATCH(etext, esearch = ?) AND REGEXP_MATCH(ftext, fsearch = ?));';
var queryResults;
var resource = {
query: sql,
timeoutMs: 1000,
esearch='r"[^a-zA-z]comfortable"',
fsearch='r"[a-z,A-z]confortable"'
};
queryResults = BigQuery.Jobs.query(resource,projectNumber);
And then have esearch and fsearch filled in with the values (which could be set elsewhere).
That does not work, according to the doc.
Any suggestions on how to get a parameter in an SQL query? (I could not find a setString function...)
Thanks!
Unfortunately, BigQuery doesn't support this type of parameter substitution. It is on our list of features to consider, and I'll bump the priority since it seems like this is a common request.
The only suggestion that I can make in the mean time is that if you are building query strings by hand, you will need to make sure you escape them carefully (which is a non-trivial operation).
I'm converting some RDO code to ODBC Provider code in .NET.
The problem is parameter names were not specified in the orignal code, but param values were retrieved by parameter name after the command was executed.
Is there anyway to have parameter names populated by the provider once the command is executed so calling code can access params by name.
Let me show you an example of the declaration of param and accessing of it.
With rdqryClntBasic
.Parameters.Add(.CreateParameter) : .Parameters(0).Direction = ParameterDirection.Input
.Parameters(0).DbType = DbType.String
.Parameters(0).Value = sClntProdCd
End With
.EffectiveDate = ToDate(rdqryClntBasic.Parameters("dtEffDt").Value)
You can now see how this "used to work in RDO/VB". For some reason it would accept this and know what the param names were after execution. I imagine it had to do another round trip to the db to get this info.
Is there anyway to mimic this behaviour in .NET for ODBC Provider (using Oracle)? Or am I stuck manually specifying the param names in the code (I understand this is the better option, but wondering what the alternative is to match the original code as closely as possible).
No, parameters in ODBC are positional not by name.