Databricks getArgument value cannot be used within Spark SQL IN clause - apache-spark-sql

I am getting a list of comma separated strings from a databricks notebook widget but I cannot use that value within Spark SQL's IN clause
I am within a %sql cell within the notebook and would like to NOT have to jump into %py or %scala cell type.
Also, I cannot use a select statement within the IN clause as the IN clause is NOT within a select filter. One gets an error: "IN/EXISTS predicate sub-queries can only be used in a Filter" if you try that syntax.
Here ITEM is a widget created like this in a %sql cell
CREATE WIDGET TEXT ITEM DEFAULT "'0111','0112'"
select *
from items_table
where ItemSKU in (string(getArgument("ITEM")));
or
select *
from items_table
where ItemSKU in (getArgument("ITEM"));
it does not return any rows
But if I directly substitute the value of the comma separated string in code, i.e.
select *
from items_table
where ItemSKU in ('0111','0112');
I get the expected result.

This is possible, try:
...
WHERE ItemSKU IN (SELECT explode(split(getArgument("ITEM"),',')))
This will also work for multi-select widgets.
The text widget option will be susceptible to user error if your users are allowed to enter text directly.

I don't think you can have multiple items as a default value of a multiselect widget, and definetely not a text widget.
If you look at the signature of the multiselect function
multiselect(name: String, defaultValue: String, choices: Seq, label: String)
You can see that the defaultValue only accepts a String while the choices is a Sequence. I know this is the scala function and you are lookign for the SQL version, but this implies that there is not meant to be a multiselect default option yet that actually has multiple selections.
Link - https://docs.databricks.com/user-guide/dev-tools/dbutils.html

Related

Open Refine: Exporting nested XML with templating

I have a question regarding the templating option for XML in Open Refine. Is it possible to export data from two columns in a nested XML-structure, if both columns contain multiple values, that need to be split first?
Here's an example to illustrate better what I mean. My columns look like this:
Column1
Column2
https://d-nb.info/gnd/119119110;https://d-nb.info/gnd/118529889
Grützner, Eduard von;Elisabeth II., Großbritannien, Königin
https://d-nb.info/gnd/1037554086;https://d-nb.info/gnd/1245873660
Müller, Jakob;Meier, Anina
Each value separated by semicolon in Column1 has a corresponding value in Column2 in the right order and my desired output would look like this:
<rootElement>
<recordRootElement>
...
<edm:Agent rdf:about="https://d-nb.info/gnd/119119110">
<skos:prefLabel xml:lang="zxx">Grützner, Eduard von</skos:prefLabel>
</edm:Agent>
<edm:Agent rdf:about="https://d-nb.info/gnd/118529889">
<skos:prefLabel xml:lang="zxx">Elisabeth II., Großbritannien, Königin</skos:prefLabel>
</edm:Agent>
...
</recordRootElement>
<recordRootElement>
...
<edm:Agent rdf:about="https://d-nb.info/gnd/1037554086">
<skos:prefLabel xml:lang="zxx">Müller, Jakob</skos:prefLabel>
</edm:Agent>
<edm:Agent rdf:about="https://d-nb.info/gnd/1245873660">
<skos:prefLabel xml:lang="zxx">Meier, Anina</skos:prefLabel>
</edm:Agent>
...
</recordRootElement>
<rootElement>
(note: in my initial posting, the position of the root element was not indicated and it looked like this:
<edm:Agent rdf:about="https://d-nb.info/gnd/119119110">
<skos:prefLabel xml:lang="zxx">Grützner, Eduard von</skos:prefLabel>
</edm:Agent>
<edm:Agent rdf:about="https://d-nb.info/gnd/118529889">
<skos:prefLabel xml:lang="zxx">Elisabeth II., Großbritannien, Königin</skos:prefLabel>
</edm:Agent>
)
I managed to split the values separated by ";" for both columns like this
{{forEach(cells["Column1"].value.split(";"),v,"<edm:Agent rdf:about=\""+v+"\">"+"\n"+"</edm:Agent>")}}
{{forEach(cells["Column2"].value.split(";"),v,"<skos:prefLabel xml:lang=\"zxx\">"+v+"</skos:prefLabel>")}}
but I can't find out how to nest the splitted skos:prefLabel into the edm:Agent element. Is that even possible? If not, I would work with seperate columns or another workaround, but I wanted to make sure, if there's a more direct way before.
Thank you!
Kristina
I am going to expand the answer from RolfBly using the Templating Exporter from OpenRefine.
I do have the following assumptions:
There is some other column left of Column1 acting as record identifying column (see first screenshot).
The columns actually have some proper names
The columns URI and Name are the only columns with multiple values. Otherwise we might produce empty XML elements with the following recipe.
We will use the information about records available via GREL to determine whether to write a <recordRootElement> or not.
Recipe:
Split first Name and then URI on the separator ";" via "Edit cells" => "Split multi-valued cells".
Go to "Export" => "Templating..."
In the prefix field use the value
<?xml version="1.0" encoding="utf-8"?>
<rootElement>
Please note that I skipped the namespace imports for edm, skos, rdf and xml.
In the row template field use the value:
{{if(row.index - row.record.fromRowIndex == 0, '<recordRootElement>', '')}}
<edm:Agent rdf:about="{{escape(cells['URI'].value, 'xml')}}">
<skos:prefLabel xml:lang="zxx">{{escape(cells['Name'].value, 'xml')}}</skos:prefLabel>
</edm:Agent>
{{if(row.index - row.record.fromRowIndex == row.record.rowCount - 1, '</recordRootElement>', '')}}
The row separator field should just contain a linebreak.
In the suffix field use the value:
</rootElement>
Disclaimer: If you're keen on using only OpenRefine, this won't be the answer you were hoping for. There may be ways in OR that I don't know of. That said, here's how I would do it.
Edit The trick is to keep URL and literal side by side on one line. b2m's answer below does just that: go from right to left splitting, not from left to right. You can then skip steps 2 and 3, to get the result in the image.
split each column into 2 columns by separator ;. You'll get 4 columns, 1 and 3 belong together, and 2 and 4 belong together. I'm assuming this will be the case consistently in your data.
export 1 and 3 to a file, and export 2 and 4 to another file, of any convenient format, using the custom tabular exporter.
concatenate those two files into one single file using an editor (I use Notepad++), or any other method you may prefer. Several ways to Rome here. Result in OR would be something like this.
You then have all sorts of options to put text strings in front, between and after your two columns.
In OR, you could use transform on column URL to build your XML using the below code
(note the \n for newline, that's probably just a line feed, you may want to use \r\n for carriage return + line feed if you're using Windows).
'<edm:Agent rdf:about="' + value + '">\n<skos:prefLabel xml:lang="zxx">' + cells.Name.value + '</skos:prefLabel>\n</edm:Agent>'
to get your XML in one column, like so
which you can then export using the custom tabular exporter again. Or instead you could use Add column based on this column in a similar manner, if you want to retain your URL column.
You could even do this in the editor without re-importing the file back into OR, but that's beyond the scope of this answer.

Querying full and sub-strings via multi-valued parameter using SQL

I am building a report with Microsoft SSRS (2012) having a multi-value parameter #parCode for the user to filter for certain codes. This works perfectly fine. Generally, my query looks like this:
SELECT ...
FROM ...
WHERE
TblCode.Code IN (#Code)
ORDER BY...
The codes are of following type (just an excerpt):
C73.0
C73.1
...
C79.0
C79.1
C79.2
Now, in additon to filtering for multiple of these codes I would like to als be able to filter for sub-strings of the codes. Meaning, when the user enters (Example 1)
C79
for #parCodes The output should be
C79.0
C79.1
C79.2
So eventually the user should be able to enter (Example 2)
C73.0
C79
for #parCodes and the output would be
C73.0
C79.0
C79.1
C79.2
I managed to implement both functionalities seperately, so either filtering for multiple "complete" codes or filterting for sub-string of code, but not both simultaneously.
I tried to do something like
...
WHERE
TblCode.Code IN (#parCode +'%')
ORDER BY...
but this screws up the Example 2. On the other hand, if I try to work with LIKE or = instead of IN statement, then I won't be able to make the parameter multi-valued.
Does anyone have an idea how to realize such functionality or whether IN statement pared with multi-valued parameters simply doesn't allow for it?
Thank you very much!
Assuming you are using SQL server
WHERE (
TblCode.Code IN (#parCode)
OR
CASE
WHEN CHARINDEX('.', Code)>0 THEN LEFT(TblCode.Code, CHARINDEX('.', TblCode.Code)-1)
ELSE TblCode.Code
END IN (#parCode)
)
The first clause makes exact match so for your example matches C73.0
The second clause matches characters before the dot character so it would get values C79.0, C79.1, C79.2 etc
Warning: Filtering using expressions would invalidate the use of an index on TblCode.Code

Populate the data in its own field

How can the data be populated in its own field? See the field called 'vSoup Text'.
SELECT [iFormID]
,[iPracID]
,[iPatID]
,[vFormName]
,[vSoapText]
,[tHTML]
,[iUserID]
,[dDATE] FROM CustomForm where iPatID = 40 and vFormName = 'Diabetes Screening'
I need to break the data when you see '/br'
The only way that I've found to do this is by using a function. You could use SSIS, but the way that I've used is using a function.
https://sqlperformance.com/2012/07/t-sql-queries/split-strings

How to use Substring in SSIS

I want to export data from SharePoint list to SQL using SSIS.
In SharePoint list, i have a column as multi select, So i am getting below value in my column
1;#control 1;#3;#control 3
I want to use substring in derived column in such a way that i should get below result
1,3
I want only ID from the given column.
I have tried below code
SUBSTRING(ColumnName,1,FINDSTRING(ColumnName,";#",1) - 1)
But it only gives me answer as
1
Can anyone please help me out.?
Because there is an unknown number of controls selected in your SharePoint Multi-Select, a Derived Column transformation is not going to work for you. You'll have to use a script.
One way to parse your string is with regular expressions. You'll have to add an output to the script transformation and assign your parsed string to that output.
Regex controlExpression = new Regex(#"control ([0-9]+)");
MatchCollection controlMatches = controlExpression.Matches(--YOUR INPUT HERE--);
String output = string.Join(",",
(controlMatches.Cast<Match>().Select(n => n.Groups[1].ToString())).ToArray());

How can I use "Expression.Not" with text field?

How can I use "Expression.Not" with text field?
I need to select all records from NHQuestionCount except "ktest"
for example this code return runtime error
NHQuestionCount[] stats = NHQuestionCount.FindAll(Order.Asc("NameFull"), Expression.Not(Expression.Eq("NameFull", "ktest")));
I can't comment on the rest of your code, but your use of Expression is exactly right.