Can't Access a global view table with SQL by a selected field code converted from a dropdown selection - apache-spark-sql

I am using a SQL notebook in Databricks/Spark community edition
%python
education_DF = sqlContext.sql('select * from global_temp.population_Globaltmp_view where `Education` = "2YD"')
display(education_DF)
The above code cell works fine and retrieves the desired rows from the view.
However the following code cell gives an error.
I am supbstituting the hard-coded value "2YD" in the WHERE clause, with a variable call education_choice that contains the same value.
It seems that I am not using variables correctly in SQL. How would I make this work?
%python
education_DF = sqlContext.sql('select * from global_temp.population_Globaltmp_view where `Education` = education_choice')
display(education_DF)
(This is the error)
org.apache.spark.sql.AnalysisException: cannot resolve '`education_choice`' given input columns: [global_temp.population_globaltmp_view.Salary, global_temp.population_globaltmp_view.Address, global_temp.population_globaltmp_view.Race, global_temp.population_globaltmp_view.MiddleI, global_temp.population_globaltmp_view.Education, global_temp.population_globaltmp_view.HairColor, global_temp.population_globaltmp_view.Age, global_temp.population_globaltmp_view.FullName, global_temp.population_globaltmp_view.City, global_temp.population_globaltmp_view.FirstName, global_temp.population_globaltmp_view.State, global_temp.population_globaltmp_view.LastName, global_temp.population_globaltmp_view.Height, global_temp.population_globaltmp_view.Fertility, global_temp.population_globaltmp_view.Employment, global_temp.population_globaltmp_view.Zip, global_temp.population_globaltmp_view.Weight, global_temp.population_globaltmp_view.Gender]; line 1 pos 72;
I tried the injection method suggested, but got a slightly different error this time. I included a screen shot of the relevant code cells. It seems that the education_choice "2YD" isn't being recognized as a literal but a field or something like that.
Shouldn't 2YD have quotes around it like "2YD"
If I hard code the WHERE clause like this: WHERE Education = "2YD" the queries works fine.
The image bellow is small but if you right click on it and "open in new tap, it is very readable.

*EDIT
You use the variable education_choice as a hardcoded string. Instead, you should inject the value of education_choice inside the SQL expression string like that.
*edit - wrap the variable with quoets.
%python
education_DF = sqlContext.sql('select * from global_temp.population_Globaltmp_view where `Education` = "{}"'.format(education_choice))
display(education_DF)

Related

Pass a variable into a SQL query in Excel Power Query

Long story short, I have a sql query pulling data from a database into a spreadsheet. I want to be able to change the Plant filter inside the query via Power Query, I have tried using a custom function and adding that into the query where the variable is set (see below)
Sql.Database("server", "database", [Query="DECLARE #FilterOnPlant AS varchar(3)#(lf)SET #FilterOnPlant = '" + filterOnPlant + "'#(lf)#(lf)SELECT
In doing this I keep getting the following error:
Expression.Error: We cannot apply operator + to types Text and Table.
Details:
Operator=+
Left=DECLARE #FilterOnPlant AS varchar(3)
SET #FilterOnPlant = '
Right=[Table]
I figure it has something to do with the '+' operator used but can't find an alternative.
I have tried using:
'&'
'+'
'and'
but they all yield the same error (with the exception of 'and' that error states that it's not being used in a logical statement which makes sense)
For further info:
The custom function to get the value I want is:
filterOnPlant = Excel.CurrentWorkbook(){[Name="Active_Plant"]}[Content]
The intention is to get the Value from a cell Named "Active_Plant"
EDIT:
after further testing I have found the issue lies with variable filterOnPlant, when I exchange that for a string value then the query works as expected. If anyone knows how I can get CELL().Value of a named range in power query would be super helpful
Solved!
In my filterOnPlant function I was missing {0}[Column1]
Original: filterOnPlant = Excel.CurrentWorkbook(){[Name="Active_Plant"]}[Content]
New (Working): filterOnPlant = Excel.CurrentWorkbook(){[Name="Active_Plant"]}[Content]{0}[Column1]

Why am I getting a `Data type mismatch` error when I add "CF" to the end of my search string in a SQL statement in Access?

The following query (qryCurLotNewProducts) produces a data set that I want process further with another query (qryBNP_CFRecordset):
//qryCurLotNewProducts
SELECT tblNewProducts.*
FROM tblNewProducts INNER JOIN tblCurLot ON (tblCurLot.CatalogNum = tblNewProducts.CatalogNum) AND
(tblNewProducts.LotNum = tblCurLot.CurLot);
When I run this second query to list only the "CF" products found in the first query, I get the `Data type mismatch in criteria expression' error.
//qryBNP_CFRecordset
SELECT qryCurLotNewProducts.*, tblABCategory.UNSPSC, tblAmount.ProductSize
FROM tblAmount RIGHT JOIN (tblABCategory RIGHT JOIN qryCurLotNewProducts ON tblABCategory.ABCategory = qryCurLotNewProducts.ABCategory) ON tblAmount.Amount = qryCurLotNewProducts.Amount
WHERE (((qryCurLotNewProducts.CatalogNum) Like "A700-###CF") AND ((qryCurLotNewProducts.DateEntered) Between #1/1/2000# And #3/1/2020#))
ORDER BY qryCurLotNewProducts.CatalogNum, Abs(qryCurLotNewProducts.LotNum);
If I remove the CF from the search string (so "A700-###"), the query correctly outputs a list containing all items that contain that pattern:
If I use strings like "A700-####F" or "A700-###ZZ" or other combinations like that, I don't get an error but rather an empty results set.
Notably, "A700-001CF", "A700-002CF", etc all create the data type error. It seems there is something about the CF key combination that is causing trouble.
Has anybody else ever seen this issue? Do I need to use some kind of delimiter to tell SQL to not view CF as some kind of special switch?
Abs(qryCurLotNewProducts.LotNum) wont work with the values for Products ending in CF. Your LotNum-Column has a text-type.
Edit: Your LotNum-Column has a text-type as you can see in your first screenshot.

Talend - Dynamic Column Name (Enterprise version)

Can anyone help me solve this case?
I have much file to process, two of them is like on below screenshot with my expected output.
I use this transformation on Talend: tFileList---tInputExcel---tUnpivotRow---tMap---tPostgresqlOutput
The output is different to my expected output. This is the screenshot of the output
Can anyone help me to reach my expected output which is like on my first picture above?
This will be pretty hard. You'd have to handle that as a text file. And whenever you found "store" value in the first column you'd update your type with the value.
Here's how I'd start:
Basically tJavaFlex begin piece would contain:
String col1Type
String colNType
main part:
if input_row.col0.equalsIgnoreCase("store") {
col1Type = input_row.col1;
col2Type = input_row.col2;
colNType = input_row.colN;
continue; /*(so this record will be Ignored for the rest of the components!)*/
}
output_row.col1Type = col1Type;
output_row.col1Value = Integer.valueOf(input_row.col1);
/*coz we have text and need numbers :( */
I think using propagate results will save you from writing down all the other fields.
And from here it would be very simple as you have key-type-value-type-value-type-value results.

create variable as a function of another variable in SQL

Okay. Basically, I want to define a complex variable within a SQL statement and then define another variable as a "function" of the first. I've searched around and can find no indication of how I might accomplish this. I can always just copypaste the code, but that makes it SO messy! See my (erroneous) example below. true file is the first variable and location is the second.
select
true_file = RIGHT(url.FULL_PATH, CHARINDEX('/',REVERSE(url.FULL_PATH)) - 1),
location = RIGHT(url.FULL_PATH, LEN(url.FULL_PATH) - LEN(true_file) - 9)
from files f
join BBLEARN_cms_doc.dbo.XYF_URLS url
on url.FILE_ID = substring(f.file_name, 6, len(f.file_name) - 7)
where f.file_name like '/xid-%'
order by url.FULL_PATH;
But obviously, we get an "invalid column name" error when trying to run any functions on true_file. What's the best way to do this?

How can I use "Expression.Not" with text field?

How can I use "Expression.Not" with text field?
I need to select all records from NHQuestionCount except "ktest"
for example this code return runtime error
NHQuestionCount[] stats = NHQuestionCount.FindAll(Order.Asc("NameFull"), Expression.Not(Expression.Eq("NameFull", "ktest")));
I can't comment on the rest of your code, but your use of Expression is exactly right.