Dynamically change project used in a SQL query - dynamic

I am using BigQuery, Standard SQL, and I want to dynamically change parts of the FROM clause, such as the project id. I have been looking for a solution for this the last 3 years - the problem has been that parameters cannot be used as inputs in the FROM clause. The benefit would be to create a stored procedure, where the project id can be passed in as an argument and can query the appropriate project. The projects would have the same datasets and table names - this would be our way of building a Master query for easy development and implementation. Instead of changing 15 clients' views, we can change the Stored Procedure once and it will push out the changes to all clients' views. However, I have always gotten hung up on dynamically changing the FROM clause!
For example:
DECLARE ProjectId STRING DEFAULT 'test_project';
SELECT col_1 FROM `#ProjectId.Dataset.Table`;
would always error out due to parameters not being able to be used in the FROM clause. However, I saw a related post on using dynamic SQL to overcome this obstacle. I've been looking into the EXECUTE IMMEDIATE function within BigQuery, as this is what has been cited to be a solution. From that post I attempted to implement in several ways:
Attempt #1:
DECLARE ProjectId STRING DEFAULT 'test_project';
EXECUTE IMMEDIATE CONCAT(
"SELECT * FROM ", #ProjectId, ".DataSet.`Table` " )
^ This gives an error "Query error: Undeclared query parameters at [2:19]"
Attempt #2:
EXECUTE IMMEDIATE CONCAT(
"SELECT * FROM ", #ProjectId, ".DataSet.`Table` " )
USING 'my-project' as ProjectId, 'my-dataset' as DataSet;
^ which gives the error "Query error: Undeclared query parameters at [1:19]"
Third and final attempt was to try declaring the parameter within the EXECUTE IMMEDIATE:
EXECUTE IMMEDIATE CONCAT(
"DECLARE ProjectId STRING DEFAULT 'test_project'; ",
"SELECT * FROM ", #ProjectId, ".DataSet.`Table` " )
USING 'my-project' as ProjectId, 'my-dataset' as DataSet;
^ which, you guessed it, results in the same error "Query error: Undeclared query parameters at [1:19]"
I am reaching out to see if anybody has had success with this? I see the value in the Dynamic SQL statements, and have read the documentation and some examples, but it still doesn't seem to work when trying to dynamically change the FROM clause. Any help is much appreciated, willing to try whatever is thrown out - excited to learn what can be done!

Just remove #:
DECLARE ProjectId STRING DEFAULT 'test_project';
EXECUTE IMMEDIATE CONCAT(
"SELECT * FROM ", ProjectId, ".DataSet.`Table` " )

Related

Pass a variable into a SQL query in Excel Power Query

Long story short, I have a sql query pulling data from a database into a spreadsheet. I want to be able to change the Plant filter inside the query via Power Query, I have tried using a custom function and adding that into the query where the variable is set (see below)
Sql.Database("server", "database", [Query="DECLARE #FilterOnPlant AS varchar(3)#(lf)SET #FilterOnPlant = '" + filterOnPlant + "'#(lf)#(lf)SELECT
In doing this I keep getting the following error:
Expression.Error: We cannot apply operator + to types Text and Table.
Details:
Operator=+
Left=DECLARE #FilterOnPlant AS varchar(3)
SET #FilterOnPlant = '
Right=[Table]
I figure it has something to do with the '+' operator used but can't find an alternative.
I have tried using:
'&'
'+'
'and'
but they all yield the same error (with the exception of 'and' that error states that it's not being used in a logical statement which makes sense)
For further info:
The custom function to get the value I want is:
filterOnPlant = Excel.CurrentWorkbook(){[Name="Active_Plant"]}[Content]
The intention is to get the Value from a cell Named "Active_Plant"
EDIT:
after further testing I have found the issue lies with variable filterOnPlant, when I exchange that for a string value then the query works as expected. If anyone knows how I can get CELL().Value of a named range in power query would be super helpful
Solved!
In my filterOnPlant function I was missing {0}[Column1]
Original: filterOnPlant = Excel.CurrentWorkbook(){[Name="Active_Plant"]}[Content]
New (Working): filterOnPlant = Excel.CurrentWorkbook(){[Name="Active_Plant"]}[Content]{0}[Column1]

select globalmap using tDBinput with Talend give the error: Invalid character constant

I have to remove the accents from the person's name, but I cannot apply the function in Talend while it works in SQL oracle.
this query works in my tDBInput component :
"SELECT '"+((String)globalMap.get("copyOfSORTIE.NOM"))+"' as nom_nom_compl,
'"+((String)globalMap.get("copyOfSORTIE.ENTETE"))+"' entete
FROM DUAL"
However, when I want to add the convert function, it doesn't work
this query does not work :
"SELECT '"+((String)globalMap.get(CONVERT("copyOfSORTIE.NOM",'US7ASCII')))+"' as nom_nom_compl,
'"+((String)globalMap.get("copyOfSORTIE.ENTETE"))+"' entete
FROM DUAL"
In my talend :
I am getting this error
What is the syntax for it to work?
Thank you!
Two things there :
I don't know the CONVERT method, but I can see that you are applying it to the key of your globalMap variable , and not the value (as if you wanted to convert "myKey" and not "myValue" which is attached to the key). Are you sure this is what you want to achieve ? if not, the syntax should be something similar to "SELECT CONVERT('"+((String)globalMap.get("copyOfSORTIE.NOM"))+"','US7ASCII') "
A useful java method implemented in talend is TalendString.removeAccents("") that you can apply directly on your talend variable, thus not using a SQL method.

error in google datastudio on using custom query

I am using a simple sql block of statements to execute and return a set of results in big query, This is working fine in big query and getting the results,I need to export this data to data studio, so in data studio i use bigquery as connector and select the project and custom query and in that I paste the contents below:
Declare metricType String;
SET metricType="compute.googleapis.com/instance/cpu/utilization";
BEGIN
IF (metricType="compute.googleapis.com/instance/cpu/usage_time")
THEN
SELECT m.value as InstanceName,metric.type as metricType,point.value.double_value as usagetime,point.interval.start_time as StartTime,point.interval.end_time as EndTime,h.value as instance_id FROM `myproject.metric_export.sd_metrics_export_fin`, unnest(resource.labels) as h,unnest(metric.labels) as m where metric.type='compute.googleapis.com/instance/cpu/usage_time' and h.key="project_id";
ELSE IF (metricType="compute.googleapis.com/instance/cpu/utilization")
THEN
SELECT m.value as InstanceName,metric.type as metricType,point.value.double_value as utilizationrate,point.interval.start_time as
StartTime,point.interval.end_time as EndTime,h.value as instance_id FROM `myproject-.metric_export.sd_metrics_export_fin`,unnest(resource.labels) as h,unnest(metric.labels) as m where metric.type='compute.googleapis.com/instance/cpu/utilization' and h.key="project_id";
END IF;
END IF;
END;
but after click "ADD" button i get the below error:
I am not sure what is this error about? I have not used any stored procedure and I am just pasting it as custom query.
Also If I try to save the results of the BigQuery into a view from the Bigquery console results pane, it gives me the error,
Syntax error: Unexpected keyword DECLARE at [1:1]
I am extremely new to datastudio and also to bigquery. Kindly Help thanks
First, I would like to make some considerations about your query. You are using Scripting in order to declare and create a loop within your query. However, since you declare and set the metricsType in the beginning of the query, it will never enter in the first IF. This happens because the value is set and it is not looping through anything.
I would suggest you to use CASE WHEN instead, as below:
SELECT m.value as InstanceName,metric.type as metricType,
CASE WHEN metric.type = #parameter THEN point.value.double_value ELSE 0 END AS usagetime,
CASE WHEN metric.type = #parameter THEN point.value.double_value ELSE 0 END AS utilizationrate,
point.interval.start_time as StartTime,point.interval.end_time as EndTime,h.value as instance_id
FROM `myproject.metric_export.sd_metrics_export_fin`, unnest(resource.labels) as h,unnest(metric.labels) as m
WHERE metric.type=#parameter and h.key="project_id";
Notice that I am using the concept of Parameterized queries. Also, for this reason this query won't work in the console. In addition, pay attention that whem you set the #parameter to "compute.googleapis.com/instance/cpu/utilization", it will have a non-null column with the usagetime and a null column named utilizationrate.
Secondly, in order to add a new data source in DataStudio, you can follow this tutorial from the documentation. After, selecting New Report, click on the BigQuery Connector > Custom Query> Write your Project id, you need to click in ADD PARAMETER (below the query editor). In the above query, I would add:
Name: parameter
Display name: parameter
Data type: text
Default value: leave it in blank
Check the box Allow "parameter" to be modified in reports. This means you will be able to use this parameter as a filter and modify its value within the reports.
Following all the steps above, the data source will be added smoothly.
Lastly, I must point that if your query ran in the Console you are able to save it as a view by clicking on Save view, such as described here.

SQL select query not working with variable parameter in my servelets

i'm trying to execute following line of code with my servelet in netbeans:
ResultSet rs = stmnt.executeQuery("select * from ZEE.WORDCOUNT where WORD =" + searchTxt);
where searchTxt is String variable.
but it says "Column 'zeeshan' is either not in any table in the FROM list or appears within a join specification and is outside the scope of the join . . . . ".
it really works fine, if i provide the hardcoded value instead of variable, as:
ResultSet rs = stmnt.executeQuery("select * from ZEE.WORDCOUNT where WORD= 'zeeshan'");
i'm not getting, what i'm missing?
You are missing the single quotes around the sql string you are constructing. So this should work:
ResultSet rs = stmnt.executeQuery("select * from ZEE.WORDCOUNT where WORD ='" + searchTxt+"'");
Please note that constructing SQL statements in this way is really dangerous, because it opens your application up for SQL injection attacks. Use bind parameters instead.
This will also allow better caching of parsed statements on many rdbms's.

Php mysql statement with set and select

I have a weird problem, when i use the query on phpmyadmin, it works. but when i use using a php script it returns an error.
Warning: mysql_fetch_array() expects parameter 1 to be resource, boolean given in
I tried to troubleshoot and discovered that the problem lies with the set statement.
this is my example code.
$sql = 'set #rank=0; select * from user;';
Please help somebody.
First Run
$sql = set #rank=0;
it will store value of rank
then run:
select * from user;
In sort you need to run both queries separately .
set statement stores values. that can be used by next executing query,
like code below :
$sql ="SET #id:=0";
$Executives=$DB->exec($sql);
$sql = "SELECT #id:=#id+1 as id,pes.* FROM profile_executive_summary as pes where profile_id=".$pid;
$Executives=$DB->fetchAssoc($sql);
See what mysql_error returns after you run mysql_query('...'). That might help. In general, mysql_query only permits one query. You can't separate them by newlines or semicolons. mysqli will do it for you though.