I've been reading up on how to query a wildcard table in BigQuery, but Data Studio doesn't seem to recognize the _TABLE_SUFFIX keyword.
Google Data Studio on using parameters
Google BigQuery docs on querying wildcard tables
I'm trying to use the recently added date parameters for a custom query in Data Studio. The goal is to prevent the custom query from scanning all partitions to save time.
When using the following query:
SELECT
*
FROM
`project-name.analytics_196324132.events_*`
WHERE
_TABLE_SUFFIX BETWEEN DS_START_DATE AND DS_END_DATE
I receive the following error:
Unrecognized name: _TABLE_SUFFIX
I expected the suffix keyword to be recognized so that the custom query is more efficient. But I get this error message. Does Data Studio not yet support this? Or is there another way?
It could be possible that you are setting the query in the wrong place. I created a DataSource from a Custom Query and the wildcard worked. The query I tested was the following, similar to yours since _TABLE_SUFFIX is a wildcard that is available in standardSQL in BigQuery:
select
*
from
`training_project.training_dataset.table1_*`
where
_TABLE_SUFFIX BETWEEN '20190625' AND '20190626'
As per your comments you are trying to add a query in the formula field of a custom parameter, however the formula field only accepts basic math operations, functions, and branching logic.
The workaround I can see is to build a select query and use it as a Custom Query in the Data Source definition so that the query can calculate any extra fields in advance (steps 5,6 and 7 from this tutorial).
Related
I am quite new to GBQ and any help is appreciated it.
I have a query below:
#Standard SQL
create or replace table `xxx.xxx.applications`
as select * from `yyy.yyy.applications`
What I need to do is to add today's date at the end of the table name so it is something like xxx.xxx.applications_<todays date>
basically create a filename with Application but add date at the end of the name applications.
I am writing a procedure to create a table every time it runs but need to add the date for audit purposes every time I create the table (as a backup).
I searched everywhere and can't get the exact answer, is this possible in Query Editor as I need to store this as a Proc.
Thanks in advance
BigQuery doesn't support dynamic SQL at the moment which means that this kind of construction is not possible.
Currently BigQuery supports Parameterized Queries but its not possible to use parameters to dynamically change the source table's name as you can see in the provided link.
BigQuery supports query parameters to help prevent SQL injection when
queries are constructed using user input. This feature is only
available with standard SQL syntax. Query parameters can be used as
substitutes for arbitrary expressions. Parameters cannot be used as
substitutes for identifiers, column names, table names, or other parts
of the query.
If you need to build a query based on some variable's value, I suggest that you use some script in SHELL, Python or any other programming language to create the SQL statement and then execute it using the bq command.
Another approach could be using the BigQuery client library in some of the supported languages instead of the bq command.
I have a table with a lot of data that I don't need, but I have to save that, however to do my job i need only some data.
so, to limit the amount of data that my query will return I have to use a filter to not get some files like the strings .js, .css, .log etc.
I'm using the "Microsoft Azure Explorer - Azure Table Storage - Query Editor" to do this, but I don't know how to use a function like SQL function "like" to filter that strings.
follow a example of data that i don't need in my return:
lg.folha.views.demonstrativodemedias.editar.min.css
27042018_144634_536_6A2FAE_AMB..xml
is it possible to filter that files using only the query builder options?
Query Builder Options
Query builder documentation: https://learn.microsoft.com/en-us/rest/api/storageservices/Querying-Tables-and-Entities?redirectedfrom=MSDN#Anchor_2
Fro now azure table doesn't support wildcard query. You could get it from here:Filtering on String Properties.
Note that the Table service does not support wildcard queries.
So you have to use eq, ne, gt, ge, lt, le these operators.
You could use property ge 'test' and RowKey lt 'test~' to get query entities whose property starts with test.
So, the Azure Table service doesn't support the function like, so I did an application in C# to do this.
doing that I solved two problems:
1 - The like function - I created a function to exclud the worong files.
2 - the excel line limit - Excel have a limit of 1.048.576 and some of my stracts return more than this, so a did a function to split the list in N Excel's File.
I'm trying the following query from the BigQuery Standard SQL documentation:
SELECT DATE_DIFF(DATE '2010-07-07', DATE '2008-12-25', DAY) as days_diff;
https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#date_diff
However, I'm receiving the following error from the UI:
Error: Encountered " "\'2010-07-07\' "" at line 1, column 23. Was expecting: ")" ... [Try using standard SQL (https://cloud.google.com/bigquery/docs/reference/standard-sql/enabling-standard-sql)]
This is a simple copy and paste from the doc into the web UI Query Editor.
Any idea on how to resolve this?
Below are examples for respectively BigQuery Legacy SQL and Standard SQL
Make sure you try code as it is in answer below - not just second lines but 2(both) lines including first line that looks like comment - but in reality important part of query as it controls which SQL dialect will be in effect!
#legacySQL
SELECT DATEDIFF(DATE('2010-07-07'), DATE('2008-12-25')) AS days_diff
and
#standardSQL
SELECT DATE_DIFF(DATE '2010-07-07', DATE '2008-12-25', DAY) AS days_diff
both returns result as below
Row days_diff
1 559
Ideally, you should consider migrating to Standard SQL
Although the answer has already been provided in the comments to your questions and by Mikhail in the other answer, let me share with you a complete answer that hopefully addresses all your doubts:
ERROR MESSAGE
As explained in the error message you are getting, [Try using standard SQL (...)]. You are trying to run this sample using Legacy SQL (which instead would use the DATEDIFF function). You are actually right, you are running the exact same query provided in the documentation, but the issue here is that the documentation you are using is for Standard SQL (the preferred query language in BigQuery), but you are instead using Legacy SQL (the default language in the old UI, the one you are using).
CHANGE THE QUERY LANGUAGE IN USE
First of all, I would like to remark the importance of using Standard SQL instead of Legacy SQL, as the former adds new functionalities and is the current recommended language to use with BigQuery. You can see the whole list of comparisons in the documentation, but if you are starting with BigQuery, I would just go straight away with Standard SQL.
Now, that being clarified, in order to use Standard SQL instead of Legacy SQL, you can have a look at the documentation here, but let me summarize the available options for you:
In the BigQuery UI, you can toggle the Use legacy SQL option inside
the Show options menu. If this option is marked, you will be using
Legacy SQL; and if it is not, you will be using Standard SQL.
You can use a prefix in your query, like #standardSQL or #legacySQL, which would ignore the default configuration and use the language you specify with this option. As an example on how to use it, please have a look at the other answer by Mikhail, who shared with you a couple of examples using prefixes to identify the language in use. You should copy the complete query (including the prefix) in the UI, and you will see that it works successfully.
Finally, as suggested by Elliott, you can use the new UI, which has just recently released in Beta access. You can access it through this link https://console.cloud.google.com/bigquery instead of the old link https://bigquery.cloud.google.com that you were using until now. You can find more information about the new BigQuery Web UI in this other linked page too.
I'm using BigQuery Standard SQL.
I'm trying to use a "$" decorator on a table in order to refer to a specific partition:
SELECT user_id
FROM `raw.events$20161109`
And I'm getting the next error:
Table "raw.events$20161109" cannot include decorator Dismiss
I am able to run the query (bq validation is ok), and the error pops right after I click on "Run Query" button.
When I use the Legacy SQL
I have no problem of doing it:
SELECT uid
FROM [raw.events$20161109]
Is there any way to run a query using the decorators with Standard SQL ?
I have to do it this way, as a lot of other procedures are based on this decorators format (using the Legacy SQL)
At this time Table Decorators in BigQuery are only available when using Legacy SQL. There is an open feature request which can be followed to see the progress on bringing this functionality to Standard SQL.
I am trying to configure Microstrategy to work with MongoDB. The Mstr advised way is to use Simba ODBC driver. The simple connection works fine. The problems start when I want to use functions e.g. get only hour out of the timestamp.
The other approach I tried is to use Apache drill and I face exactly the same problem.
Select code, name from offer
Code and name are attributes of some documents in collection called offer. This works fine.
Select date(interactionDateTime) from interactionrecord
This fails. I tried different syntax postgres - date_part, to_date - Oracle, another one from MySQL..., EXTRACT etc.
You should be able to use the scalar functions listed here: https://msdn.microsoft.com/en-us/library/ms714639(v=vs.85).aspx
To extract the hour out of a time, use the HOUR() scalar function.