We have a partitioned table in google bigquery that we query using the _PARTITIONTIME field (otherwise the queries will cost too much).
How can I make Tableau use _PARTITIONTIME pseudo column?
When configuring your datasource in Tableau, select "Google BigQuery" from the list of available sources, go through the OAuth dance, and then select your project and dataset.
At this point, you will be presented with a list of tables in the dataset, as well as an option to use "New Custom SQL" at the bottom. Select this option, and enter your query exactly as you have been using it. Assuming that the query contains a segment similar to the below:
...WHERE _PARTITIONTIME BETWEEN TIMESTAMP(“2016-05-01”) AND TIMESTAMP(“2016-05-06”)
Now, highlight the dates within that where clause and click on the "Insert Parameter" dropdown menu in the query editor. This will allow you to parameterize your query and dynamically choose the dates you want to query from within your Tableau workbook!
Hopefully this helps!
A more flexible solution would be to just drag and drop the table as you would normally.
After that click 'datasource - convert to custom SQL'
This way tableau has written most of the SQL for you.
Last part....
DONT add a where clause and stop user exploration
Add this instead:
_PARTITIONTIME AS mypartitiondate
Now you can use this as you would any other date column but you are using partition time instead. Caveats to that are if you use this column you cant drill in below the level of the partitions (eg trying to see hourly trades on a daily partitioned dataset). That would require user knowledge to know to start using another date column for that part instead.
Related
I have tables of stocks in my BigQuery. The names of the table is simply the ticker of the stocks: AAPL, AMZN, MSFT, etc.
"{dataset_name}.AAPL"
"{dataset_name}.AMZN"
"{dataset_name}.MSFT"
I am on Google Sheets that's trying to create a query to dynamically SELECT from the tables one by one, based on the stock names.
So I'm trying to use a parameter on the table name, and I tried to use the wildcard function from Bigquery.
SELECT * FROM `dataset_name.*` WHERE _TABLE_SUFFIX = 'AAPL'
The result is Query valid. Will process 0 bytes. and returns nothing.
I then tried modifying the query and make it like this
SELECT * FROM `dataset_name.A*` WHERE _TABLE_SUFFIX = 'APL'
And it works...
The problem is, I cannot wildcard the whole table name. I can only use it for a suffix name, with at least a letter at the front of the name. However in my case, the whole table name is dynamic. How to make a query for a dynamic whole table name?
Any help would be appreciated.
This is an expected behavior in BigQuery, as per this GCP Documentation:
In order to execute a standard SQL query that uses a wildcard table, BigQuery automatically infers the schema for that table. BigQuery uses the schema for the most recently created table that matches the wildcard as the schema for the wildcard table.
It is possible in your scenario that you have tables in your dataset that has different schema, and BigQuery references to the most recently created table.
One way to dynamically SELECT without using _TABLE_SUFFIX in Bigquery is to use execute_immediate and put the suffix into a variable. However, this seems to be a limitation in Google Sheets. To request to have this feature in Google Sheets, you may file a Feature Request.
I have a custom SQL table I make in Tableau as a stand-alone datasource
SELECT SUM(AMOUNT)/100 AS MONTHLY_REWARDS,CLINIC_ID,DATE_TRUNC('month',INSERTED_AT) AS MONTH,'Total Amount' AS TYPE
FROM "MY_SCHEMA".REWARDS
WHERE CLINIC_ID IS NOT NULL
GROUP BY CLINIC_ID,MONTH,TYPE
I'm not modifying or casting CLINIC_ID in any way in this summary. Yet when I access this field in a workbook I can no longer use it as a User Filter, but I can for MONTH or TYPE from my summary.
And when I use just the REWARDS table as the datasource (no custom SQL), it has no trouble using it as a User Filter.
I've tried right-clicking in the Worksheet and converting to string.
When I do, it becomes a User Filter option, but using it as a filter results in no data (because it is actually an Int in the data). When I convert it back to Number (decimal), it is no longer a User Filter.
Finally, if I try to have the custom SQL with the rest of my tables and join on Clinic ID, it says the types do not match even though they come directly from the same table with no modifications.
What am I doing wrong and how can I start using Custom SQL as a table in these scenarios? I'd like to be able to use Clinic ID from the Custom SQL as a User Filter as well as a key to join to other tables if I want.
I am trying to import data from a SQL server into power BI. There is a section on the advanced options called SQL statement.
I know that the SQL statement for what I require is:
Select TOP 1000 * from [Table]
How do I write this in the Power Bi at the time of data source / import. So that it runs this statement for each of the tables I plan to import?
You can try this at the time of importing SQL Server data.
After loading data you can keep and remove rows using keep rows as shown below
If all the tables you want are on the same database, then you can navigate to that database as the first step in your query.
From there, filter down to select just the tables you want.
(You can see the preview of the cell selected in the bottom pane.)
Now that you've got the tables you want, you can apply a TopN function to the entire column (I chose top 3).
Table.TransformColumns(#"Filtered Rows",{{"Data", each Table.FirstN(_,3), type table}})
A quick way to add this step is to do a transformation on a text column and then just replace the column and the function applied. For example, if you format the Schema column to UPPERCASE using the GUI, it will add the step
Table.TransformColumns(#"Filtered Rows",{{"Schema", Text.Upper, type text}})
from which you can swap out the column, function, and type for what you actually want (see previous).
At this point, your tables are all trimmed to the top N rows and you can load each one to its own query by right-clicking on the table cell and choosing "Add as New Query". Alternatively, you can right-click on the Database query in the left pane (see the first image) and choose "Reference". This creates a new query from which you can simply click on the Table you want and it will return just that one.
Note: The former method will automatically name the new query after the table you expanded but the latter would work better if you wanted to change your N value since it doesn't recreate the whole query.
Either way, if you right-click on the last applied step in each of these new tables, you can choose "View Native Query" and you can see that the statement passed back to the server is a simple select top 3.
select top 3
[$Table].[DealSpecificKey] as [DealSpecificKey],
[$Table].[DateInvestment] as [DateInvestment],
[$Table].[DateInvestmentKey] as [DateInvestmentKey],
[$Table].[DateRedemption] as [DateRedemption],
[$Table].[DateRedemptionKey] as [DateRedemptionKey]
from [dbo].[AuxDaysInvested] as [$Table]
I've been searching the internet for hours trying to figure out if the following is even possible:
To choose the AS400 query records directly from Excel.
I haven't found any solution or description of how this could be achieved, which makes me guess that it's simply not possible. However, I haven't seen anyone confirm that it is impossible.
So my question is: Is this possible? And if it is, could you point me in the right direction in order for me to start learning how to do it?
I know its possible to run a query from Excel, and then adding parameters via SQL statements, but in my case, this presents several problems that could be avoided by choosing the records before the query is executed.
Example:
I have a query with a column (lets call it ColVal) that can hold the values 1 and/or 2. In the AS400 program under the menu "Work with queries" and then "Choose records" I can specify which records the query should contain when it has run based on the value in ColVal. This means i can get three different situations (A, B and C) when i run the query:
A) The query only contains records where the value in ColVal is 1
B) The query only contains records where the value in ColVal is 2
C) The query contains records where the value in ColVal is either 1 or 2
The goal is to be able to choose which situation I want from Excel in order to circumvent opening and using the AS400 program.
However, using situation C and then editing the query in Excel with an SQL statement to mimic situation A or B is not an option, as this means the query still contains undesired records.
This whole thing boils down to the following: Is it even possible to run the query from Excel essentially changing the data it contains and not just outputting it to excel? If this is possible, is it then possible to pass a parameter to the AS400 system and use it to create situation A, B or C?
I hope this example makes sense.
Edit - New example
Say i have different customers A and B. I can open the AS400 program and run a query in which i have specified that I only want data on customer A. I can then open Excel and use filters (as Hambone described) on the query to determine which records I want to output. However, if I want to work with data from customer B, I have to open the AS400 again and run the query with different parameters. I would like to be able to "change" my dataset from customer A to B from Excel, without having to include both in my recordset and then filter out one of them.
I imagined this is doable if you could pass a parameter to the AS400. The AS400 then runs the query using this parameter as the criteria for which records should be stored in the query. This means that if the parameter is Customer B, then there is no way to acces data from customer A, without running the query through AS400 again.
Any ideas are greatly appreciated :)
Follow up to my comment, here is a quick primer on how to run an ODBC query directly in MS Excel using Microsoft Query. This is very different than Power Query, which you referenced, in that MS Query is standard with Excel -- it's not a plug-in. This is relevant because it means everyone has it. If you are deploying a solution to others, that's an important consideration.
To start an MS Query in Excel, go to the data tab, select "From Other Sources" -> "Microsoft Query."
A list of your ODBC connections will come up. Pick the one that you want and select "OK."
It may or may not ask you for a login (depending on which ODBC connection you use and how its configured).
The next part is important. MS Query is going to try to have you use its builder to create the query. If you have the SQL, skip this part. It's horrible. Click "Cancel" on the query wizard, and then click the "SQL" button to enter your own SQL. If you can, make sure the result set is small (like use where 1 = 2 in the query).
When MS Query returns results, click the button next to the SQL Button to have it return the results to the spreadsheet. It looks like a little door.
From here, any time you want to refresh the query, you can simply right-click the data table in Excel and select "refresh." Alternatively you can go to the data tab on the ribbon and select "Refresh."
By the way if you have linked pivot tables and charts, the "Refresh All" option will refresh those as well, in the correct order.
To edit your query at any time, right-click on the table in Excel, go to Table-External Data Properties:
Then Click on the Connection Properties icon (highlighted below)
Click on the second tab (Definition) and edit the SQL Directly.
Parameters can be declared simply by inserting a bare "?" in place of your literal.
In other words, if your query looks like this:
select *
from users
where user_id = 'hambone'
Just change it to:
select *
from users
where user_id = ?
Excel will prompt you for a user id before it runs the query. From here, you also have the option of putting the parameter value in a cell within the spreadsheet and having the query read it from there. You'll see these when you right-click the table and go to the "Parameters" menu option.
Let me know if this helps or is unclear.
-- EDIT 7/23/2018 --
To follow up on your latest edit, it is possible to handle the scenario you describe, where you want to be able to filter on a value, or if none is given, then not have a filter. You see this a lot when you present multiple filter options to the user and you want a blank to mean "no filter," which is obviously counter to the way SQL works.
However, you can hack SQL to still make it work:
select * from activities
where
(activity = ? or ? is null) and
(energy = ? or ? is null)
In this example you have to declare four parameters instead of two, two for each.
You might also have to play with datatypes, depending on the RDBMS (for example for numerics you might have to say ? = 0 instead of ? is null or even ? = '' for text).
Here is a working example where a single filter was applied on the query above and you can clearly see the second one did not have an impact.
Yes it's possible. You need to use an ODBC driver to connect to the AS400 and retrieve the data. The driver and documentation are Here
I need help in creating an query interface with access database.
In brief, with this query interface I want to see calculated future dates for different steps of a process based on the date the process actually started.
The future dates will always be at a fixed number of days after the start date. I hope I am able to explain this in an understandable manner.
I was thinking of using access forms? Please help me in this. I am not sure of how to proceed with this.
If your data is already in MS Access, then using Access Forms would be your easiest method of displaying that data. If the data is elsewhere, such as in Sql Server, you may be better suited in the long run using a different display technology.
That being said, to select a number of dates, as you would do in Access, you can use the Date Add function. If you had a table Processes with a column StartDate, you could use the query
SELECT *, DateAdd("d",5,StartDate) as "5 Days", DateAdd("d",36,StartDate) as "36 Days"
FROM Processes
to generate a record set to bind your form to. Binding that query to a new form is easy. You just need to change the record source by:
Right click anywhere blank in your new form
Select Properties
Change to the data tab
Click the button next to the text box labeled "Record Source"
Build your query using the built in editor (or, to paste the given SQL, right click in the designer view and select SQL view, then paste)
Close the query building dialog and use your new fields. You can drag them from the field list onto the design surface.