More of a curiosity question really. I load data into Power BI report from Google BigQuery (using native Google BigQuery connector in Power BI). All works fine, but for some reason I don't see this query in BigQuery's query history.
Did anyone experience something similar and knows the reason why this happens or how to change that (if at all possible)?
If I do exactly the same thing but using simba ODBC connector, I see this query in BigQuery's query history as expected.
Never seen that before. I am always able to find the query history no matter what 3rd party connection I used. Could you confirm the GCP service-account or auth-account and the GCP project for BQ job query that you used for your native Google BigQuery connector in Power BI?
Please make sure you have the access to the query history of that GCP account in that BQ job project.
Related
I have an App Engine scheduled job which runs everyday and look for rows in a PostgreSQL table (hosted in gcp not a cloudsql) which meets a criteria to archive. If the criteria is met, it connects to BigQuery and streams the data to big query. Everyday there are few records qualify for archiving and we write to BigQuery. Is this the cost effective way or we can try loading data using Cloud Functions? https://cloud.google.com/solutions/performing-etl-from-relational-database-into-bigquery
App Engine and Cloud Functions have different purposes. You should use App Engine if you want to deploy a full application in a serverless environment. If you need to integrate services in the cloud, use Cloud Function. In your case it seems that Cloud Functions fits better.
It's important to remember that Cloud Function has a time limitation: the maximum time which your code has to run is 9 minutes.
You can find this and other limitations here
Furthermore, you can find here a pricing calculator for GCP products.
If you have any further questions, please let me know.
I'm having datasets in bigquery google cloud project,
I want to load this data from powerbi,So that I can create visualizations using this bigquery data sets.
Currently I'm able to see the google bigquery in getdata option in powerbi desktop,I tried to connect to bigquery but I'm not abe to see my bigquery dataset in the list,I'm using the same credentials for powerbi and google bigquery.
is there any thig that I need to enable in google cloud side so that it will be visible in the list when I connect from powerbi?
These are the following steps you can follow to connect GBQ to Power BI.
You can write a query and save the table in Google big Query.
Go in Power BI, get data and click on more and scroll down there you will find "Google Big Query" click on that.
Search your saved table and click on "Direct Query"
I think you will be good to go!!
I would not recommend using Power BI for Big Query analysis. As of Q1 2019 Power BI still do not support project billing unless manually editing the connexion script, it does not support Sheets federated tables, do not support nested and repeated fields and Power BI DML is nearly unusable with BQ. Microsoft do not seems to be willing to develop the compatibility between Power BI and Big Query further.
Learn more in this article (i am the author): https://medium.com/#remy_david/which-bi-tool-for-big-query-d9eb838ff7ad
Im working with Google Analytics 360 with a big client and I want to download all the data to BigQuery.
The objective is to have the data available to create some AI algorithms and make a deep analyse.
I want to connect to bigquery from a desktop SQL client like execute query, is this possible?
Thanks!
I want to connect to bigquery from a desktop SQL client like execute query, is this possible?
Check Simba Drivers for Google BigQuery
The intent of these drivers is to help users connect the power of Google BigQuery with existing tooling and infrastructure that does not have native integration. If you are building new integrations with BigQuery, we recommend you consider the native API. The functionality exposed by an ODBC/JDBC connection is necessarily a subset of the full capabilities of BigQuery.
Meantime, asking just in case - have you tried already Native UI?
- https://bigquery.cloud.google.com
I finally decided to ask this (after a lot of google searching):
So we use Power BI for data visualization and thus in it are some calculated dashboards / data outputs which are used to monitor data quality etc. I want to be able to historical log these results so that over-time we can monitor progress i.e. was data quality improved. This is the end of the initial problem.
One approach to this problem was to connect to PowerBI from the MS-SQL side - hoping we can then set timed triggers to do the log by READING THE POWER-BI DASHBOARDS: So how do I query that (I have already developed a method to determine the connection using the Power-BI port as described here:
EXPORTING DATA FROM POWER BI DESKTOP TO MS-SQL
This is a screenshot from one of my MS-SQL connections through "Analysis Services":
I am assuming the objects named like "LocalDateTable_" are the actual BI analysis I want to query. "New Query" is an MDX type of Query. Should I go this route for my problem (logging powerbi analyses)?
At first this sounds crazy but on reflection I guess it was only a matter of time, and a sign of the maturity of Power BI solutions ...
I would use the SQL Server Profiler to capture the queries generated while you use your dashboard & report.
https://insightsquest.com/2017/05/07/profiler-trace-for-power-bi-desktop/
Then I would build an SSIS package to run the MDX queries and deliver the datasets to SQL Server, with extra columns e.g. StartTime.
We are very pleased with the combination BigQuery <-> Tableau Server with live connection. However, we now want to work with a data extract (500MB) on Tableau Server (since this datasource is not too big and is used very frequently). This takes too much time to refresh (1.5h+). We noticed that only 0.1% is query time and the rest is data export. Since the Tableau Server is on the same platform and location, latency should not be a problem.
This is similar to the slow export of a BigQuery table to a single file, which can be solved by using "daisy chain" option (wildcards). Unfortunately we can't use similar logic with a Google BigQuery data extract refresh in Tableau...
We have identified some approaches, but are not pleased with our current ideas:
Working with incremental refresh: our existing BigQuery table rows can change: these changes can only be applied in Tableau if you do a full refresh
Exporting the BigQuery table to GCS using the daisy chain option and making a Tableau data extract using the Tableau SDK: this would result in quite some overhead...
Writing a Dataflow job using a custom sink for Tableau Server (data extracts).
Experimenting with a Tableau web connector that communicates directly with the BigQuery API: I don't think this will be faster? I didn't see anything about parallelizing calls with the Tableau web connecector, but I didn't try this approach yet.
We would prefer a non-technical option, to limit maintenance... Is there a way to modify the Tableau connector to make use of the "daisy chain" option for BigQuery?
You've uploaded the data in BigQuery. Can't you just use the input for that load job (a CSV perhaps) as input for Tableau?
When we use Tableau and BigQuery we also notice that extracts are slow but we generally don't do that because you lose BigQuery's power. We start with a live data connection at first, and then (if needed) convert this into a custom query that aggregates that data into a much smaller datasets which extracts in just a few seconds.
Another way to achieve higher performance with BigQuery and Tableau is aggregating or joining tables on beforehand. JOINs on huge tables can be slow, so if you use a lot of those you might consider generating a denormalised dataset which does all of the JOIN-ing first. You will get a dataset with a lot of duplicates and a lot of columns. But if you select only what you need in Tableau (hide unused fields!) then these columns won't count in your query cost.
One recommendation I have seen is similar to your point 2 where you export the BQ table to Google Cloud Storage and then use the Tableau Extract API to create a .tde from the flat files in GCS.
This was from an article on the Google Cloud site so I'd assume it would be best practice:
https://cloud.google.com/blog/products/gcp/the-switch-to-self-service-marketing-analytics-at-zulily-best-practices-for-using-tableau-with-bigquery
There is an article here which provides a step by step guide to achieving the above.
https://community.tableau.com/docs/DOC-23161
It would be nice if Tableau optimised the BQ connector for extract refresh using the BigQuery Storage API. We too have our Tableau Server environment in the same GCP zone as our BQ datasets and experience slow refresh times.