For Data Visualisation purposes, I am using Databricks to create dashboards.
This is achieved by creating charts in a notebook and linking those charts to the dashboard.
But I'm facing an issue in refreshing the data linked to the dashboard.
I tried creating a job to refresh the entire notebook. The job succeeded but the dashboard linked to it was still showing previous data.
I believe Databricks Jobs creates a separate instance, runs the workflows and discards the instance.
With this, I won’t be able to get the updated data in the dashboard. I have to manually click on the Run All button.
Is there any other way this could be done automatically?
Related
I started BigQuery trial account just before
and created sample table using a source with Drive URI & CSV.
When I clicked my uploaded table, it shows table information like schema and details.
I remember that there was a preview menu but I can't find it now.
Is there anyway to activate table preview menu?
When your table is created from a file from Google Drive, by default "Preview" tab is not available since it is considered as an external table. I tested it to verify the behavior.
Created table sourcing from Google Drive:
What I could suggest is to load your data from Google Drive to BigQuery, so the "Preview" tab will be available. Also if you load your data in BigQuery, running queries will be faster compared from querying the table externally.
I have a view that is connected to a google sheet via connected sheets.
I'm trying to let a user refresh the data by giving them access in GCP.
I've tried giving access at the project, dataset and view levels. But every time they get the error: "Query failed, no access to the connected BigQuery table"
I'm giving the role of bigquery.user and bigquery.dataviewer.
What could be causing this?
Please, make sure if the user you wish to give access to the BigQuery data in Google Sheets has:
An Enterprise Plus or G Suite Enterprise for Education account
Access to BigQuery
A project with billing setup in BigQuery
A BigQuery Job Creator role on the selected billing project
BigQuery Data Viewer role on the datasets containing the selected table
According to the documentation:
If you share a sheet with someone who doesn't meet the criteria
above, they'll be able to see analysis created with Connected Sheets
and perform regular Sheets operations, but they won't be able to
refresh it or create their own connected sheet.
Additionally, have a look for the another SO thread and Using Connected Sheets documentation.
I connected my PostgreSQL DB to Power BI and then built a dashboard. Now I want to refresh the dashboard automatically once a day. I saw that there is an option to do that using the Power BI web version, so I have published the dashboard. In addition, I have pinned the report to the live view option. Then, I have determined the schedule refresh and added a new row to the DB. After the refresh time passed, nothing happened. I pressed the Refresh button in the Power BI web version, and still, nothing happened. However, when I went back to the Power BI desktop application and pressed the refresh button, the dashboard did refresh.
What am I doing wrong?
Thanks!
It looks like your PostgreSQL database is installed on-premise and Power BI Online server, which runs in the cloud, can't connect to your database, which runs in your internal network.
You need to install Power BI Gateway to allow Power BI to connect to your database, and configure it.
Your Power BI app is pulling data right from the source. The on-line web version requires a data gateway, either personal or on-prem depending on your business needs. Remember you can only use live refresh with a single source.
https://powerbi.microsoft.com/en-us/gateway/
I'm attempting to connect BigQuery to Looker. I am pulling sample data from a Google Sheets document to a BigQuery dataset; this part is working fine, as my internal BigQuery queries are running just fine for this dataset. Using this documentation from the Looker forums, I tried to create a service account key to connect my BigQuery dataset to Looker. Unfortunately, the documentation is slightly out of date: Google now asks which service account (compute engine default service account, app engine default service account, or a new service account that can have any of multiple roles) you want to attach the key to.
Thus far, I have tried using P12 keys created for the compute engine default service account, the app engine default service account, as well as a new Project Owner service account. When I create the connection in Looker, the admin page confirms that the connection "can connect, can cancel queries, can run simple select query" (I need it to do more complex things, but am just trying to connect at all right now). Using the SQL Runner to test a simple select 10 query out, I was able to query the public datasets, e.g. hacker_news or usa_names. However, whenever I tried to run the same query on my personal sample dataset, I received this error:
Failed to retrieve data - The job encountered an internal error during execution and was unable to complete successfully.
The permissions for the base Google Sheet that the BigQuery project is pulling from are set to be viewable by my coworkers who have the link. I have also been adding each service account I test as an editor (which I assume has the highest permissions). At this point, I am creating new service accounts with each of the different possible roles to see if it's a permissions issue from the role perspective. Nothing has worked so far, so any insight would be helpful!
UPDATE: I have created a new table within the same BigQuery dataset. The new table was created using a CSV file, which was simply a download of my previous table in Google Sheets. I updated the connection to Looker. When I wrote a select 10 query pulling from the new table, it worked fine and ran very quickly. This seems to imply that the problem is something about the permissions between Google Sheets and Google BigQuery.
I've been wanting to do something like this myself for a bit, saw this question, and decided to dig in.
First thing I found was this "documentation" over in the looker discourse:
https://discourse.looker.com/t/live-spreadsheets-in-databases/2698/7
In there, it describes the steps necessary to get this working.
Two important things that you are probably missing, based on your description of events so far (since it sounds like you've already attached the sheet to your dataset and are able to query it from the BigQuery UI):
Make sure you share the Google Sheet with the service account you are using to connect Looker to BigQuery. This is the Username from the Connections tab of the Admin page in Looker.
Make sure you have enabled the Drive and Sheets APIs for your google project. You can do that via The API Library. Just search for "Drive" (or "Sheets"), click on the name, and then click on the "Enable" button from the API detail page.
Once I did the above, I had to wait a few minutes before things started working. I'll go out on a limb and guess that this was because Looker needed to cycle it's internal connection pool before the permissions would reset and work. So you may need to run a few failing queries, or wait out the connection pool before this will go into effect.
Hope that helps.
When running a report on the Tableau Cloud against BigQuery, there is no Query History.
Even when I refresh the datasource with a new day of data, and I see that the report now shows the new date, there is no Query History. I need to see the History to calculate costs and to understand what Tableau is doing behind the scenes.
Make sure to use same user for tableau and for retrieving query history
There is a flag administrators can use on the jobs API to retrieve jobs for all users in the project they administer. See the allUsers flag described at:
https://cloud.google.com/bigquery/docs/reference/v2/jobs/list
This functionality is not available in the UI but you can use the command line tool (bq) to retrieve jobs across users. It has a flag that enables this option.