I would like to see if any abnormal data loading issues in Bigquery tables through alerts based on some rules, Any possibility to do that with cloud logging and alert policies ?
Your question is a bit unclear, are you looking for notifications about the table itself (load times / size etc) or what is in the table (the data)?
For what the table itself has already answered it here BigQuery - Scheduled Query Update Notification Email
If its what is in the table you are better off using python to automate this, there are plenty of tutorials about how to link BigQuery to Python and automate it and below is a similar query that should point you on the right track for automating an email from a condition
How to send email alert through python if a string is found in a csv file?.
Related
I have a Firebase application which is uploading events with parameters. I need to be able to view those events in order to debug some issues we're having in production. I can only see the tables which are generated nightly in BigQuery. I can find references online saying that BigQuery allows viewing real time data. What I can't find is any straightforward instructions on how to create those views.
Is it possible? If so, can someone give me instructions that even a complete newb could follow?
We have decided to use the BigQuery APIs for information we want to see immediately in the database.
I have a View (Table A) in Big Query which was created from a Google Sheet. It updates live which is perfect.
I have then connected that View to another View (Table B) in Bigquery. Let’s call this combined View, Table C.
In Tableau Desktop I try to connect to Table C, but it comes up with an authentication issue because Tableau cannot pass on authentication to Google Sheets.
Has anyone found a solution or workaround? Using service accounts, or even cloud functions or a scheduled query which saves the results of Table A as a table every time the google sheet is saved.
This has been asked before in the following link, but hasn’t received a step-by-step solution and I do not have enough stackoverflow reputation to comment:
BigQuery Credential Problems when Accessing Google Sheets Federated Table
https://community.tableau.com/thread/207871
We want to perform a test on BigQuery with scheduled queries.
The test retrieves a table from a dataset and, basically, copies it in another dataset (for which we have permission as owners) in another project. So far, we managed to do that with a script we wrote in R against the BigQuery API in a Google Compute Engine instance but we want/need to do it with scheduled queries in BigQuery.
If I just compose a query for retrieving the initial table data and I try to schedule it, I see there's a project selector but it's disabled so seems like I'm tied to the project for the user I'm logging in with.
Is this doable or am I overdoing it and using the API is the only option to do this?
Is this doable or am I overdoing it and using the API is the only option to do this?
The current scheduler logic doesn't allow this and for that reason, the project drop-down is disabled in the webUI.
As an example, I tried setting this schedule Job
CREATE TABLE IF NOT EXISTS `projectId.partitionTables.tableName` (Field0 TIMESTAMP) --AS SELECT * FROM mydataset.myothertable
And this is the error returning from the transferAPI
You will need to ask BigQuery team to add this option to future version of th scheduler API
I'm trying to write data from Google Spreadsheets to a BigQuery Table.
Are there any sources which I can tap into to learn how to do this?
(Something like this would be awesome:
https://developers.google.com/apps-script/articles/bigquery_tutorial)
Thanks.
What have you tried so far?
Before writing actual code, I see two ways:
Send data through a POST request, using https://developers.google.com/apps-script/reference/url-fetch/url-fetch-app and https://developers.google.com/bigquery/loading-data-into-bigquery#loaddatapostrequest.
Otherwise you could upload the data to Google Cloud Storage, and insert a job that loads it into BigQuery. Take a look at http://blog.knoldus.com/2013/01/19/google-apps-script-to-store-data-on-google-cloud-sorage/.
I am using BigQuery for SEO reasons. I am a search TC and I am a little confused why you are not using the Google Forum as I thought that was standard. What I want to use BigQuery for is to pull when my competitors change data on their website and which pages that were changed. So I need the URL that was changed and when it was changed (date) so I can also pull the page title and description to see what they are doing different than I am.
Is there anyone that knows how to use BigQuery to pull:
Date the page was changed
URL
Title
Description
We've switched to using Stack Overflow for support for many of our developer products, such as BigQuery. There's a great community here on StackOverflow, and the interface for formatting technical questions and interacting with the community is fantastic.
BigQuery does not collect the data for you-- it's a cloud service for performing ad hoc queries on massive datasets. Before first performing the queries, you need to upload the data to the service (as a CSV format).
So, if you have a job which collects this data -- URL, title, description, date and perhaps a hash of the webpage, you could potentially ingest a CSV file of this data into BigQuery and use it to understand when webpages have changed.
Of course, there are also 3rd-party services (such as Changedetection.com) which may be easier to use for your purposes.