We want to perform a test on BigQuery with scheduled queries.
The test retrieves a table from a dataset and, basically, copies it in another dataset (for which we have permission as owners) in another project. So far, we managed to do that with a script we wrote in R against the BigQuery API in a Google Compute Engine instance but we want/need to do it with scheduled queries in BigQuery.
If I just compose a query for retrieving the initial table data and I try to schedule it, I see there's a project selector but it's disabled so seems like I'm tied to the project for the user I'm logging in with.
Is this doable or am I overdoing it and using the API is the only option to do this?
Is this doable or am I overdoing it and using the API is the only option to do this?
The current scheduler logic doesn't allow this and for that reason, the project drop-down is disabled in the webUI.
As an example, I tried setting this schedule Job
CREATE TABLE IF NOT EXISTS `projectId.partitionTables.tableName` (Field0 TIMESTAMP) --AS SELECT * FROM mydataset.myothertable
And this is the error returning from the transferAPI
You will need to ask BigQuery team to add this option to future version of th scheduler API
I would like to make an app in Xcode where users get scored based on football results. For this, I need to update a table of football scores every weekend. What would be the best way to go about this? I assume it would be through SQL queries, but what is the industry standard for grabbing up to date data?
The solution is going to be dependent upon your DBMS but a couple ideas to get you started.
Utilizing SQL server you can created a job and schedule it run on a server a specific schedule, you can write TSQL code to look at CSV files and update your database tables
You can write an integration package with SSIS and schedule that package to run weekly
You can write a solution in Python or another scripting language and schedule that to run on your server with windows task manager
When running a report on the Tableau Cloud against BigQuery, there is no Query History.
Even when I refresh the datasource with a new day of data, and I see that the report now shows the new date, there is no Query History. I need to see the History to calculate costs and to understand what Tableau is doing behind the scenes.
Make sure to use same user for tableau and for retrieving query history
There is a flag administrators can use on the jobs API to retrieve jobs for all users in the project they administer. See the allUsers flag described at:
https://cloud.google.com/bigquery/docs/reference/v2/jobs/list
This functionality is not available in the UI but you can use the command line tool (bq) to retrieve jobs across users. It has a flag that enables this option.
I have an SQL Azure Database instance which provide data to a windows 8 app. The data in my database should be updated periodically (weekly). Is the any way to make it? I'm thinking of write a app which will run weekly and update the database. But still don't know how to make it run on Window Azure? Please help!
Thank you,
There are a number of ways to achieve this, does the data however need to come from a different source or can it be calculated?
Either way, seeing as you're already knee deep in SQL Azure I would suggest putting your logic into a worker role that can be scheduled to run your updates once a week. This would give you a great opportunity to do calculations and/or fetch data externally.
Azure gives you the flexibility to scale this worker role into numerous instances as well depending on the work load.
Here is a nice intro tutorial on creating a worker role on Azure: link
Write the application and set it to run through your cron job manager on a weekly time schedule. The cron job manager should be provided by your host.
I have an archiving process that basically deletes archived records after a set number of days. Is it better to write a scheduled SQL job or a windows service to accomplish the deletion? The database is mssql2005.
Update:
To speak to some of the answers below, this question is regarding an in house application and not a distributed product.
It depends on what you want to accomplish.
Do you want to store the deleted archives somewhere? Log the changes? An SQL Job should perform better since it is run directly in the database, but it is easier to give a service acces to resources outside the database. So it depends on what you want to do,,,
I would think a scheduled SQL job would be a safer solution since if the database is migrated to a new machine, someone doing the migration might forget that there is a windows service involved and forget to start/install it on the new server.
In the past we've had a number of SQL Jobs run. Recently, however, we've been moving to calling those processes from .Net code as a client application run from a windows schedule task, for two reasons:
It's easier to implement features like logging this way.
We have other batch jobs that don't run in the database, and therefore must be in windows scheduled tasks. This way all the batch jobs of any type will be listed in one place.
Please note that regardless of how you do it, for this task you do not want a service. Services run all day, and will consume a bit of the server's ram all day.
In this, you have a task you need to run, and run once a day, every day. As such, you'd either want a job in SQL Server or as Joel described an application (console or winforms) that was setup on a schedule to execute and then unload from the server's memory space.
Is this for you/in house, or is this part of a product that you distribute.
If in house, I'd say the SQL job. That's just another service too.
If it's part of a product that you distribute, I would consider how the installation and support will be.
To follow on Corey's point, if this is externally distributed, will you need to support SQL Express? If not, I'd go with the SQL job directly. Otherwise, you'll have to get more creative as SQL Express does not have the SQL Agent that comes with the full versions of SQL 2005 (as well as MSDE). Without the SQL Agent, you'll need another way to automatically fire the job. It could be a windows service, a scheduled task (calling a .NET app, powershell script, VBscript, etc.), or you could try to implement some trigger in SQL Server directly.