I have a dataset, which consists of ~600 tables. I want to create a new dataset with no tables and same name using CLI.
At the moment I'm iterating through all of the tables and dropping them with "bq rm" 1 by one, but it takes ~20min. Maybe I can simply drop dataset without removing the tables first?
Use the -r flag. For example:
bq rm -rf dataset_name
The -f flag means "force", so the command won't prompt for confirmation.
You can just simply use contextual menu in BigQuery UI to delete dataset
When you try and dataset has tables in it - you will see below form
Just type your dataset_name to confirm deletion - and you are done!
Related
We have a BigQuery dataset that has some long list of tables (with data) in it. Since I am taking over a data pipeline, which I want to familiarize myself with by doing tests, I want to duplicate those dataset/tables without copying-truncating the tables. Essentially, I want to re-create those tables in a test dataset using their schema. How can this be done in bq client?
You have a couple of options considering you don't want to copy the data but the schema:
1.- extract the schema for each table and then create new ones just empty.
$ bq show --schema --format=prettyjson [PROJECT_ID]:[DATASET].[TABLE] > [SCHEMA_FILE]
$ bq mk --table [PROJECT_ID]:[NEW_DATASET].[TABLE] [SCHEMA_FILE]
2.- run a query with LIMIT 0 and setting a destination table.
bq query "SELECT * FROM [DATASET].[TABLE] LIMIT 0" --destination_table [NEW_DATASET].[TABLE]
I have a few processes that yield a partitioned table in BigQuery and, when the datasource has issues I end up with, say 100 partitions of a table that are useless and I need to delete them.
Does anyone know how to achieve that via the BQ UI?
I've tried using the DML with no result... apparently it only deletes the content within the partitions, not the partitions.
Any particular reasons to physically delete the partitions instead of clear content? Alternatively, you can try
bq rm -f -t [PROJECT_ID]:[DATASET].[TABLE]$[20170101]
Is there a way to copy a date-sharded table to another dataset via the bq utility?
My current solution is generating a bash script to copy each day one-by-one and splitting the work, but more efficient would be to do everything in parallel:
#!/bin/sh
bq cp old_dataset.table_20140101 new_dataset_20140101
..
bq cp old_dataset.table_20171001 new_dataset_20171001
You can specify multiple source tables but only a single destination table (refer to this question), so this may not work for you. However, if your data is date-partitioned (instead of sharded) then you can copy the table in one command.
I recommend you convert the sharded table into a date-partitioned table which will be effectively copying all the sharded tables to a new table. You can do this with the following command:
bq partition old_dataset.table_ new_dataset.partitioned
I'm using the Google SDK bq command. How do I change the name of a table? I'm not seeing this at https://cloud.google.com/bigquery/bq-command-line-tool
You have to copy to a new table and delete the original.
$ bq cp dataset.old_table dataset.new_table
$ bq rm -f -t dataset.old_table
I don't think there is a way just rename table
What you can do is COPY table to a new table with desired name (copy is free of charge) and then delete original table
The only drawback I see with this is that if you have long term stored data - I think you will lose storage discount (50%) for that data
In my work came into a set of tables which all have the same schema. Until now no one made descriptions to the columns in the table and I want to add my description to them. My question is how to add my schema's descriptions to all the tables (about hundreds of them) without going every table and change it manualy?
Thanks!
By using bq command line tool, you can prepare a quick script on Linux.
Firstly, list all the tables by using "bq show" command, secondly in a loop put the needed descriptions to the tables by using "bq update" command.
Just for further help, here a link explaining how to write shell scripts:
http://www.wikihow.com/Write-a-Shell-Script-Using-Bash-Shell-in-Ubuntu