Big query make a quick backup with many tables - google-bigquery

Question

Currently I'm copying tables with something like this:
#!/bin/sh
export SOURCE_DATASET="BQPROJECTID:BQSOURCEDATASET"
export DEST_PREFIX="TARGETBQPROJECTID:TARGETBQDATASET._YOUR_PREFIX"
for f in `bq ls -n TOTAL_NUMBER_OF_TABLES $SOURCE_DATASET |grep TABLE | awk '{print $1}'`
do
export CLONE_CMD="bq --nosync cp $SOURCE_DATASET.$f $DEST_PREFIX$f"
echo $CLONE_CMD
echo `$CLONE_CMD`
done
(script from here), but it takes ~20min (because of ~600 tables). Maybe there is another way (preferably faster), to make a backup?

Answer 1

As a suggestion you may use Scheduling queries to schedule recurring queries in BigQuery, with this option you will be able to schedule your backups on a daily, weekly, monthly or custom periodicity, leaving the backups of your tables for nights or weekends. You can find more information about it in the following link.
But remember, the time that you backup takes will depend on your tables size.
Well, due to you mentioned that Scheduling queries is not an option for you, another option you can try is run your cp command in the background, this because you are working with a for loop and you are waiting to finish each process, instead of that you can run multiple process in background to get better performance. I made a simple script to test it and it works! First I made test without background process:
#!/bin/bash
start_global=$(date +'%s');
for ((i=0;i<100;i++))
do
start=$(date +'%s');
bq --location=US cp -a -f -n [SOURCE_PROJECT_ID]:[DATASET].[TABLE]
[TARGET_PROJECT_ID]:[DATASET].[TABLE]
echo "It took $(($(date +'%s') - $start)) seconds to iteration umber:
$i"
done
echo "It took $(($(date +'%s') - $start_global)) seconds to the entire
process"
It takes me around 5 seconds per table copied (160 Mb approx), so I spend more less 10 minutes in that process, so I modified the script to use background process:
#!/bin/bash
start_global=$(date +'%s');
for ((i=0;i<100;i++))
do
bq --location=US cp -a -f -n [SOURCE_PROJECT_ID]:[DATASET].[TABLE]
[TARGET_PROJECT_ID]:[DATASET].[TABLE] &
pid_1=$! # Get background process id
done
if wait $pid_1
then
echo -e "Processes termination successful"
else
echo -e "Error"
fi
echo "It took $(($(date +'%s') - $start_global)) seconds to the entire
process"
In this way I only spend 3 minutes to finish the execution.
You may adapt this idea to your implementation, just consider the quotas for Copy jobs, you can check it here.