Bulk delete of data transfers in BigQuery - google-bigquery

Question

over time, many different people have created a test transfer in our demo project and never cleaned it up. I would love to bulk delete all transfers. Does anyone know of a way to do this?

Answer 1

I've built something similar on Cloud Workflows.
Essentially you need a Workflow that will do the steps for you start with BigQuery Data Transfer API and issue a few commands:
get a list of transfer configs
loop through them
issue a delete API call for them
https://cloud.google.com/bigquery-transfer/docs/reference/datatransfer/rest
My code was to cleanup Cloud Workflows (you need to adapt a little bit this one with the Transfer Service API calls)
#cloud workflow to cleanup
main:
steps:
- initialize:
assign:
- project: "marton-data"
- location: "us-central1"
- getList:
call: WorkflowsList
args:
project: ${project}
location: ${location}
result: items
- loopItems:
call: WorkflowListLoopItems
args:
items: ${items}
result: res
- final:
return: ${res}
WorkflowsList:
params: [project,location]
steps:
- list:
call: googleapis.workflows.v1.projects.locations.workflows.list
args:
parent: ${"projects/"+project+"/locations/"+location}
pageSize: 100
result: listResult
- documentFound:
return: ${listResult.workflows}
WorkflowDelete:
params: [name]
steps:
- delete:
call: googleapis.workflows.v1.projects.locations.workflows.delete
args:
name: ${name}
result: deleteResult
- logStep:
call: sys.log
args:
text: ${"Calling "+name+" \r\n Results returned "+json.encode_to_string(deleteResult)}
- deleteDone:
return: ${deleteResult}
WorkflowListLoopItems:
params: [items]
steps:
- init:
assign:
- i: 0
- result: ""
- check_condition:
switch:
- condition: ${items[i].name=="projects/marton-data/locations/us-central1/workflows/workflow_cleanup"}
next: assign_loop
- condition: ${len(items) > i}
next: iterate
next: exit_loop
- iterate:
steps:
- process_item:
call: WorkflowDelete
args:
name: ${items[i].name}
result: result
- assign_loop:
assign:
- i: ${i+1}
next: check_condition
- exit_loop:
return: ${result}
logMessage:
params: [collection]
steps:
- log:
call: http.post
args:
url: https://logging.googleapis.com/v2/entries:write
auth:
type: OAuth2
body:
entries:
- logName: ${"projects/" + sys.get_env("GOOGLE_CLOUD_PROJECT_ID") + "/logs/workflow_logger"}
resource:
type: "audited_resource"
labels: {}
jsonPayload: ${collection}