GCSToBigQueryOperator not working in composer-2.1.0-airflow-2.3.4 - google-bigquery

Question

After a recent upgrade to composer-2.1.0-airflow-2.3.4 the GCSToBigQueryOperator is no longer able to find data in buckets to upload to BigQuery.
All other aspects of the DAGs still work.
The usage is as follows
gcs_to_bq = GCSToBigQueryOperator(
task_id = f"transfer_{data_type}_to_bq_task",
bucket = os.environ["GCS_BUCKET"],
source_objects = file_names,
destination_project_dataset_table = os.environ["GCP_PROJECT"] + f".creditsafe.{data_type}",
schema_object = f"dags/schema/creditsafe/{data_type}.json",
source_format = "CSV",
field_delimiter = '|',
quote_character = "",
max_bad_records = 0,
create_disposition = "CREATE_IF_NEEDED",
ignore_unknown_values = True,
allow_quoted_newlines = True,
allow_jagged_rows = True,
write_disposition = "WRITE_TRUNCATE",
gcp_conn_id = 'google_cloud_default',
skip_leading_rows = 1,
dag = dag
)
The error from the API is
google.api_core.exceptions.NotFound: 404 GET
{ "error": { "code": 400, "message": "Unknown output format: media:", "errors": [ { "message": "Unknown output format: media:", "domain": "global", "reason": "invalidAltValue", "locationType": "parameter", "location": "alt" } ] } }
The error delivered by Cloud Composer is
google.api_core.exceptions.NotFound: 404 GET https://storage.googleapis.com/download/storage/v1/b/[BUCKET_HIDDEN]/o/data%2Fcreditsafe%2FCD01%2Ftxt%2F%2A.txt?alt=media: No such object: [BUCKET_HIDDEN]/data/creditsafe/CD01/txt/*.txt: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)
I can't see the cause of the error. The reference to the GCS location has not changed and appears correct while the gcp_conn_id appears sufficient for all other tasks. I'm at a loss.

Answer 1

A fix for this above issue has now been made
https://github.com/apache/airflow/pull/28444
It is unclear how long it will take for this to be integrated into the Cloud Composer libraries.

Answer 2

GCSToBigQueryOperator does not support wildcard *.csv. For your requirement, you can try the below steps:
You can attach to a pod in the composer environment by running the below commands :
gcloud container clusters get-credentials --region __GCP_REGION__ __GKE_CLUSTER_NAME__
kubectl get pods -n [Namespace]
kubectl exec -it [Worker] -n [Namespace] -- bash
You can run the below command to identify the google provider package,
pip list | grep -i goo | grep provider
If the output of the above command is a different version from 8.3.0 then change the version to apache-airflow-providers-google ==8.3.0.

Answer 3

The release 8.5.0 of apache-airflow-providers-google that cames with airflow (>2.3.4 and <2.5.1) introduced several critical regressions nottably:
GCSToBigquery operator is broken as it ignores its options https://github.com/apache/airflow/pull/27961
This means that all custom settings specified in the operator (delimiter, formatting, null values, wildcards on files) are no longer sent to BigQuery leading to unexpected results.
Until Google releases a composer version based on airflow 2.5.1 the workaround is to upgrade the apache-airflow-providers-google library (or to use a composer version based on apache <=2.2.5).
No need to connect via gcloud/kubectl to change the apache-airflow-providers-google version, you can change it directly in the Composer UI, via PyPi Packages page (or via the terraform provider).
I can confirm that on the latest (today) composer : composer-1.20.4-airflow-2.4.3 configuring the apache-airflow-providers-google ==8.8.0 (latest) solves those issues for me.
But as mentioned previously this is only an workaround and your mileage might vary...
configuring custom PyPI packages