Custom Delimiters in CSV Data while exporting data from BigQuery to GCS bucket? - google-bigquery

Question

Background:
I have a GA Premium account. I currently have the following process setup:
The raw data from the GA account flows into BigQuery.
Query the Bigquery tables.
Export query results to a GCS bucket. I export it in a CSV and gzipped format.
Export the CSV gzipped data from the GCS bucket to my Hadoop cluster HDFS.
Generate hive tables from the data on the cluster by using the comma as the field delimiter.
I run Steps 1-3 programmatically using the BigQuery REST API.
Problem:
My data contains embedded commas and newlines within quotes in some of the fields. When I generate my hive tables, the embedded commas and newlines are causing shifts in my field values for a row or are causing nulls in the records in the hive table.
I want to clean the data by either removing these embedded commas and newlines or by replacing them with custom delimiters within the quotes.
However, the catch is that I would like to do this data cleaning at Step 3 - while exporting to GCS. I looked into the possible query parameters that I can use achieving this but did not find any. The possible parameters that you can use to populate the configuration.extract object are listed at: https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.extract
Here is snippet of the code that does the data export from Bigquery tables to GCS bucket.
query_request = bigquery_service.jobs()
DATASET_NAME = "#######";
PROJECT_ID = '#####';
DATASET_ID = 'DestinationTables';
DESTINATION_PATH = 'gs://bucketname/foldername/'
query_data = {
'projectId': '#####',
'configuration': {
'extract': {
'sourceTable': {
'projectId': PROJECT_ID,
'datasetId': DATASET_ID,
'tableId': #####,
},
'destinationUris': [DESTINATION_PATH + my-files +'-*.gz'],
'destinationFormat': 'CSV',
'printHeader': 'false',
'compression': 'GZIP'
}
}
}
query_response = query_request.insert(projectId=constants.PROJECT_NUMBER,
body=query_data).execute()
Thanks in advance.

Answer 1

EDIT: looks like I misunderstood your question. You wanted to modify the values to not include commas and new lines. I thought your issue was only commas and that the fix would be to just not use commas as deliminators.
To be clear, there is no way to make the modification while exporting. You will need to run another query to produce a new table.
Example:
SELECT x, y, z,
REGEXP_REPLACE(
REGEXP_REPLACE(
REGEXP_REPLACE(bad_data, '%', '%45'),
'\n', '%20'
)
',', '%54'
) FROM ds.tbl
This will encode the bad_data field in a query string compatible format. Remember to run this query with large results enabled if necessary.
A java.net.URLDecoder or something similar should be able to decode if you don't want to do it by hand later.
You can set the field delimiter of the export object.
https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.extract.fieldDelimiter
query_request = bigquery_service.jobs()
DATASET_NAME = "#######";
PROJECT_ID = '#####';
DATASET_ID = 'DestinationTables';
DESTINATION_PATH = 'gs://bucketname/foldername/'
query_data = {
'projectId': '#####',
'configuration': {
'extract': {
'sourceTable': {
'projectId': PROJECT_ID,
'datasetId': DATASET_ID,
'tableId': #####,
},
'destinationUris': [DESTINATION_PATH + my-files +'-*.gz'],
'destinationFormat': 'CSV',
'fieldDelimiter': '~',
'printHeader': 'false',
'compression': 'GZIP'
}
}
}
query_response = query_request.insert(projectId=constants.PROJECT_NUMBER,
body=query_data).execute()

Answer 2

With Python:
from google.cloud import bigquery
client = bigquery.Client()
bucket_name = 'my-bucket'
project = 'bigquery-public-data'
dataset_id = 'samples'
table_id = 'shakespeare'
destination_uri = 'gs://{}/{}'.format(bucket_name, 'shakespeare.csv')
dataset_ref = client.dataset(dataset_id, project=project)
table_ref = dataset_ref.table(table_id)
job_config = bigquery.ExtractJobConfig()
job_config.field_delimiter = ';'
extract_job = client.extract_table(
table_ref,
destination_uri,
# Location must match that of the source table.
location='US') # API request
extract_job.result()