Working with url-encoded values in BigQuery - google-bigquery

Question

I work with gzipped log files which contain url-encoded columns. (a space character is encoded as "%20", etc).
My plan was to import these files directly from Google Cloud Storage into BigQuery.
I did not find any option in the Load config to automatically decode values during the import.
I guess you wouldn't advice using a series of REGEXP_REPLACE in all my queries.
Any idea which would avoid parsing all the logs and escape all these characters before importing them to BigQuery (which would be dangerous if one of them is the separator) ?

Answer 1

The accepted answer if for Legacy SQL.
For Standard SQL:
#standardSQL
CREATE TEMPORARY FUNCTION DECODE_URI_COMPONENT(path STRING)
RETURNS STRING
LANGUAGE js AS """
if (path == null) return null;
try {
return decodeURIComponent(path);
} catch (e) {
return path;
}
""";
WITH source AS (SELECT "/work.json?myfield=R%C3%A9gions%2CSport" AS path)
SELECT DECODE_URI_COMPONENT(REGEXP_EXTRACT(path, r"[?&]myfield=([^&]+)")) AS myfield FROM source
This returns:
myfield
---------------
Régions,Sport

Answer 2

Most likely you already ended up with something like below :o)
SELECT url FROM
js(
(SELECT url FROM
(SELECT 'http://example.com/query?q=my%20query%20string' AS url),
(SELECT 'http://example.com/query?q=your%20query%20string' AS url),
(SELECT 'http://example.com/query?q=his%20query%20string' AS url)
),
// Input columns.
url,
// Output schema.
"[
{name: 'url', type:'string'}]",
// The function.
"function(r, emit) {
var url = decodeURI(r.url);
emit({
url: url
});
}"
)
https://cloud.google.com/bigquery/user-defined-functions