Splitting out a field in Big Query - google-bigquery

Question

I have searched around and can not find much on this topic (maybe bad search terms :). I have a table, Protopayload.resource, that gets Apache logging information. As a result the field I am interested in contains multiple values that I need to search against. The field is formatted in a php URL style.
i.e.
/?id=13242134123&ver=12&os_bits=64&os_type=mac&lng=EN
This makes all searches end up with really long regexes to get data. Then join statements to combine data.
Example search to combine mac/win stats
SELECT
t1.date, t1.wincount, COALESCE(t2.maccount, 0) AS maccount
FROM (
SELECT
DATE(metadata.timestamp) AS date,
INTEGER(COUNT(protoPayload.resource)) AS wincount
FROM (TABLE_DATE_RANGE(tablename, DATE_ADD(CURRENT_TIMESTAMP(), -30, 'DAY'), CURRENT_TIMESTAMP() ))
WHERE
(REGEXP_MATCH(protoPayload.resource, r'ver=[11,12'))
AND protoPayload.resource CONTAINS 'os=win' GROUP BY date ) t1
LEFT JOIN (
SELECT
DATE(metadata.timestamp) AS date,
INTEGER(COUNT(protoPayload.resource)) AS maccount
FROM (TABLE_DATE_RANGE(tablename, DATE_ADD(CURRENT_TIMESTAMP(), -30, 'DAY'), CURRENT_TIMESTAMP() ))
WHERE
(REGEXP_MATCH(protoPayload.resource, r'cv=[p,m][17,16,15,14]'))
AND protoPayload.resource CONTAINS 'os=mac' GROUP BY date ) t2
ON
t1.date = t2.date
ORDER BY t1.date
What I was thinking was to use similar regex searches. Create a new table. Then save the data to a new table with relation fields. Then fix future logging so it logs to the table correctly.
My questions are this valid solution, or is there a much easier way to accomplish this in Google BigQuery? Is there a better way to transform the data?
Thanks again for any input!

Answer 1

You can use a SQL function to parse the key-value pairs into an array, which will generally be faster than using JavaScript. For example,
#standardSQL
CREATE TEMPORARY FUNCTION ParseKeys(queryString STRING)
RETURNS ARRAY<STRUCT<key STRING, value STRING>> AS (
(SELECT
ARRAY_AGG(STRUCT(
entry[OFFSET(0)] AS key,
entry[OFFSET(1)] AS value))
FROM (
SELECT SPLIT(pairString, '=') AS entry
FROM UNNEST(SPLIT(REGEXP_EXTRACT(queryString, r'/\?(.*)'), '&')) AS pairString)
)
);
SELECT ParseKeys('/?foo=bar&baz=2');
Now you can build on this with a function that pivots the keys into struct fields:
#standardSQL
CREATE TEMP FUNCTION GetAttributes(queryString STRING) AS (
(SELECT AS STRUCT
MAX(IF(key = 'id', CAST(value AS INT64), NULL)) AS id,
MAX(IF(key = 'ver', CAST(value AS INT64), NULL)) AS ver,
MAX(IF(key = 'os_bits', CAST(value AS INT64), NULL)) AS os_bits,
MAX(IF(key = 'os_type', value, NULL)) AS os_type,
MAX(IF(key = 'lng', value, NULL)) AS lng
FROM UNNEST(ParseKeys(queryString)))
);
Putting everything together, you can try out the GetAttributes function with some sample input:
#standardSQL
CREATE TEMPORARY FUNCTION ParseKeys(queryString STRING)
RETURNS ARRAY<STRUCT<key STRING, value STRING>> AS (
(SELECT
ARRAY_AGG(STRUCT(
entry[OFFSET(0)] AS key,
entry[OFFSET(1)] AS value))
FROM (
SELECT SPLIT(pairString, '=') AS entry
FROM UNNEST(SPLIT(REGEXP_EXTRACT(queryString, r'/\?(.*)'), '&')) AS pairString)
)
);
CREATE TEMP FUNCTION GetAttributes(queryString STRING) AS (
(SELECT AS STRUCT
MAX(IF(key = 'id', CAST(value AS INT64), NULL)) AS id,
MAX(IF(key = 'ver', CAST(value AS INT64), NULL)) AS ver,
MAX(IF(key = 'os_bits', CAST(value AS INT64), NULL)) AS os_bits,
MAX(IF(key = 'os_type', value, NULL)) AS os_type,
MAX(IF(key = 'lng', value, NULL)) AS lng
FROM UNNEST(ParseKeys(queryString)))
);
SELECT url, GetAttributes(url).*
FROM UNNEST(['/?id=13242134123&ver=12&os_bits=64&os_type=mac&lng=EN',
'/?id=2343645745&ver=15&os_bits=32&os_type=linux&lng=FR']) AS url;

Answer 2

You can always use Javascript UDFs for maximum flexibility. They will be slower than a pure SQL solution, but you'll be able to code around its limitations.
For example:
#standardSQL
CREATE TEMPORARY FUNCTION parse(query STRING)
RETURNS STRUCT<id STRING, ver STRING, os_bits STRING, os_type STRING, lng STRING>
LANGUAGE js AS """
function parseQueryString(query) {
// http://codereview.stackexchange.com/a/10396
var map = {};
query.replace(/([^&=]+)=?([^&]*)(?:&+|$)/g, function(match, key, value) {
(map[key] = map[key] || []).push(value);
});
return map;
}
return parseQueryString(query)
""";
WITH urls AS
(SELECT 'id=13242134123&ver=12&os_bits=64&os_type=mac&lng=EN' query
UNION ALL
SELECT 'id=13242134124&ver=12&os_bits=64&os_type=mac&lng=EN1&lng=EN2' query
)
SELECT query, parse(query) as parsed
FROM urls;.

Answer 3

I see few issues in the query in your question
1. looks like regexp is not correct and will not capture what you expect
2. query is heavily over-engineered and can be quite simplified
Below is to address above points
SELECT
DATE(metadata.timestamp) AS date,
SUM(REGEXP_MATCH(protoPayload.resource, r'ver=(11|12)\b')
AND protoPayload.resource CONTAINS 'os_type=win'
) AS wincount,
SUM(REGEXP_MATCH(protoPayload.resource, r'cv=(p|m)(17|16|15|14)\b')
AND protoPayload.resource CONTAINS 'os_type=mac'
) AS maccount
FROM (TABLE_DATE_RANGE(tablename, DATE_ADD(CURRENT_TIMESTAMP(), -30, 'DAY'),
CURRENT_TIMESTAMP() ))
GROUP BY date
Please note: you query in question is written with BigQuery Legacy SQL, so I keep my answer in same dialect