Finding the journey made by users in Google BigQuery - sql

Question

I'm looking to find the journey made by users on a particular website. The schema of my dataset is the same as Google Merchandise Store, which can be found here: https://support.google.com/analytics/answer/3437719?hl=en
From the Google BigQuery cookbook, I've implemented and modified the SQL code provided to get the sequence of hits made by every customer.
SELECT
fullVisitorId AS id,
visitId AS visitid,
visitNumber AS visitnumber,
h.hitNumber AS hitNumber,
CASE
WHEN h.eventInfo.eventAction = "Lead" THEN "Lead"
WHEN h.eventInfo.eventAction = "Homepage" THEN "Homepage"
WHEN h.eventInfo.eventAction = "Search" THEN "Search"
WHEN h.eventInfo.eventAction = "High Intent Use" THEN "High Intent Use"
WHEN h.eventInfo.eventAction = "Listing Page" THEN "Listing Page"
END AS journey
FROM
`dataset`,
UNNEST(hits) AS h
WHERE
h.type="PAGE"
OR h.type="EVENT"
ORDER BY
fullVisitorId,
visitId,
visitNumber,
hitNumber
A snippet of the result I got is as follows:
fullVisitorId visitId visitNumber hitnumber journey
001 1001 1 1 Homepage
001 1001 1 2 Search
001 1001 1 3 null
001 1001 1 4 Search
001 1001 1 5 Listing Page
001 1001 1 6 Lead
001 1001 1 2 Search
001 1001 1 7 Lead
002 1002 1 1 Search
...
What I need is to get another column which shows the journey taken by each visitor before the first "Lead", while ignoring the duplicates (for eg if the visitor searches for 5 pages back-to-back, the journey should only show "Search" once)
ie. for visitor 001 on visit 1001, the column will show:
Homepage -> Search -> Listing Page -> Lead
I hope the question is clear. Appreciate any help given! :)

Answer 1

Below is for BigQuery Standard SQL and applies extra logic to your existing/current query
#standardSQL
SELECT
fullVisitorId, visitId,
STRING_AGG(journey, ' -> ' ORDER BY visitNumber, hitnumber) journey_path
FROM (
SELECT
fullVisitorId, visitId,
MIN(visitNumber) visitNumber, MIN(hitnumber) hitnumber, journey
FROM (
SELECT *, COUNTIF(journey = 'Lead') OVER(win) grp
FROM `your_current_query`
WINDOW win AS (
PARTITION BY fullVisitorId, visitId
ORDER BY visitNumber, hitnumber
ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING
)
)
WHERE grp = 0
GROUP BY fullVisitorId, visitId, journey
)
GROUP BY fullVisitorId, visitId
so you can just use your existing query as below
#standardSQL
WITH `your_current_query` AS (
SELECT
fullVisitorId AS id,
visitId AS visitid,
visitNumber AS visitnumber,
h.hitNumber AS hitNumber,
CASE
WHEN h.eventInfo.eventAction = "Lead" THEN "Lead"
WHEN h.eventInfo.eventAction = "Homepage" THEN "Homepage"
WHEN h.eventInfo.eventAction = "Search" THEN "Search"
WHEN h.eventInfo.eventAction = "High Intent Use" THEN "High Intent Use"
WHEN h.eventInfo.eventAction = "Listing Page" THEN "Listing Page"
END AS journey
FROM
`dataset`,
UNNEST(hits) AS h
WHERE
h.type="PAGE"
OR h.type="EVENT"
)
SELECT
fullVisitorId, visitId,
STRING_AGG(journey, ' -> ' ORDER BY visitNumber, hitnumber) journey_path
FROM (
SELECT
fullVisitorId, visitId,
MIN(visitNumber) visitNumber, MIN(hitnumber) hitnumber, journey
FROM (
SELECT *, COUNTIF(journey = 'Lead') OVER(win) grp
FROM `your_current_query`
WINDOW win AS (
PARTITION BY fullVisitorId, visitId
ORDER BY visitNumber, hitnumber
ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING
)
)
WHERE grp = 0
GROUP BY fullVisitorId, visitId, journey
)
GROUP BY fullVisitorId, visitId
--- ORDER BY fullVisitorId, visitId
and if to follow your result example - above should produce below result
Row fullVisitorId visitId journey_path
1 001 1001 Homepage -> Search -> Listing Page -> Lead
2 002 1002 Search

Answer 2

I'd suggest using STRING_AGG to make a string of the journey steps, adding DISTINCT into your selection will only show individual journey steps once per user.
Something like:
STRING_AGG(DISTINCT(journey), '->') as propensity_banding_subset
You could then use some regex to clip off after the first 'lead', unless somebody can suggest a better method to do this in the original string aggregation?

Answer 3

I took Mikhails great approach and brought it to a more scalable version for those who have really large amounts of data. The idea is the same, but applied to a subquery on the hits array.
SELECT
fullVisitorId AS id,
visitId AS visitid,
visitNumber AS visitnumber,
ARRAY(
(SELECT AS STRUCT *
FROM
(SELECT AS STRUCT
hitNumber,
page.pagePath, -- pagePath instead of CASE-WHEN with events
count(page.pagePath) over (win) elNumber
FROM t.hits
WHERE type IN ('PAGE', 'EVENT')
WINDOW win AS (
PARTITION BY page.pagePath
ORDER BY hitnumber
ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING
)
ORDER BY hitNumber)
WHERE elNumber=0
-- instead of 'Lead' I used '/signin.html'
AND hitNumber < (SELECT MIN(hitNumber) FROM t.hits WHERE page.pagePath='/signin.html')
)
) AS journey
FROM
`bigquery-public-data.google_analytics_sample.ga_sessions_20170801` t
limit 1000
I used the actual sample data and couldn't find the events from the example there so I simply used page paths. But it should be easily adoptable.
Also this one returns nested data, not a flat table, which again saves space when saving the result as a table and is faster when performing queries on it.
There is also no grouping involved - everything happens within the subquery only on the array which allows very fast processing due to parallization.