clickhouse approach for word frequency count on textual field - word-count

Question

I have a Clickhouse table, where one of the fields contains a textual description (~300 words).
For example Reviews:
Rev_id Place_id Stars Category Text
1 12 3 Food Nice food but a bad dirty place.
2 31 4 Sport Not bad, they have everything.
3 55 1 Bar Poor place,bad audience.
I'd like to make some word count analysis, such as general word frequency count (how many times each of the words has appeared) or top-K words per Category.
In the example:
word count
bad 3
place 2
...
Is there a way to do it solely in ClickHouse without involving programming languages?

Answer 1

SELECT
arrayJoin(splitByChar(' ', replaceRegexpAll(x, '[.,]', ' '))) AS w,
count()
FROM
(
SELECT 'Nice food but a bad dirty place.' AS x
UNION ALL
SELECT 'Not bad, they have everything.'
UNION ALL
SELECT 'Poor place,bad audience.'
)
GROUP BY w
ORDER BY count() DESC
┌─w──────────┬─count()─┐
│ │ 4 │
│ bad │ 3 │
│ place │ 2 │
│ have │ 1 │
│ Poor │ 1 │
│ food │ 1 │
│ Not │ 1 │
│ they │ 1 │
│ audience │ 1 │
│ Nice │ 1 │
│ but │ 1 │
│ dirty │ 1 │
│ a │ 1 │
│ everything │ 1 │
└────────────┴─────────┘
SELECT CATEGORY, ....
GROUP BY CATEGORY, w

Answer 2

If it applicable in your case I would consider using the alphaTokens as a more efficient one.
SELECT
category,
arrayJoin(arrayFilter(x -> NOT has(['a', 'the', 'but' /*.. exclude stopwords */], x), alphaTokens(text))) token,
count() count
FROM
(
/* test data */
SELECT data.1 AS rev_id, data.2 AS place_id, data.3 AS stars, data.4 AS category, data.5 AS text
FROM
(
SELECT arrayJoin([
(1, 12, 3, 'Food', 'Nice food but a bad dirty place.'),
(4, 12, 3, 'Food', ' the the the the good food ..'),
(2, 31, 4, 'Sport', 'Not bad,,, they have everything.'),
(3, 55, 1, 'Bar', 'Poor place,bad audience..')]) AS data
)
)
GROUP BY category, token
ORDER BY count DESC
LIMIT 5;
/*
┌─category─┬─token────┬─count─┐
│ Food │ food │ 2 │
│ Food │ bad │ 1 │
│ Bar │ audience │ 1 │
│ Food │ Nice │ 1 │
│ Bar │ Poor │ 1 │
└──────────┴──────────┴───────┘
*/
Example of using topK:
SELECT
category,
arrayReduce('topK(3)',
arrayFilter(x -> (NOT has(['a', 'the', 'but' /*.. exclude stopwords */], x)), groupArrayArray(alphaTokens(text)))) AS result
FROM
(
/* test data */
SELECT data.1 AS rev_id, data.2 AS place_id, data.3 AS stars, data.4 AS category, data.5 AS text
FROM
(
SELECT arrayJoin([
(1, 12, 3, 'Food', 'Nice food but a bad dirty place.'),
(4, 12, 3, 'Food', ' the the the the good food ..'),
(2, 31, 4, 'Sport', 'Not bad,,, they have everything.'),
(3, 55, 1, 'Bar', 'Poor place,bad audience..')]) AS data
)
)
GROUP BY category;
/* result
┌─category─┬─result─────────────────┐
│ Bar │ ['Poor','place','bad'] │
│ Food │ ['food','Nice','bad'] │
│ Sport │ ['Not','bad','they'] │
└──────────┴────────────────────────┘
*/
ps: probably make sense to lower all strings/tokens before processing