Most frequent combination of words/characters SQL Server - sql

Question

I am analysing data in a 'RawDataDescriptions' table with a 'description' field that was open end for users to input.
I'm looking for ways to broadly categorise the descriptions by phrases or a string of characters that frequently show up (including a count of how many times they occur).
I have no specific words or phrases to look for necessarily where i can use a 'like' statement, instead i'm looking for commonalities between the fields.
Whilst looking for this through other questions, i managed to find a query which i adjusted for my own table that gets out the most common word (Pasted below), but of course one word alone provides little -if any -insight into the descriptions.
Is it possible to make a query that would provide a count of phrases and not just single words? if so, what would the main components of it be?
WITH E1(N) AS
(
SELECT 1
FROM (VALUES
(1),(1),(1),(1),(1),(1),(1),(1),(1),(1)
) t(N)
),
E2(N) AS (SELECT 1 FROM E1 a CROSS JOIN E1 b),
E4(N) AS (SELECT 1 FROM E2 a CROSS JOIN E2 b)
SELECT
x.Item,
COUNT(*)
FROM RawDataDescriptions p
CROSS APPLY (
SELECT
ItemNumber = ROW_NUMBER() OVER(ORDER BY l.N1),
Item = LTRIM(RTRIM(SUBSTRING(p.[Description], l.N1, l.L1)))
FROM (
SELECT s.N1,
L1 = ISNULL(NULLIF(CHARINDEX(' ',p.[Description],s.N1),0)-
s.N1,4000)
FROM(
SELECT 1 UNION ALL
SELECT t.N+1
FROM(
SELECT TOP (ISNULL(DATALENGTH(p.[Description])/2,0))
ROW_NUMBER() OVER (ORDER BY (SELECT NULL))
FROM E4
) t(N)
WHERE SUBSTRING(p.[Description] ,t.N,1) = ' '
) s(N1)
) l(N1, L1)
) x
WHERE x.item <> ''
GROUP BY x.Item
ORDER BY COUNT(*) DESC
*Edit - not doable. Alternative desired outcome:
Sample table
Id | Description
---+--------------------------
01 | Customer didn't like it
02 | Person liked it
03 | Person didn't like it
04 | Client didn't like it
05 | person liked it
#Parameter = 3
Desired result :
string | count
-----------------+-------
didn't like it | 3
Person liked it | 2
edit 2** the original question was doable - see answer

Answer 1

Here is one option. I have several concerns, like punctuation, control characters, and especially performance on large tables
Example
Declare #RawDataDescriptions Table ([Id] varchar(50),[Description] varchar(50))
Insert Into #RawDataDescriptions Values
('01','Customer didn''t like it')
,('02','Person liked it')
,('03','Person didn''t like it')
,('04','Client didn''t like it')
,('05','person liked it')
;with cte as (
Select Id
,B.*
From #RawDataDescriptions A
Cross Apply (
Select RetSeq = Row_Number() over (Order By (Select null))
,RetVal = LTrim(RTrim(B.i.value('(./text())[1]', 'varchar(max)')))
From (Select x = Cast('<x>' + replace((Select replace(A.[Description],' ','§§Split§§') as [*] For XML Path('')),'§§Split§§','</x><x>')+'</x>' as xml).query('.')) as A
Cross Apply x.nodes('x') AS B(i)
) B
)
Select Phrase
,Cnt = count(*)
From cte A
Cross Apply (
Select Phrase = stuff((Select ' '+RetVal
From cte
Where ID = A.ID
and RetSeq between A.RetSeq and A.RetSeq+2
Order By RetSeq
For XML Path('')),1,1,'')
) B
Where Phrase like '% % %'
Group By Phrase
Having count(*)>1
Order By 2 Desc
Returns
Phrase Cnt
didn't like it 3
Person liked it 2
UPDATE - TVF - Better Performance
I decided that I may want to turn this into a a Table-Valued Function, and was shocked by the performance gains. For example, I have 130,000 descriptions from FRED (Federal Reserve Economic Data), and I was able to generate a list of common phrases (n words) in 9 seconds.
Usage
Select Phrase = B.RetVal
,Cnt = count(*)
From YourTable A
Cross Apply [dbo].[tvf-Str-Parse-Phrase](A.YourColumn,' ',4) B
Group By B.RetVal
Having count(*)>1
Order By 2 Desc
The TVF if Interested
CREATE FUNCTION [dbo].[tvf-Str-Parse-Phrase] (#String varchar(max),#Delimeter varchar(25),#WordCnt int)
Returns Table
As
Return (
with cte as (
Select RetSeq = Row_Number() over (Order By (Select null))
,RetVal = LTrim(RTrim(B.i.value('(./text())[1]', 'varchar(max)')))
From (Select x = Cast('<x>' + replace((Select replace(#String,#Delimeter,'§§Split§§') as [*] For XML Path('')),'§§Split§§','</x><x>')+'</x>' as xml).query('.')) as A
Cross Apply x.nodes('x') AS B(i)
)
Select RetSeq = Row_Number() over (Order By (Select Null))
,B.RetVal
From cte A
Cross Apply (Select RetVal = stuff((Select ' '+RetVal From cte Where RetSeq between A.RetSeq and A.RetSeq+#WordCnt-1 For XML Path('')),1,1,'') ) B
Where B.RetVal like Replicate('% ',#WordCnt-1)+'%'
);
--Select * from [dbo].[tvf-Str-Parse-Phrase]('This is some text that I want parsed',' ',4)

Answer 2

You can enable Microsoft Full Text Index on the table & perform these kind of finding frequent words and characters analysis on the table.
sys.dm_fts_index_keywords_by_document
sys.dm_fts_index_keywords_by_property
sys.dm_fts_index_keywords_by_document