PostgreSQL - Get all subgroups from text array - sql

Question

I have a table with Job Titles in it
e.g.
Need a Barista on the Weekend
Need a Barista, 24$ an hour
Needed on the weekend, baby sitter, 24$ an hour
I am trying to get a count of unique phrases
e.g.
- 2 x Need a
- 2 x Need a Barista
- 2 x on the Weekend
- 2 x on the
- 2 x 24$ an hour
I have run created a table to turn my text into an array of words
CREATE TABLE IF NOT EXISTS job_words (
source VARCHAR,
title VARCHAR,
words VARCHAR[]
)
I have split my titles and inserted as words into this table
insert into job_words
select 'job-title', raw_title, string_to_array(raw_title, ' ') from jobs
The longest sentences have 49 words in them
I would like to find any phrase that is between 2 and 10 words long
Happy to use another table to write into or just a direct query if it is possible
Sample Query to Get some Sample Data
select cardinality(words) no_of_words, words, title from job_words
where cardinality(words) > 4 and cardinality(words) < 10 and title ilike 'need a%'
order by title limit 100
Sample Data
8;"{Need,a,baby,sitter,for,an,amazing,girl}"
7;"{Need,a,baby,sitter,for,casual,sitting}"
8;"{Need,a,babysitter,for,our,19,months,old}"
9;"{Need,a,babysitter,for,our,4,year,old,son}"
9;"{Need,a,babysitter,for,our,little,reyon,19,months}"
7;"{Need,a,babysitter,-,look,no,further}"
5;"{Need,a,babysitter,or,tutor?}"
9;"{Need,a,baby,sitter,tonight,kids,are,already,sleeping}"
6;"{Need,a,Baker,now?,I'm,available!}"
6;"{Need,a,barista,all,rounder,ASAP}"
8;"{NEED,A,BARISTA???,full,time,or,part,time.}"
5;"{Need,a,brick,labourer,urgently}"
9;"{Need,a,care,giver,for,a,Month,old,baby}"
7;"{Need,a,Carer,-,After,School,hours}"
7;"{Need,a,Carpenter,-,build,a,cubby}"
5;"{Need,a,Carwash,staff,asap}"
5;"{Need,a,catering,assistant,job}"
9;"{Need,a,change,from,customer,service?,Look,no,further!}"
5;"{Need,a,change,of,scenery?}"
6;"{NEED,A,CLEANER,-,asap,start}"
6;"{Need,a,cleaner,for,daily,work}"
6;"{Need,a,cleaner,for,daily,work}"
9;"{Need,a,Cleaner,for,hotel,in,Belmont,near,Geelong}"
9;"{Need,a,Cleaner,for,hotel,in,Fyansford,near,Geelong}"
9;"{Need,a,Cleaner,for,hotel,in,Queenscliff,near,Geelong}"
5;"{Need,a,cleaner,for,mcdownal}"
7;"{Need,a,cleaner,for,tomorrow,pay,cash}"
7;"{Need,a,cleaner,for,tomorrow,pay,cash}"
5;"{Need,a,cleaner,in,Brisbane}"
6;"{Need,a,cleaner,in,Roxburghpark,Area}"
7;"{Need,a,cleaner,on,a,weekly,basis}"
7;"{Need,a,cleaner,on,Sunday,18th,June}"
9;"{Need,a,cleaning,team,for,your,building,or,office?}"
8;"{Need,a,concreter,to,start,full,time,/paving}"
6;"{Need,a,contract,climber,on,Tuesday}"
7;"{Need,a,cook,for,Road,Trip,Film}"
6;"{Need,a,delivery,driver,in,kew}"
8;"{Need,a,dishwasher,-,Wetherill,Park,6,days}"
7;"{Need,admin,done,for,hair,salon,asap}"
7;"{Need,admin,done,for,hair,salon,asap}"
7;"{Need,admin,done,for,hair,salon,asap}"
6;"{Need,a,driver,at,8:00,tonight}"
8;"{Need,a,driver,for,my,4.5,tomne,truck}"
7;"{Need,a,driver,in,a,Korean,restaurant}"
6;"{NEED,A,EXPERIENCED,CAR,WASH,STUFF}"
8;"{Need,a,flexible,babysitter,to,suit,shift,work}"
7;"{Need,a,fridge,picked,up,tommorow,Saturday}"
7;"{Need,after,school,care,with,pick,up}"
5;"{need,a,fulltime,female,cleaner}"
5;"{Need,a,full,time,job}"
8;"{Need,a,full,time,nanny,at,Baulkham,Hills}"
6;"{Need,a,"fun,","reliable,",interactive,babysitter}"
9;"{Need,a,gardener,/,labourer,tomorrow,for,5,hours}"
6;"{Need,a,gardener,or,labourer,tomorrow}"
6;"{Need,a,girl,for,sharing,room}"
8;"{Need,a,girl,or,boy,for,cleaning,job}"
6;"{Need,a,good,Barista,in,putney}"
8;"{Need,a,good,painter,for,the,next,month}"
7;"{Need,a,gyprock,setter,for,monday,23/01/17}"
9;"{Need,a,handy,person,in,our,new,work,shop}"
8;"{Need,a,helper,for,a,house,removals,truck}"
5;"{Need,a,house,Cleaner,?}"
6;"{Need,a,house,cleaner?,Call,now}"
6;"{Need,a,house,cleaner?,CALL,NOW!}"
8;"{Need,a,house,cleaner,for,this,afternoon,$30p/h}"
5;"{Need,a,housekeeper,tomorrow,morning}"
9;"{Need,a,HR,driver,for,one,day,a,week}"
8;"{need,a,invester,for,the,new,restaurant,Urgently}"
7;"{Need,a,job,asap,will,start,tomorrow}"
9;"{Need,a,job,?,Backpackers,wanted,+,FREE,ACCOMODATION}"
5;"{Need,a,job,for,weekend.}"
8;"{need,a,job,in,day,time,and,weekends}"
7;"{Need,a,job,of,cleaning,or,handkitchen}"
5;"{NEED,A,JOB?!,Start,immediately!!}"
5;"{need,a,job,(student,here)}"
6;"{Need,a,job,to,start,asap}"
9;"{Need,a,kitchen,hand,for,Indian,take,away,shop}"
8;"{Need,a,labourer.,Easy,work.,Monday,or,Tues}"
6;"{need,a,labourer,for,7,weeks}"
5;"{Need,a,labourer,for,today}"
5;"{Need,a,labourer,next,week}"
9;"{Need,a,last,minute,barista,or,chef?!,Staff,cancelled?!}"
9;"{Need,a,live,in,nanny,for,our,2,sons}"
8;"{Need,a,local,Electrician?,Look,no,further,:)}"
8;"{Need,a,male,cleaner,for,a,busy,restaurant}"
6;"{Need,a,man,and,a,ute!!!}"
9;"{NEED,A,MAN,AND,UTE,MONDAY,26th,AFTER,5PM}"
9;"{NEED,A,MANSPOWER,TO,HELP,US,IN,OUR,MOVING}"
8;"{Need,an,after-school,nanny,for,month,of,October}"
5;"{Need,a,Nanny,/,Babysitter}"
6;"{Need,a,nanny,for,2,kids}"
9;"{Need,a,nanny,for,3,days,after,school,care}"
7;"{Need,a,Nanny,for,7,year,old}"
9;"{Need,a,nanny,for,a,few,days,a,week}"
9;"{Need,a,nanny,for,after,school,pick,and,care}"
8;"{Need,a,nanny,for,immediate,start,on,Thursdays}"
9;"{Need,a,nanny,for,my,3,year,old,daughter}"
9;"{Need,a,nanny,for,my,3,year,old,daughter}"
6;"{Need,a,nanny,for,one,day.}"
7;"{Need,a,nanny,for,"Rydges,",Campbelltown,5.45-10.30pm}"

Answer 1

I got this. Is this the result you expected?
phrase_part count
------------------------------------------------------------------------- -----
{"Need","a"} 32
{"Need","a","cleaner"} 9
{"a","cleaner"} 9
{"cleaner","for"} 5
{"Need","a","babysitter"} 5
{"a","babysitter"} 5
{"a","cleaner","for"} 5
{"Need","a","cleaner","for"} 5
{"for","hotel","in"} 3
{"near","Geelong"} 3
{"hotel","in"} 3
{"Cleaner","for","hotel"} 3
{"for","hotel"} 3
{"babysitter","for","our"} 3
{"for","our"} 3
{"babysitter","for"} 3
{"a","Cleaner","for"} 3
{"Need","a","babysitter","for"} 3
...
{"NEED","A"} 2
{"a","cleaner","for","daily"} 2
{"Need","a","change"} 2
{"daily","work"} 2
{"pay","cash"} 2
{"a","cleaner","in"} 2
{"Need","a","cleaner","for","tomorrow","pay","cash"} 2
{"for","casual","sitting"} 1
{"concreter","to","start","full"} 1
{"a","change","from","customer","service?","Look"} 1
{"a","contract","climber"} 1
{"Roxburghpark","Area"} 1
{"Need","a","Carwash"} 1
...
If this is your expected result here's the query for it. But I am not sure if you should do this with a huge data set!
I began with the plain phrases instead of your sample data with the arrays. Additionally I added an id column for each phrase:
WITH phrases as (
SELECT
*,
row_number() over (partition by id) nth_word -- B
FROM (
SELECT
id,
unnest(string_to_array(phrase, ' ')) as word -- A
FROM testdata.phrases
)s
)
SELECT
phrase_part,
count(phrase_part) FILTER (WHERE cardinality(phrase_part) >= 2) -- E
FROM (
SELECT
*,
array_agg(b.word) over (partition by a.id, a.nth_word order by a.id, a.nth_word, b.nth_word) --D
as phrase_part
FROM phrases a
JOIN phrases b -- C
ON (a.id = b.id AND a.nth_word <= b.nth_word)
) s
GROUP BY phrase_part
ORDER BY COUNT DESC
A: formatting the plain phrases into single word arrays and expand the table to one word per line
B: adding a word counter to identify the nth word of the phrase with a window function
C: cross join the phrases with themself; better speaking: join on word with each following of the same phrase
D: this window function aggregates the phrase words. It creates a result like
id word nth_word id word nth_word phrase_part
-- ------------ -------- -- ------------ -------- -------------------------------------------------------------------------
1 Need 1 1 Need 1 {"Need"}
1 Need 1 1 a 2 {"Need","a"}
1 Need 1 1 baby 3 {"Need","a","baby"}
1 Need 1 1 sitter 4 {"Need","a","baby","sitter"}
1 Need 1 1 for 5 {"Need","a","baby","sitter","for"}
1 Need 1 1 casual 6 {"Need","a","baby","sitter","for","casual"}
1 Need 1 1 sitting 7 {"Need","a","baby","sitter","for","casual","sitting"}
1 a 2 1 a 2 {"a"}
1 a 2 1 baby 3 {"a","baby"}
1 a 2 1 sitter 4 {"a","baby","sitter"}
E: grouping by phrases and counting the elements. The filter clause allows you to count different cardinalities.