sql substring contains - sql

Question

I have a datasetA with a long narrative field. This field is called "narrative."
I have datasetB full of animal terms, such as "dog", "cat", "mouse". This field is called "animals."
I would like to flag any instance where the animal names are found in the narrative of datasetA, and to create a new field in datasetA, called "animal_found" which pulls that name.
For example, if the word "dog" is found in a narrative, the animal_found field for that record will populate "dog"
if the word "dog" and "cat" is found, the animal_found field will show "dog,cat"
Any thought on how to code this in SQL?

Answer 1

If you are using SQL Server, there is a way with Dynamic SQL but it's not very elegant nor performant.
DECLARE #Animal nvarchar(100)
DECLARE cur CURSOR LOCAL FORWARD_ONLY FOR
SELECT Animal FROM datasetB
OPEN cur
FETCH NEXT FROM cur INTO #Animal
WHILE ##FETCH_STATUS = 0
BEGIN
DECLARE #Query NVARCHAR(MAX)
SELECT #Query = 'SELECT Columns FROM datasetA where narrative like ''%' + #Animal + '%'''
exec sp_executeSql #Query
FETCH NEXT FROM cur INTO #Animal
END
CLOSE cur
DEALLOCATE cur
The way to do it would probably be to create a temp table or something like that. Then insert the results into your temp table and format it the way you want to. But as I said, cursors are not really performant. But it works

Answer 2

Not SQL, but within a data step this can be done relatively easily.
load lookup data into a temporary array
Loop through list and search text for the data
Concatenate results as you loop
NOTE: This does not handle extra 's' at the end of words, so you may want to consider how you'll handle frog vs frogs since those are technically not the same word. You cannot just switch to find because of partial matches in other words but you could duplicate the loop twice or modify the FIND to check for both at the same time. I'll leave that to you to solve.
*fake text data;
data statements;
infile cards;
input narrative $100.;
cards;
This is some random text with words that are weirhd such as cat, dog frogs, and any other weird names
This is a notehr rnaodm text with word ssuch as bird and cat
This has nothing in it
This is another phrages with elephants
;
run;
*fake words;
data words;
input word $20.;
cards;
cat
dog
frog
bird
elephant
;;;;
run;
*lookup;
data want;
*loads data set M into temporary array M;
array W(5) $20. _temporary_;
if _n_=1 then do j=1 to 5;
set words;
W(j)=word;
end;
*main data set to work with;
length found $100.;
found = '';
set statements;
do i=1 to dim(w);
x = findw(narrative, W(i), " ,", 'ir');
if x > 0 then found = catx(", ", trim(found), w(i));
*for debugging, comment out/delete as needed;
put "N=" _N_ " | I= " i;
put "Statement = " narrative;
put "Word = " w(i);
put "Found = " x;
put "---------------";
end;
run;

Answer 3

SAS SQL is the wrong tool for aggregating rows into a concatenation result (csv string).
SQL can be used to obtain the found items to be concatenated and data step DOW loop for concatenating:
proc sql;
create view matched_animals as
select narrative, animal from
narratives left join animals on narrative contains trim(animal)
order by narratives, animal;
data want;
length animal_found $2000;
do until (last.narrative);
set matched_animals;
by narrative;
animal_found = catx(',',animal_found,animal);
end;
run;
This will work but may run out of resources depending on the cardinality of the narratives and animals tables and the matching rate.
A data step approach can utilize hash object, countw and scan, or findw. There are two approaches, with way2 the probable best / most typical use case.
* Thanks Reeza for sample data;
data narratives;
infile cards;
input narrative $100.;
cards;
This is some random text with words that are weirhd such as cat, dog frogs, and any other weird names
This is a notehr rnaodm text with word ssuch as bird and cat
This has nothing in it
This is another phrages with elephants
;
run;
data animals;
input animal $20.;
cards;
cat
dog
frog
bird
elephant
;;;;
run;
data want;
set narratives;
length animals_found_way1 animals_found_way2 $2000;
if _n_ = 1 then do;
if 0 then set animals(keep=animal); * prep pdv;
declare hash animals(dataset:'animals');
animals.defineKey('animal');
animals.defineDone();
declare hiter animals_iter('animals');
end;
* check each word of narrative for animal match;
* way 1 use case: narratives shorter than animals list;
do _n_ = 1 to countw(narrative);
token = scan(narrative, _n_);
if animals.find(key:token) = 0 then
animals_found_way1 = catx(',', animals_found_way1, token);
loopcount_way1 = sum (loopcount_way1, 1);
end;
* check each animal for match;
* way 2 use case: animal list shorter than narratives;
do while (animals_iter.next() = 0);
if findw(narrative, trim(animal)) then
animals_found_way2 = catx(',', animals_found_way2, animal);
loopcount_way2 = sum(loopcount_way2, 1);
end;
put;
drop token animal;
run;

Answer 4

If the list of animals is not too long, try this method and see how it performs. I tested this on SQL Server 2017.
with
cte1 as
(select 'I have a dog, a cat and a bunny as my pets' narrative union all
select 'I have a horse, a bunny and a dog as my pets' union all
select 'I have a cat as my pet' union all
select 'I have a dog as my pet' union all
select 'I have nothing')
,cte2 as
(select 'cat' animals union all
select 'dog' union all
select 'parrot' union all
select 'bunny' union all
select 'horse')
select
narrative,
string_agg(case when narrative like concat('%',animals,'%') then animals end,',') animals_found
from cte1 cross join cte2
group by narrative;
Fiddle