SAS - Proc SQL or Merge - Trying to optimise an INNER join that includes a string search (index) - indexing

Question

I've a rudimentary SAS skillset, most of which involves "proc sql", so feel free to challenge the fundamental approach of using this.
I'm attempting to match one set of personal details against another set, the first having some ~400k rows and the other 22 million. The complexity is that the 400k rows feature previous names and postcodes as well as current ones (all on the same row), so my approach (code below) was to concatenate all of the surnames together and all of the postcodes together and search for the string from the second table (single name and postcode) within the concatenated strings using the index(source, excerpt) function.
proc sql;
CREATE TABLE R4 AS
SELECT DISTINCT
BS.CUST_ID,
ED.MATCH_ID
FROM T_RECS_WITH_CONCATS BS
INNER JOIN T_RECS_TO_MATCH ED
ON LENGTH(ED.SinglePostcode) > 4
AND index(BS.AllSurnames,ED.SingleSurname) > 0
AND index(BS.AllPostcodes,ED.SinglePostcode) > 0
;
QUIT;
In the above, AllSurnames can contain up to 9 surnames (delimited by |), and AllPostcodes up to 9 concatenated postcodes (again, delimited by |).
The downside of this is of course that it takes forever to run. Is there are more efficient way of doing this, either within a proc sql step or a real data step?

Answer 1

Here is a way using HASH component object
Presume the data sets are named SHORT_MANY and TALL_ONE. Use the data in SHORT_MANY to populate a multidata hash table that can operate as a lookup for values being checked in TALL_ONE.
Using just surname and postal code as the lookup key could result in many false matches.
Example (with numeric surname & postcode)
data SHORT_MANY;
do cust_id = 1 to 400;
array Surnames surname1-surname9;
array Postcodes postcode1-postcode9;
call missing (of surnames(*));
do index = 1 to dim(surnames);
surnames(index) = ceil (100000 * ranuni(123));
postcodes(index) = ceil ( 99999 * ranuni(123));
if ranuni(123) < 0.15 then leave;
end;
output;
end;
run;
data TALL_ONE(keep=match_id surname postcode forcemark);
do match_id = 1 to 22000;
surname = ceil(100000 * ranuni(1234));
postcode = ceil( 99999 * ranuni(1234));
forcemark = .;
if ranuni(123) < 0.15 then do; * randomly ensure some match will occur;
point = ceil(400*ranuni(123));
set SHORT_MANY point=point;
array surnames surname1-surname9;
array postcodes postcode1-postcode9;
do until (surname ne .);
index = ceil(9 * ranuni(123));
surname = surnames(index);
postcode = postcodes(index);
end;
forcemark = point;
end;
output;
end;
stop;
run;
data WHEN_TALL_MEETS_SHORT(keep=cust_id match_id index);
if 0 then set TALL_ONE SHORT_MANY ; * prep pdv (for hash host variables);
if _n_ = 1 then do;
length index 8;
declare hash lookup(multidata: 'yes');
lookup.defineKey('surname', 'postcode');
lookup.defineData('cust_id', 'index');
lookup.defineDone();
do while (not lookup_filled);
SET SHORT_MANY end=lookup_filled;
array Surnames surname1-surname9;
array Postcodes postcode1-postcode9;
do index = 1 to dim(surnames) while (surnames(index) ne .);
surname = surnames(index);
postcode = postcodes(index);
lookup.add();
end;
end;
end;
call missing (surname, postcode, cust_id, index);
set TALL_ONE;
rc = lookup.find(); * grab just first match -- has_next/find_next to retrieve other lookup matches;
run;