Snowflake SQL, how to lookahead until a certain occurrence of a value - sql

Question

Below is a sample of the text that I am working with.
---
info1:
* val: "A"
---
Type:
* answers:
* - !<string>
* val: "B"
* - !<string>
* val: "C"
---
info2:
* val: "D"
---
And I am trying to select the following text:
Type:
* answers:
* - !<string>
* val: "B"
* - !<string>
* val: "C"
I was trying to use a look ahead, but haven't had much success.
REGEXP_SUBSTR(col, 'Type:(.*---)')
Here I am trying to look up until the next occurrence of '---', but I think I misunderstanding how it works.

Answer 1

REGEXP_SUBSTR is rather limited in snowflake, in native SQL, so when you tell is to match multiple lines and match newlines via REGEXP_SUBSTR(t, '(Type:.*)---',1,1,'mes',1) the regexp is greddy thus:
SELECT '---
info1:
* val: "A"
---
Type:
* answers:
* - !<string>
* val: "B"
* - !<string>
* val: "C"
---
info2:
* val: "D"
---' as t
,REGEXP_SUBSTR(t, '(Type:.*)',1,1,'mes',1) as r1
,REGEXP_SUBSTR(t, '(Type:.*)---',1,1,'mes',1) as r2;
gives you too much data:
Type:
* answers:
* - !<string>
* val: "B"
* - !<string>
* val: "C"
---
info2:
* val: "D"
So one idea is if --- is always a section marker, is to string split on that first and then regex
WITH input as (
select '---
info1:
* val: "A"
---
Type:
* answers:
* - !<string>
* val: "B"
* - !<string>
* val: "C"
---
info2:
* val: "D"
---' as t
)
select t, c.value::string as part, REGEXP_SUBSTR(part, 'Type:.*',1,1,'mes') as r1
from input,
lateral flatten(input=>split(t, '---')) c;
gives
T PART R1
--- info1: * val: "A" --- Type: * answers: * - !<string> * val: "B" * - !<string> * val: "C" --- info2: * val: "D" ---
--- info1: * val: "A" --- Type: * answers: * - !<string> * val: "B" * - !<string> * val: "C" --- info2: * val: "D" --- info1: * val: "A"
--- info1: * val: "A" --- Type: * answers: * - !<string> * val: "B" * - !<string> * val: "C" --- info2: * val: "D" --- Type: * answers: * - !<string> * val: "B" * - !<string> * val: "C" Type: * answers: * - !<string> * val: "B" * - !<string> * val: "C"
--- info1: * val: "A" --- Type: * answers: * - !<string> * val: "B" * - !<string> * val: "C" --- info2: * val: "D" --- info2: * val: "D"
--- info1: * val: "A" --- Type: * answers: * - !<string> * val: "B" * - !<string> * val: "C" --- info2: * val: "D" ---
from which you should be able to progress.
Also if you need really complex regexp, you can make a javascript-UDF and use the javascript regexp engine..

Answer 2

You don't need regexp lookahead to get the string you want, it's just eg.
REGEXP_SUBSTR(col, '(^Type:\\s+(^[*].*$\\s+)*)^---', 1, 1, 'm', 1)
If you need regexps with lookahead, etc, use JavaScript RegExps via a function wrapper, eg.
CREATE OR REPLACE FUNCTION RegExp_Match("STRING" VARCHAR, "REGEXP" VARCHAR)
RETURNS VARIANT LANGUAGE JAVASCRIPT STRICT IMMUTABLE AS
'return STRING.match(REGEXP);';
CREATE OR REPLACE FUNCTION RegExp_Match("STRING" VARCHAR, "RX" VARCHAR, "FLAGS" VARCHAR)
RETURNS VARIANT LANGUAGE JAVASCRIPT STRICT IMMUTABLE AS
'return STRING.match(new RegExp(RX, FLAGS));';
SELECT RegExp_Match('<aA>', '(?<=<)(.)\\1(?=>)', 'i');
-- RegExp with lookback, back reference and lookahead ignoring case
=> [ "aA", "a" ]

Answer 3

Snowflake does not support look-aheads or look-behinds, but it does support group extraction (and nested group extraction). These can be used within REGEXP_REPLACE or REGEXP_SUBSTR. In this case, I prefer REGEXP_SUBSTR as you are looking to extract from, as opposed to replace within. In my example below you will see both.
You have 3 dashes (-) that work as your delimiter, the issue becomes that you have a dash within your data. I would advise replacing the delimiter with something that will NOT exist within your data, I chose a tilde (~).
The code sample below will work.
Notes:
Column C2 operates on the KEEPS_LEADING_SPACE column which does not remove the leading space. In this case, the leading space is the first capture group.
Column C3 operates on the actual data column but assumes that the leading space is one or more of a space, carriage return, newline, or vertical space.
Capture Groups:
(~([^~]+))? - captures something starts with a tilde (~) zero or more times, and within it
([^~]+) - captures anything that is not a tilde (~) one or more times
USE ROLE SYSADMIN;
USE WAREHOUSE PUBLIC_WH;
USE UTIL_DB.PUBLIC;
CREATE OR REPLACE TEMP TABLE REGEXP_TEST
AS
SELECT $1::VARIANT AS C1
FROM VALUES
($$
---
info1:
* val: "A"
---
Type:
* answers:
* - !<string>
* val: "B"
* - !<string>
* val: "C"
---
info2:
* val: "D"
---
$$);
SELECT C1
,REPLACE(C1,'---','~') AS KEEPS_LEADING_SPACE
,REGEXP_SUBSTR(KEEPS_LEADING_SPACE,'(~([^~]+))?',1,4,'is') AS C2
,REGEXP_SUBSTR(REGEXP_REPLACE(C1,'[\s\r\n\v]?-{3}','~'),'(~([^~]+))?',1,3,'is') AS C3
FROM REGEXP_TEST
;
Regex in Snowflake does support negatives but I tend to find them hard to work with if you are looking for more than one character, but in this case we have one character to negate the tilde [^~].