How are non-capturing groups, i.e., (?:), used in regular expressions and what are they good for?
Let me try to explain this with an example.
Consider the following text:
http://stackoverflow.com/
https://stackoverflow.com/questions/tagged/regex
Now, if I apply the regex below over it...
(https?|ftp)://([^/\r\n]+)(/[^\r\n]*)?
... I would get the following result:
Match "http://stackoverflow.com/"
Group 1: "http"
Group 2: "stackoverflow.com"
Group 3: "/"
Match "https://stackoverflow.com/questions/tagged/regex"
Group 1: "https"
Group 2: "stackoverflow.com"
Group 3: "/questions/tagged/regex"
But I don't care about the protocol -- I just want the host and path of the URL. So, I change the regex to include the non-capturing group (?:).
(?:https?|ftp)://([^/\r\n]+)(/[^\r\n]*)?
Now, my result looks like this:
Match "http://stackoverflow.com/"
Group 1: "stackoverflow.com"
Group 2: "/"
Match "https://stackoverflow.com/questions/tagged/regex"
Group 1: "stackoverflow.com"
Group 2: "/questions/tagged/regex"
See? The first group has not been captured. The parser uses it to match the text, but ignores it later, in the final result.
EDIT:
As requested, let me try to explain groups too.
Well, groups serve many purposes. They can help you to extract exact information from a bigger match (which can also be named), they let you rematch a previous matched group, and can be used for substitutions. Let's try some examples, shall we?
Imagine you have some kind of XML or HTML (be aware that regex may not be the best tool for the job, but it is nice as an example). You want to parse the tags, so you could do something like this (I have added spaces to make it easier to understand):
\<(?<TAG>.+?)\> [^<]*? \</\k<TAG>\>
or
\<(.+?)\> [^<]*? \</\1\>
The first regex has a named group (TAG), while the second one uses a common group. Both regexes do the same thing: they use the value from the first group (the name of the tag) to match the closing tag. The difference is that the first one uses the name to match the value, and the second one uses the group index (which starts at 1).
Let's try some substitutions now. Consider the following text:
Lorem ipsum dolor sit amet consectetuer feugiat fames malesuada pretium egestas.
Now, let's use this dumb regex over it:
\b(\S)(\S)(\S)(\S*)\b
This regex matches words with at least 3 characters, and uses groups to separate the first three letters. The result is this:
Match "Lorem"
Group 1: "L"
Group 2: "o"
Group 3: "r"
Group 4: "em"
Match "ipsum"
Group 1: "i"
Group 2: "p"
Group 3: "s"
Group 4: "um"
...
Match "consectetuer"
Group 1: "c"
Group 2: "o"
Group 3: "n"
Group 4: "sectetuer"
...
So, if we apply the substitution string:
$1_$3$2_$4
... over it, we are trying to use the first group, add an underscore, use the third group, then the second group, add another underscore, and then the fourth group. The resulting string would be like the one below.
L_ro_em i_sp_um d_lo_or s_ti_ a_em_t c_no_sectetuer f_ue_giat f_ma_es m_la_esuada p_er_tium e_eg_stas.
You can use named groups for substitutions too, using ${name}.
To play around with regexes, I recommend http://regex101.com/, which offers a good amount of details on how the regex works; it also offers a few regex engines to choose from.
You can use capturing groups to organize and parse an expression. A non-capturing group has the first benefit, but doesn't have the overhead of the second. You can still say a non-capturing group is optional, for example.
Say you want to match numeric text, but some numbers could be written as 1st, 2nd, 3rd, 4th,... If you want to capture the numeric part, but not the (optional) suffix you can use a non-capturing group.
([0-9]+)(?:st|nd|rd|th)?
That will match numbers in the form 1, 2, 3... or in the form 1st, 2nd, 3rd,... but it will only capture the numeric part.
?: is used when you want to group an expression, but you do not want to save it as a matched/captured portion of the string.
An example would be something to match an IP address:
/(?:\d{1,3}\.){3}\d{1,3}/
Note that I don't care about saving the first 3 octets, but the (?:...) grouping allows me to shorten the regex without incurring the overhead of capturing and storing a match.
HISTORICAL MOTIVATION:
The existence of non-capturing groups can be explained with the use of parenthesis.
Consider the expressions (a|b)c and a|bc, due to priority of concatenation over |, these expressions represent two different languages ({ac, bc} and {a, bc} respectively).
However, the parenthesis are also used as a matching group (as explained by the other answers...).
When you want to have parenthesis but not capture the sub-expression you use NON-CAPTURING GROUPS. In the example, (?:a|b)c
Let me try this with an example:
Regex Code: (?:animal)(?:=)(\w+)(,)\1\2
Search String:
Line 1 - animal=cat,dog,cat,tiger,dog
Line 2 - animal=cat,cat,dog,dog,tiger
Line 3 - animal=dog,dog,cat,cat,tiger
(?:animal) --> Non-Captured Group 1
(?:=)--> Non-Captured Group 2
(\w+)--> Captured Group 1
(,)--> Captured Group 2
\1 --> result of captured group 1 i.e In Line 1 is cat, In Line 2 is cat, In Line 3 is dog.
\2 --> result of captured group 2 i.e comma (,)
So in this code by giving \1 and \2 we recall or repeat the result of captured group 1 and 2 respectively later in the code.
As per the order of code (?:animal) should be group 1 and (?:=) should be group 2 and continues..
but by giving the ?: we make the match-group non captured (which do not count off in matched group, so the grouping number starts from the first captured group and not the non captured), so that the repetition of the result of match-group (?:animal) can't be called later in code.
Hope this explains the use of non capturing group.
It makes the group non-capturing, which means that the substring matched by that group will not be included in the list of captures. An example in ruby to illustrate the difference:
"abc".match(/(.)(.)./).captures #=> ["a","b"]
"abc".match(/(?:.)(.)./).captures #=> ["b"]
Groups that capture you can use later on in the regex to match OR you can use them in the replacement part of the regex. Making a non-capturing group simply exempts that group from being used for either of these reasons.
Non-capturing groups are great if you are trying to capture many different things and there are some groups you don't want to capture.
Thats pretty much the reason they exist. While you are learning about groups, learn about Atomic Groups, they do a lot! There is also lookaround groups but they are a little more complex and not used so much.
Example of using later on in the regex (backreference):
<([A-Z][A-Z0-9]*)\b[^>]*>.*?</\1> [ Finds an xml tag (without ns support) ]
([A-Z][A-Z0-9]*) is a capturing group (in this case it is the tagname)
Later on in the regex is \1 which means it will only match the same text that was in the first group (the ([A-Z][A-Z0-9]*) group) (in this case it is matching the end tag).
tl;dr non-capturing groups, as the name suggests are the parts of the regex that you do not want to be included in the match and ?: is a way to define a group as being non-capturing.
Let's say you have an email address example#example.com. The following regex will create two groups, the id part and #example.com part. (\p{Alpha}*[a-z])(#example.com). For simplicity's sake, we are extracting the whole domain name including the # character.
Now let's say, you only need the id part of the address. What you want to do is to grab the first group of the match result, surrounded by () in the regex and the way to do this is to use the non-capturing group syntax, i.e. ?:. So the regex (\p{Alpha}*[a-z])(?:#example.com) will return just the id part of the email.
A Simple Answer
Use them to ensure one of several possibilities occur here (?:one|two) or an optional phrase camp(?:site)? or in general, anywhere you want to establish a group/phrase/section without needing to refer to it specifically.
They keep your captured group(s) count to a minimum.
I cannot comment on the top answers to say this: I would like to add an explicit point which is only implied in the top answers:
The non-capturing group (?...)
does not remove any characters from the original full match, it only reorganises the regex visually to the programmer.
To access a specific part of the regex without defined extraneous characters you would always need to use .group(<index>)
Well I am a JavaScript developer and will try to explain its significance pertaining to JavaScript.
Consider a scenario where you want to match cat is animal
when you would like match cat and animal and both should have a is in between them.
// this will ignore "is" as that's is what we want
"cat is animal".match(/(cat)(?: is )(animal)/) ;
result ["cat is animal", "cat", "animal"]
// using lookahead pattern it will match only "cat" we can
// use lookahead but the problem is we can not give anything
// at the back of lookahead pattern
"cat is animal".match(/cat(?= is animal)/) ;
result ["cat"]
//so I gave another grouping parenthesis for animal
// in lookahead pattern to match animal as well
"cat is animal".match(/(cat)(?= is (animal))/) ;
result ["cat", "cat", "animal"]
// we got extra cat in above example so removing another grouping
"cat is animal".match(/cat(?= is (animal))/) ;
result ["cat", "animal"]
To complement other good answers in this thread, I want to add an interesting observation that I came across.
Finding: You can have a capturing group inside a non-capturing group.
Problem Details: Have a look at below regex for matching web urls:
var parse_url_regex = /^(?:([A-Za-z]+):)(\/{0,3})([0-9.\-A-Za-z]+)(?::(\d+))?(?:\/([^?#]*))?(?:\?([^#]*))?(?:#(.*))?$/;
Here is my input url string:
var url = "http://www.ora.com:80/goodparts?q#fragment";
The first group in my regex (?:([A-Za-z]+):) is a non-capturing group which matches the protocol scheme (http) and colon (:) character. In continuation it becomes http:. But when I ran below code:
console.debug(parse_url_regex.exec(url));
I could see that the 1st index of the returned array was containing the string http instead (Refer screenshot).
At this point, my thinking was that http and colon : both will not get reported in the output as they are inside a non-capturing group. If the first regex group (?:([A-Za-z]+):) is a non-capturing group then why it is returning http string in the output array?
Explanation: So if you notice, ([A-Za-z]+) is a capturing group (not having ?: at the beginning). But this capturing group is itself inside a non-capturing group (?:([A-Za-z]+):) followed by a : character. That's why the text http still gets captured but the colon : character which is falling inside the non-capturing group (but outside the capturing group ) doesn't get reported in the output array.
In complex regular expressions you may have the situation arise where you wish to use a large number of groups some of which are there for repetition matching and some of which are there to provide back references. By default the text matching each group is loaded into the backreference array. Where we have lots of groups and only need to be able to reference some of them from the backreference array we can override this default behaviour to tell the regular expression that certain groups are there only for repetition handling and do not need to be captured and stored in the backreference array.
(?: ... ) acts as a group ( ... ) but doesn't capture the matched data. It's really much more efficient than a standard capture group. It is use when you want to group something but don't need to reuse it later. #Toto
Let me take to you an example of geo coordinate, the below matches two groups
Latitude,Longitude
([+-]?\d+(?:\.\d+)?),([+-]?\d+(?:\.\d+)?)
Lets take one ([+-]?\d+(?:\.\d+)?)
co-ordinate can be whole number like 58 or could be 58.666
Hence the optional (.666) second part (\.\d+)? is mentioned.
(...)? - for optional
But it is parenthesised, that will be another group of match. and we dont want two matches one for 58 and another for .666, we need single latitude as match. Here comes non-capturing group (?:)
with non-capturing group [+-]?\d+(?:\.\d+)?, 58.666 and 58 both are single match
Its extremely simple, We can understand with simple date example, suppose if the date is mentioned as 1st January 2019 or 2nd May 2019 or any other date and we simply want to convert it to dd/mm/yyyy format we would not need the month's name which is January or February for that matter, so in order to capture the numeric part, but not the (optional) suffix you can use a non-capturing group.
so the regular expression would be,
([0-9]+)(?:January|February)?
Its as simple as that.
I think I would give you the answer. Don't use capture variables without checking that the match succeeded.
The capture variables, $1, etc, are not valid unless the match succeeded, and they're not cleared, either.
#!/usr/bin/perl
use warnings;
use strict;
$_ = "bronto saurus burger";
if (/(?:bronto)? saurus (steak|burger)/)
{
print "Fred wants a $1";
}
else
{
print "Fred dont wants a $1 $2";
}
In the above example, to avoid capturing bronto in $1, (?:) is used.
If the pattern is matched , then $1 is captured as next grouped pattern.
So, the output will be as below:
Fred wants a burger
It is Useful if you don't want the matches to be saved.
Open your Google Chrome devTools and then Console tab: and type this:
"Peace".match(/(\w)(\w)(\w)/)
Run it and you will see:
["Pea", "P", "e", "a", index: 0, input: "Peace", groups: undefined]
The JavaScript RegExp engine capture three groups, the items with indexes 1,2,3. Now use non-capturing mark to see the result.
"Peace".match(/(?:\w)(\w)(\w)/)
The result is:
["Pea", "e", "a", index: 0, input: "Peace", groups: undefined]
This is obvious what is non capturing group.
I have database have thousand of unknow string they may be emails ,phonenum
BUT they are not for me mean they are not email or cell num for me they are only string for me but i want their common pattern so here is the string for example purposes
link to example click here
now what i want is this file out put if pattern matcehs 3 time here what i am doing is
DECLARE #strs2 nvarchar(255)
DECLARE #patternTable table(
id int ,
order by p.pat
but my example return this
485-2889
485-2889
) 485-2889
) 485-2889
.aol.com/aol/search?
.aol.com/aol/search?
gmail.com
gmail.com
but i want to add this for pattern
[a-zA-Z 0-9] [a-zA-Z 0-9] [a-zA-Z 0-9] - 485-2889
for gmail
[a-zA-Z 0-9] [a-zA-Z 0-9]# gmail.com
First of all, this is much more work than it might seem.
As far as I can say it's going to be method with heavy processing (and probably not something you want to do with a cursor in SQL (cursors are sort of bad in terms of efficiency).
You have to define a way for your code to identify a pattern. You will also have to work in priorities where a set of strings matches multiple patterns. For instance if you implement following pattern criteria (in your example):
BK-M18B-48
BK-M18B-52
BK-M82B-44
BK-M82S-38
BK-M82S-44
BK-R50B-58
BK-R50B-62
.....
should generate BK-[A-Z]-[0-9][0-9][A-Z]-[0-9][0-9]
Then next set can have multiple patterns as a result:
fedexcarepackage#outlook.com (example added for explanations)
fedexcarepackage#office.com
fedexcourierexpress#pisem.net
fedexcouriers#gmail.com ( another example added for explanations)
.....
Can generate :
fedexc%#%.% (as you said)
fedexc%#% (depending on processing)
fedexc[A-Z][A-Z]....%#%[A-Z]....[A-Z].[A-Z][A-Z][A-Z] (alphanumeris with '%' to compensate for length difference)
in addition to that if you take away fedexcarepackage#outlook.com from string list you get 1 additional pattern that you probably don't want to have:
fedexc%#%i%.% (because they have 'i' somewhere between the '#' and '.' (dot)
Anyway, that is something you will have to consider with your design.
I'll give you some basic logic you can work with:
Create a functions to identify each distinct pattern (1 pattern / function). For instnace, 1 function to check for static pieces of string (and attaching wildcards); Another to detect [A-Z],[0-9] patterns that match your conditions for this pattern to be valid; more if needed for different patterns.
Create a function to test a string with your pattern. So say you have 4 string, you find a pattern when comparing first 2 of them. Then you use this function to test if pattern applies to 3rd and 4th strings.
Create a function to test if 2 patterns are mutually exclusive. For instance 'PersonA#yahoo.%' and 'PersonA#%.net' patterns are not mutually exclusive, if they were both tested to be true. 'Person%#yahoo.com' and 'PersonB#yahoo.com' are mutually exclusive (both patterns cannot be true, so 1 is redundant.
Create a function to combine patterns that are NOT mutually exclusive (probably includes the use of function in 2nd and 3rd point). So 'PersonA#yahoo.%' and 'PersonA#%.net' can be combined into 'PersonA#%.%'
Once you have that setup, loop through each text line, and compare Current line to the next against each pattern criteria. Record any patterns you find (in a variable dedicated to that criteria, (don't mix them just yet).
Next comes the hardest part, safest way is to compare each pattern you find against each of the strings, to rule out the ones that don't apply to all strings. However, you could probably work out a way to combine patterns (in the same category) without cross checking
Finally, after you narrowed own your pattern list to 1 pattern per pattern type. Combine them into 1 or eliminate the ones
Keep in mind that in your pattern detection functions, you'll probably have to test each line multiple times and combine patterns. Some pseudo code to demonstrate:
Function CompareForStringMatches (String s1, String s2){ -- it should return a possible pattern found.
Array/List pattern;
int patternsFound=0;
For(i = 0, to length of shorter string){
For(x = 0, to length of shorter string){
if(longerString.contains(shorterString.substring(from i, to x)){
--record the pattern somewhere as:
pattern[patternsFound] = Replace(longerString, shorterString.Substring(from i, to x), '%') --pattern = longerString with substring replaced with '%' sign
patternsFound = patternsFound+1;
}
}
}
--After loops make another loop to check (partial) patterns against each other to eliminate patterns that are part of a larger pattern
--for instance Comparing 'random#asd.com' and 'sundom#asd.com' the patterns below should be found:
---compare'%andom#asd.com' and '%ndom#asd.com' and eliminate the first pattern, because both are valid, but second pattern includes the first one.
--You will have a lot of similar matches, but if you do this, you should end up with only a few patterns.
--after first cycle of checks do another one to combine patterns, where possible(for instance if you compare 'random#asd.com' and 'sundom#asd.net' you will end up with these 2 patterns'%ndom#asd.com' and 'Random#asd.%'.
--Since these patterns are true (because they were found during a comparison) you can combine them into '%ndom#asd.%'
--when you combine/eliminate all patterns, you should only have 1 left
return pattern[only pattern left];
}
PS: You can do things, much more efficiently, but if you have no idea where to start out, you probably need to do it the long way and work on improvements from first working prototypes.
Edit/Update
I suggest you make a wildcard detection method and then apply other patter checks you implement before it.
Wildcard detection for comparison of 2 strings (pseudo code), heavy processing version :
Compare 2 strings, check if every possible segment of shorter string is within longer:
for(int i = 0; i<shorterString.Length;i++){
for(int x = 0; i<shorterString.Length;i++){
if(longerString.contains(shorterString.substring(i,x))){ --from i to x
possiblePattern.Add(longerString.replace(shorterString.substring(i,x),'*')
--add to pattern list
}
}
--Next compare partal matches and eliminate ones that are a part of larger pattern
--So '*a#gmail.com' and '*na#yahoo.com' comparison should eliminate '*na#gmail.com', because if shorter pattern (with more symbols removed) is valid, then similar one with an extra symbol is part of it
--When that is done, combine remaining matches if there's more than 1 left.
--Remember, all patterns are valid if your first loop was correct, so '*#gmail.com' and 'personA#*.com' can be combined into '*#*.com
}
As for the alphanumeric detection. I would suggest you start by checking length of all strings. If they are the same, run the wildcard pattern detection method (for all of them). When done ONLY look for patern matches in wildcards.
So, You'll get a pattern like BK-*-* from wildcard detection run. On second iteration loop take 2 strings and only extract sub-strings that are represented by wildcard characters (use an array or an equivalent to store sub-strings, make sure not to combine both wildcards of a single string into 1 string).
So if you compare with pattern found above (BK-*-*) :
BK-M18B-48
BK-M18B-52
You should get following string sets to process after eliminating static characters:
Set 1:M18B and 48
Set 2:M18B and 52
Compare each character to opposite string in same position and check if characters match your category (like if String1[0].isaLetter AND String2[0].isaLetter). If they do add that 1 character to a pattern, if not either:
Add a wildcard character (will lead to pattern like BK-[A-Z]*[0-9][0-9]-[0-9][0-9]. If you do this combine adjacent wildcard characters to 1.
Pattern is false and you should abbort the ch'eck returning no patterns.
Use this basic logic to loop through strings, create (and store!!!!) patterns for each set of 2 strings. Loop through patterns, with wildcard detection (possibly a lighter version) to combine/eliminate paterns. So if you get patterns like '#yahoo.com' and '#gmail.com' from different sets of strings you should combine them into '#.com'
Keep in mind there's lots of room for optimization here.
I need to match two ipaddress/hostname with a regular expression:
Like 20.20.20.20
should match with 20.20.20.20
should match with [http://20.20.20.20/abcd]
should not match with 20.20.20.200
should not match with [http://20.20.20.200/abcd]
should not match with [http://120.20.20.20/abcd]
should match with AB_20.20.20.20
should match with 20.20.20.20_AB
At present i am using something like this regular expression: "(.*[^(\w)]|^)20.20.20.20([^(\w)].*|$)"
But it is not working for the last two cases. As the "\w" is equal to [a-zA-Z0-9_]. Here I also want to eliminate the "_" underscore. I tried different combination but not able to succeed. Please help me with this regular expression.
(.*[_]|[^(\w)]|^)10.10.10.10([_]|[^(\w)].*|$)
I spent some more time on this.This regular expression seems to work.
I don't know which language you're using, but with Perl-like regular expressions you could use the following, shorter expression:
(?:\b|\D)20\.20\.20\.20(?:\b|\D)
This effectively says:
Match word boundary (\b, here: the start of the word) or a non-digit (\D).
Match IP address.
Match word boundary (\b, here: the end of the word) or a non-digit (\D).
Note 1: ?: causes the grouping (\b|\D) not to create a backreference, i.e. to store what it has found. You probably don't need the word boundaries/non-digits to be stored. If you actually need them stored, just remove the two ?:s.
Note 2: This might be nit-picking, but you need to escape the dots in the IP address part of the regular expression, otherwise you'd also match any other character at those positions. Using 20.20.20.20 instead of 20\.20\.20\.20, you might for example match a line carrying a timestamp when you're searching through a log file...
2012-07-18 20:20:20,20 INFO Application startup successful, IP=20.20.20.200
...even though you're looking for IP addresses and that particular one (20.20.20.200) explicitly shouldn't match, according to your question. Admittedly though, this example is quite an edge case.
I'd like to create a regular expression such that when I compare the a string against an array of strings, matches are returned with the regex ignoring certain characters.
Here's one example. Consider the following array of names:
{
"Andy O'Brien",
"Bob O'Brian",
"Jim OBrien",
"Larry Oberlin"
}
If a user enters "ob", I'd like the app to apply a regex predicate to the array and all of the names in the above array would match (e.g. the ' is ignored).
I know I can run the match twice, first against each name and second against each name with the ignored chars stripped from the string. I'd rather this by done by a single regex so I don't need two passes.
Is this possible? This is for an iOS app and I'm using NSPredicate.
EDIT: clarification on use
From the initial answers I realized I wasn't clear. The example above is a specific one. I need a general solution where the array of names is a large array with diverse names and the string I am matching against is entered by the user. So I can't hard code the regex like [o]'?[b].
Also, I know how to do case-insensitive searches so don't need the answer to focus on that. Just need a solution to ignore the chars I don't want to match against.
Since you have discarded all the answers showing the ways it can be done, you are left with the answer:
NO, this cannot be done. Regex does not have an option to 'ignore' characters. Your only options are to modify the regex to match them, or to do a pass on your source text to get rid of the characters you want to ignore and then match against that. (Of course, then you may have the problem of correlating your 'cleaned' text with the actual source text.)
If I understand correctly, you want a way to match the characters "ob" 1) regardless of capitalization, and 2) regardless of whether there is an apostrophe in between them. That should be easy enough.
1) Use a case-insensitivity modifier, or use a regexp that specifies that the capital and lowercase version of the letter are both acceptable: [Oo][Bb]
2) Use the ? modifier to indicate that a character may be present either one or zero times. o'?b will match both "o'b" and "ob". If you want to include other characters that may or may not be present, you can group them with the apostrophe. For example, o['-~]?b will match "ob", "o'b", "o-b", and "o~b".
So the complete answer would be [Oo]'?[Bb].
Update: The OP asked for a solution that would cause the given character to be ignored in an arbitrary search string. You can do this by inserting '? after every character of the search string. For example, if you were given the search string oleary, you'd transform it into o'?l'?e'?a'?r'?y'?. Foolproof, though probably not optimal for performance. Note that this would match "o'leary" but also "o'lea'r'y'" if that's a concern.
In this particular case, just throw the set of characters into the middle of the regex as optional. This works specifically because you have only two characters in your match string, otherwise the regex might get a bit verbose. For example, match case-insensitive against:
o[']*b
You can add more characters to that character class in the middle to ignore them. Note that the * matches any number of characters (so O'''Brien will match) - for a single instance, change to ?:
o[']?b
You can make particular characters optional with a question mark, which means that it will match whether they're there or not, e.g:
/o\'?b/
Would match all of the above, add .+ to either side to match all other characters, and a space to denote the start of the surname:
/.+? o\'?b.+/
And use the case-insensitivity modifier to make it match regardless of capitalisation.