Regular Expressions

Who am I?

• Jeremy Holland (jeremy.holland@gmail.com)
• I have been a professional programmer, mostly using Perl, since the Clinton administration.
• I work for a semiconductor in manufacturing operations.
  • We have production systems runnning SunOS 4.1.1 on a Sun 3/80.... both of which were EOL'ed back when I was in junior high.
  • We also have RHEL 8 servers running on 24-core servers.
  • Compatibility is the name of my game.

• These notes are available at http://www.jeremyholland.info/rgx.html
  • Though not much else is...

Table of Contents The Regular Expression Grammar Rules For Matching Storing Regexes Using Regexes Regex Memory Interpolating into Regexes Evaluated Replacements Extended Regexes Greed Tips

• Regular expressions (commonly called regexes) are one of Perl's most powerful features.
• In reality, regular expressions are an entirely separate programming language embedded into Perl. They define a grammar by which strings can be recognized.
• There are entire graduate-level courses on regular expressions—although these tend to focus on the regex theory, implementation, and optimization, not on practical use.
• In this course, we will focus on practical use.
  • Regular expressions are primarily used for three things:
    1. Matching strings of text against patterns (does this string look like a phone number?)
    2. Parsing text to extract information (extract the area code from this phone number)
    3. Altering strings by deleting or swapping out a substring that matches a pattern (remove the area code to get the base phone number)
  • Regexes are far more robust than simply using eq to test whether one string equals another, or using substr and index to pick apart a string.
• Most modern programming languages have some support for regular expressions, but it is often "tacked on" in the form of an external libraries. In some cases, it is only a partial implementation of regexes.
• Perl's strength lies in how regular expressions are integrated into the language as part of the basic syntax and how Perl supports extremely advanced regular expressions.
• Also, the regex engine in Perl is highly optimized. Therefore, working with string data using regular expressions tends to be very fast, even for complex operations.

The Regular Expression Grammar

The Basics

• A regex is represented as a string of text, surrounded by slashes.
  • One or more pattern modifiers can appear after the final slash; these will be covered later.
• Unless you use positional anchors (covered later) the pattern will match anywhere in the string. Therefore, the pattern
     /Bob/
  will match
     "Bob"
     "Bob is a guy"
     "My name is Bob"
     "I am called Bobby"
     "Let's see if Bob can go."
  
• In fact, it will match any string that has "Bob" as a substring.
• Note that regexes are (by default) case-sensitive. Thus, the above pattern will not match any of:
     "bob"
     "BOB"
     "bob is a guy."
     "My name is bob"
     "I am called bobby"
     "Let's see if BOB can go."
  

Quantifiers

• So far, we haven't replicated any functionality beyond index, but regular expressions can be far more complex. The next layer of functionality is quantifiers. These allow a sub-pattern (for example, a string or character) to match multiple times in a specific place.
• Use the + operator to match a sub-pattern one or more times:
     /(iss)+/
  will match any string that has the string "iss" in it one or more times:
     "Kissimmee"
     "Mississippi"
     "miss"
     "Miss Mississippi"
  but will not match any string that does not have "iss" in it.
  • Note in the final example that only the first "iss" was part of the match.
• Note that quantifiers apply to a single sub-pattern, which all else being equal is a single letter.
  • In other words, /ab+/ matches strings like "xabx", "xabbbx", and "xabbbbbx" (as well as, of course, strings like "xabababababx").
  • You can create a sub-pattern with parentheses: /(ab)+/ matches strings like "xabx", "xababx", and "xabababababx" (and "xababxababxababx").
• Combine the quantifier with other sub-patterns for more sophisticated matches:
     /have a (very )+nice day/
  will match
     "have a very nice day"
     "have a very very nice day"
     "have a very very very nice day"
  and so forth (as well as strings like "I hope you have a very nice day today!" ), but will not match any of the following:
     "have a nice day"
     "have a very very "
     "very very very nice day"
     "very very very "
  all three sub-patterns must be matched for the overall regex to match.
  • The string "nice day very have a " also fails to match, because the sub-parts are all there, but not in order.
  • The string "have a completely very good and nice day" fails to match, because the sub-parts are all there, but not adjacently.
• Notice the space is inside the parentheses. Regexes are very sensitive to space and positioning, so be careful!
  • For example, consider the pattern
       /have a (very)+ nice day/
    Note the slight difference from the pattern above: the space is now outside the quantifier. The strings this pattern actually matches are:
       "have a very nice day"
       "have a veryvery nice day"
       "have a veryveryvery nice day"
    
  • Similarly, consider the pattern
       /have a (very )+ nice day/
    Here there is a space both inside and outside the quantifier. The strings this matches are:
       "have a very  nice day"
       "have a very very  nice day"
       "have a very very very  nice day"
    Note the double spaces. One is used to match the quantified sub-pattern; the other is needed to match the space outside the quantifier.
• Use the * operator to match a sub-pattern zero or more times:
     /(iss)*/
  will still match
     "Kissimmee"
     "Mississippi"
     "miss"
  and will also match strings with zero occurrence of "iss" in them. This is obviously not very useful by itself (because every string has zero or more occurrences of "iss" in it), but when used in combination with other sub-patterns...
     /have a (very )*nice day/
  will match strings containing
     "have a very nice day"
     "have a very very nice day"
     "have a very very very nice day"
  etc., and this time will also match:
     "have a nice day"
  but will still not match any of the following:
     "have a very very "
     "very very very nice day"
     "very very very "
  
  • Again, be careful of spacing:
       /have a (very)* nice day/
    matches
       "have a very nice day"
       "have a veryvery nice day"
       "have a veryveryvery nice day"
    and
       "have a  nice day" (note the double spaces)
• Use the ? operator to match a sub-pattern zero or one times:
     /have a (very )?nice day/
  will match strings containing
     "have a nice day"
     "have a very nice day"
  and nothing else.
  • Final warning regarding spacing:
       /have a (very)? nice day/
    matches
       "have a very nice day"
    and
       "have a  nice day" (again, note the double spaces)
• Use braces for arbitrary quantification
  • sub-pattern{count} matches exactly count instances of sub-pattern
  • sub-pattern{min,max} matches between min and max instances of sub-pattern, inclusive
  • sub-pattern{min,} matches at least min instances of sub-pattern
  • sub-pattern{0,max} matches at most max instances of sub-pattern (or none)
  • sub-pattern{1,max} matches at most max instances of sub-pattern (but at least one)
  • Examples:
       /have a (very ){3}nice day/
       /have a (very ){0,3}nice day/
       /have a (very ){1,5}nice day/
       /have a (very ){3,5}nice day/
       /have a (very ){2,}nice day/
    
  • Note these can all be represented without braces:
       /have a (very )(very )(very )nice day/
       /have a (very )?(very )?(very )?nice day/
       /have a (very )(very )?(very )?(very )?(very )?nice day/
       /have a (very )(very )(very )(very )?(very )?nice day/
       /have a (very )(very )+nice day/
    But the braces are much easier to interpret.
  • Also, note that the +, *, and ? quantifiers can be represented using braces:

    the quantifier...	equivalent to...
    sub-pattern+	sub-pattern{1,}
    sub-pattern*	sub-pattern{0,}
    sub-pattern?	sub-pattern{0,1}

    but again, the simple quantifiers are easier to understand in most cases.
    • And /(subpattern)+/ is equivalent to /(subpattern)(subpattern)*/
  • Remember that patterns can match anywhere in the string, so the pattern /x{3}/ will match any string with three xs in a row, including strings with four xs in a row (or more).
    • Typically quantifiers are used with positional anchors (see below) or are adjacent to other sub-patterns that limit the match.
    • For example, the pattern /ax{3}a/ will match strings containing "axxxa", (exactly three xs surrounded by as). Four xs surrounded by as (axxxxa) won't match.

Alternation

• Alternation allows you to choose one of a list of possible choices.
• Use a vertical bar (|) for alternation:
     /man|bear|pig/
  Will match all strings containing "man", "bear", or "pig".
• Note that /abc|xyz/ is equivalent to
     /(abc)|(xyz)/
  not
     /(ab)(c|x)(yz)/
  
• Spaces and alternation don't interact "intuitively": /A B|C D/ is the same as
     /(A B)|(C D)/
  not
     /(A )(B|C)( D)/
  
• This may not be what you expect, given the above example /ab+/, (which is equivalent to /(a)(b+)/).
  • The explanation has to do with the precedence of operators in regexes.
  • When using alternation and spaces, (and in general when using regexes) parentheses are recommended for clarity.
  • In practical use, something like:
       /I live in (ME|NH|VT|MA|RI|CT)/
    will match statements entered by people who live in New England.

Character Classes

• Several symbols represent a character class, or any of a group of characters:
  • \s means "any space character" (in practical use, this is spaces, tabs, and newlines)
  • \S means "any non-space character"
  • \d means "any digit"
  • \D means "any non-digit"
  • \w means "any word character (letters, numbers, underscores)"
  • \W means "any non-word character"
  • . (a dot) means "any single character" except newline (unless you use the /s pattern modifier, in which case it matches a newline as well; see below).
• To define a custom character class, use brackets, and possibly dashes:
  
  • A single character means "this character is part of the class".
  • Two characters separated by a dash means "all the characters (ASCIIbetically) between the two, inclusively".
  • The class can contain multiple characters and ranges, which are simply placed adjacently.
  • Examples:
    • [13579] matches any single odd digit.
    • [a-m] matches any lowercase letter between a and m, while
    • [A-M] matches any uppercase letter in the same range.
    • [A-Ma-m] matches either
• A complement class matches any character except those specified. To specify a complement class, use a caret ("^") as the first character inside the brackets.
  Examples:
  • The character class
       [^A-Z]
    matches anything except an uppercase letter.
    
  • The character class
       [^aeiouAEIOU]
    matches anything except a vowel.
• Character classes can include the "named" classes:
     [\dA-F]
  matches any (uppercase) hex digit. The class
     [^\d\s]
  matches any character besides digits and whitespace.
• Be careful when mixing uppercase and lowercase letters.
  • For example,
       [A-Za-z]
    matches anything from A to Z or a to z, or in other words, any letter character.
  • However
       [A-z]
    matches any character between A (ASCII value 65) and z (ASCII value 122), which includes all upper- and lower-case letters but also includes brackets and other punctuation characters you probably did not intend to be part of the character class.
• Discussion question: what characters are in the class [^\W_]?
• No matter how many characters are in a class, a class only matches one character (it can be one of any of the characters in the class). So the pattern
     /[ABCDEFGHIJKLMNOPQRSTUVWXYZ]/
  despite having 26 characters in it, only matches a single uppercase letter.
  • If you want to match more than one of a class, you must replicate the class, as in
       /[ABCDEFGHIJKLMNOPQRSTUVWXYZ][ABCDEFGHIJKLMNOPQRSTUVWXYZ]/
    which matches two uppercase letters.
  • You may also (preferably) use quantification (see above).
• Classes can be used with repetition and alternation operators. Example:
     /\d{5}(-\d{4})?/
  matches a ZIP code.
     /[aeiou]+/
  matches one or more vowels.
• Another example:
     /have a .*day/
  (remember that dot is a character class) matches
     "have a nice day"
     "have a very nice day"
     "have a Tuesday"
     "have a rotten day"
     "have a Ms3!a 54dPO2$ -)eEq1day"
     "have a         day"
     "yo, dude, have a day!!!"
  
• Classes are more efficient than alternation, so the pattern
     /([aeiou])+/
  will match much faster than the pattern
     /(a|e|i|o|u)+/
  even though they are functionally equivalent.

Anchors

• Anchors match conditions in the string but do not match characters.
• ^ matches the beginning of the string (or at the beginning of any "line", if you use the /m pattern modifier and the string contains embedded newlines)
• $ matches at the end of the string before the newline (if present) or at the end of any "line", under the circumstances given above
• The pattern /^hi/ will match the strings
     "hi"
     "hi there"
     "history"
  but will not match
     "this"
     "I said hi"
      " hi" (note the space)
      "Hillside" (note the upper-case)
  
• Similarly, the pattern /st$/ will match the strings
     "st"
     "most"
     "mnopqrst"
     "Come here Fast"
  but will not match
     "state"
      "st " (note the space)
      "Come here FAST" (note the upper-case)
  
• Using them together: Remember, patterns can match anywhere in the string being matched. Therefore, recall the pattern
     /have a (very )+nice day/
  will match
     "have a very nice day"
     "have a very very nice day"
     "have a very very very nice day"
  etc., but will also match
     "DO NOT have a very nice day, you bonehead"
  which is slightly different in tone... Add anchors to correct this:
     /^have a (very )+nice day$/
  will no longer match strings besides
     "have a very nice day"
     "have a very very nice day"
     "have a very very very nice day"
  and so forth
  
• Note that $ matches the end of the string, before the newline if any.
  • For example, the pattern /^hello$/ matches both "hello" and "hello\n" (the latter being similar to what you would read in from a file).
• Word boundaries:
  • Consider the string "ab_c .xyz" (why you are considering this particular string is left to the imagination). Where are the "words"?
  • \b matches at a word boundary (between a \w character and a \W character, or at the beginning/end of a string if the first/last character is a word character):
       "ab_c .xyz!!"
  • \B matches at a non-word-boundary (between two \w characters or between two \W characters, or at the beginning/end of a string if the first/last character is a non-word character)):
       "ab_c .xyz!!"
  • As you recall, the pattern /it/ will match any string with the letters "it" in it, including:
       "it"
       "spit"
       "itch"
       "fritter"
       "this is it!"
       "is it here?"
       "hit it" (Only one needs to match for the whole string to match.)
    If you are looking for strings with the word "it" by itself, use the \b anchor:
       /\bit\b/
    will match only the following from the above list:
       "it"
       "this is it!"
       "is it here?"
       "hit it"
    Alternatively,
       /\Bit\B/
    will match only the following from the above list:
       "fritter"
    This is because \B does not match at the beginning and end of strings. To match strings containing words with the letters "it" but not the word "it" itself, you must do:
       /(it\B)|(\Bit)/
    
• An important point: anchors do not match characters. They match positions or conditions within the string.

Escaped Characters

• To literally match one of the characters in the regex grammar (or a slash), escape it with a backslash.
• Example 1:
     /^\$[a-zA-Z_]\w*\[[0-9]+\]$/
  matches legal Perl array lookups (with a numeric index), such as $foo[3].
  • Note the backslash before the first dollar sign (because it's meant literally) and the lack of backslash before the second one (because it's matching the end of the string).
  • Note also which brackets are escaped (literal) and which are defining character classes.
• Example 2:
     /^\d+\.\d+\.\d+\.\d+$/
  matches IP addresses (you might then want to check that each part was between 0 and 255).
• Inside a character class, you do not need to escape anything except [, ], /, and \.
  • You don't need to escape -; instead put it as the first (after any initial ^) or last character in the class and Perl will figure out what you mean.
  • You don't need to escape ^; just put it anywhere besides first.
  • Bonus discussion question: what character(s) do the following classes match?
       [--^]  [^-^]  [-^]
       [^--^]  [^-]  [^^]
• If your pattern contains slashes, you have to escape them. If your pattern contains a lot of slashes, this can get ugly:

  if ($filename =~ m/^(\/?([^\/]+\/)*([^\/]+)(\.[^\/]+)?$/) { # it is an absolute filename! }

  • In this case, you may use braces to delimit your pattern, which means you don't need to escape the slashes:

    if ($filename = m{^(/?([^/]+/)*([^/]+)(\.[^/]+)?$}) { # it is an absolute filename! }

    • OK, maybe this isn't clear, but it's clearER.

Case Sensitivity

• As mentioned, regular expressions are case-sensitive.
• It is always possible to write a regex that ignores case, by using [a-zA-Z] for a letter instead of [a-z] or [A-Z] and by specifying literal text as [Pp][Ee][Rr][Ll].
• However, if you have long strings of literal text, this is very cumbersome.
• Instead, use the /i modifier to cause the regex to match case-insensitively.
• Just add an i after the regex, like this: /^PERL$/i;.
• Example: the code

  1 my @strings = ("I am Bobby", "I AM BOBBY", "i am bobby"); 2 3 foreach my $string (@strings) { 4 print " case-sensitive: "; 5 print "\"$string\" " . ($string =~ /Bob/ ? "matches" : "doesn't match") . " /Bob/\n"; 6 print "case-insensitive: "; 7 print "\"$string\" " . ($string =~ /Bob/i ? "matches" : "doesn't match") . " /Bob/i\n\n"; 8 }

  outputs

  case-sensitive: "I am Bobby" matches /Bob/ case-insensitive: "I am Bobby" matches /Bob/i case-sensitive: "I AM BOBBY" doesn't match /Bob/ case-insensitive: "I AM BOBBY" matches /Bob/i case-sensitive: "i am bobby" doesn't match /Bob/ case-insensitive: "i am bobby" matches /Bob/i

Rules For Matching

• In addition to the regex grammar itself, remember two things and you won't be steered wrong:
  1. Patterns always try to match the LONGEST LEFTMOST substring.
     • In other words, if a pattern can match two places in a string, it always matches as far to the left as possible.
     • If there are two possible matches at that leftmost spot, the longest one is the substring that is actually matched.
  2. When a pattern is matched against a string, the string must match all sub-patterns (be they individual characters, anchors, parenthetical expressions, alternations, or quantifications) IN-ORDER, ADJACENTLY.

Storing Regexes

• Regular expressions are just strings, and as such can be stored in a scalar variable, but there is a special function to facilitate this: qr//.
  • So this:

    if ($zip_data =~ m/^\d{5}(-\d{4})?$/) { print "Valid ZIP/ZIP+4 code\n"; }

    is the same as this:

    1 my $rgx_zip = qr/^\d{5}(-\d{4})?$/; 2 3 if ($zip_data =~ m/$rgx_zip/) { 4 print "Valid ZIP/ZIP+4 code\n"; 5 }

• This is helpful because it is a form of encapsulation: regexes are defined away from where they are used.
• It is especially beneficial as a form of code re-use: storing regexes that will be used in different places throughout your program.
• Finally, it facilitates use creating commented multi-line regexes using the /x modifier (see below); it is unwieldy to use /x in-place.

Using Regexes

Matching Using Regexes

• The most basic regex operation is determining whether a string matches a given pattern. This is done using the =~ operator (pronounced "matches") and the m/pattern/ construct.

  1 my @strings = qw(apple banana cucumber durian potato); 2 foreach my $string (@strings) { 3 if ($string =~ m/^([bcdfghjklmnpqrstvwxyz][aeiouy])+$/) { 4 print "\"$string\" matches CVCV...CV\n"; 5 } 6 else { 7 print "\"$string\" does not match\n"; 8 } 9 }

  outputs:

  "apple" does not match "banana" matches CVCV...CV "cucumber" does not match "durian" does not match "potato" matches CVCV...CV

  • Note m in front of the pattern. This is technically not required for simple pattern matching but it's best practice to always include it.
• In scalar (typically Boolean) context, the pattern match operation returns a true value if the string matches the pattern and false if not.
• What m// returns in list context is covered later.
• The "does not match" operator, !~, is simply the inverse of =~. It is primarily used in Boolean context.
• Note that m// works on any string, including non-lvalues such as a function return. Therefore, you can use m// to test whether a function returned a string that matches a certain pattern, or to test whether a constant string matches a (presumably variable) pattern.
• If you use the "global match" modifier ("/g"), each match remembers where it left off. Matching the same pattern against the same string repeatedly does a progressive match.
  • The code
       "abcdefghijklmno" =~ m/[aeiou]/g (match one vowel)
    will match four substrings of the string if called repeatedly:
       abcdefghijklmno
  • The code
       "abcdefghijklmno" =~ m/[aeiou]./g (match a vowel followed by another character)
    will match three substrings:
       abcdefghijklmno
    • It can't match where the "o" is, because there is no character after it.
  • The code
       "abcdefghijklmno" =~ m/.[aeiou]/g (match a character followed by a vowel)
    will match three substrings:
       abcdefghijklmno
    • It can't match where the "a" is because there is no character preceding it.
  • The code
       "abcdefghijklmno" =~ m/^[aeiou]/g (match a vowel at the beginning of the string)
    will only match one substring:
       abcdefghijklmno
    as will the code
       "abcdefghijklmno" =~ m/[aeiou]$/g (match a vowel at the end of the string)
    which matches:
       abcdefghijklmno
    • These can't match any of the other vowels because they aren't at the beginning/end of the string.
    • It is almost always pointless to use a progressive match that includes anchors like ^ and $ unless it is a string with embedded newlines and the regex has the /m modifier.
  • The code
       "abcdefghijklmno" =~ m/[aeiou].{3}/g (match a vowel followed by three characters)
    will match three substrings:
       abcdefghijklmno
    • But the code
         "abcdefghijklmno" =~ m/[aeiou].{4}/g (match a vowel followed by four characters)
      will only match two substrings:
         abcdefghijklmno
      • It can't match where the "e" is because the leave-off point has already progressed beyond it (the "e" was one of the four characters following the "a".)
  
  A code-based example:

  1 my $string = "this 47 string 390 has numbers 000 embedded in it."; 2 my $match_count = 0; 3 while (1) { 4 if ($string =~ m/\d+/g) { 5 print "Found match ending before position " . pos($string) . "\n"; 6 ++$match_count; 7 } 8 else { 9 print "No more matches!\n"; 10 last; 11 } 12 } 13 print "Found $match_count total matches.\n";

  outputs

  Found match ending before position 7 Found match ending before position 18 Found match ending before position 34 No more matches! Found 3 total matches.

  • Note that the location where the next global match will start is accessible using the pos function.
  • When using progressive matching, beware positional anchors, especially absolute anchors, in the pattern.
  • Also be aware of non-matching stuff in between matching sections of the string. This is not so important in the example above, where the pattern allows us to only match what we want, but many patterns must match every character.

Substitution Using Regexes

• Another common regex operation is substituting. This is accomplished using the s/pattern/replacement/ construct with the =~ operator. A simple example: the code

  my $string = "Bahama mama"; print "Original String: \"$string\"\n"; $string =~ s/ma/pop/; print "Modified String: \"$string\"\n";

  which outputs

  Original String: "Bahama mama" Modified String: "Bahapop mama"

• s/// returns the number of substitutions made, regardless of context.
  • Therefore it doesn't make much sense to call s/// in list context.
  • In Boolean context, this helpfully evaluates to "true" if the substitution succeeded and "false" otherwise.
  • The value will always be 0 or 1 unless the substitution uses the "global" modifier ("/g", as it was with m//). This causes the s/// to find and replace ALL matching substrings with the replacement:

    1 my $string = "Bahama mama"; 2 print "Original String: \"$string\"\n"; 3 my $repl_count = ($string =~ s/ma/pop/g); 4 print "Modified String: \"$string\"\n"; 5 print " - made $repl_count replacements\n";

    outputs

    Original String: "Bahama mama" Modified String: "Bahapop poppop" - made 3 replacements

• Note that s/// is destructive. If you want to remember the original string, you must make a copy first. You can do this simply in one operation, (called "substitution en-passant"), like this:

  my $orig_string = "Bahama mama"; (my $mod_string = $orig_string) =~ s/ma$/pop/; print "Original String: \"$orig_string\"\n"; print "Modified String: \"$mod_string\"\n";

  which outputs

  Original String: "Bahama mama" Modified String: "Bahama mapop"

  • A side effect of s///'s destructiveness is that unlike m//, it can only be applied to lvalues. You cannot use s/// on constant strings, the return values of subroutine calls, etc. Code like this:

    "constant string" =~ s/a/b/g;

    blows up like this:

    Can't modify constant item in substitution (s///) at subst_const.pl line 1, near "s/a/b/g;" Execution of subst_const.pl aborted due to compilation errors.

  • Not that this type of operation makes any sense without first storing the value into a variable.
  • Substitution en-passant works when the original value is a non-lvalue:

    (my $string = "This is a non-lvalue") =~ s/a/x/g; print $string;

    outputs

    This is x non-lvxlue

• The replacement is treated as a normal double-quoted string. Therefore you can interpolate variables into it, as well as hash lookups, array indexes, etc.

Split Using Regexes

• We have already seen the split function used with the first argument to it being a single character that is used as a delimiter.
• However, the first argument to split can be a regular expression. That regex is matched repeatedly in the target string (although never overlapping) and the text in between each match is what is returned as the array of substrings.
• So for example, using just a single-character delimiter:

  my $string = "Johnson, Bill,345-F24-134A , x1457 ,Nashua"; my @fields = split ",", $string; print Dumper \@fields;

  outputs

  $VAR1 = [ 'Johnson', ' Bill', '345-F24-134A ', ' x1457 ', 'Nashua' ];

  • All the useless spaces around the commas are preserved.

  Using a regex as the delimiter yields much better results:

  my $string = "Johnson, Bill,345-F24-134A , x1457 ,Nashua"; my @fields = split /\s*,\s*/, $string; print Dumper \@fields;

  outputs

  $VAR1 = [ 'Johnson', 'Bill', '345-F24-134A', 'x1457', 'Nashua' ];

• Another common example is to use split /\s+/ to split a string on one or more whitespace characters, including spaces and tabs.
• Note that the first argument to split is the one place where you do not need the m in front of the slashes. It is obvious by the fact that you are calling split that the first argument is a regex.
• split is not destructive and can be applied to non-lvalues:

  my ($key, $value) = split /\s*=\s*/, read_config_line(), 2;

• If you give split an empty regex, it returns an array of the individual characters from the original string:

  my @chars = split //, "yee-ha!"; print Dumper \@chars;

  prints

  $VAR1 = [ 'y', 'e', 'e', '-', 'h', 'a', '!' ];

Regex Memory

• As seen, parentheses are used in regexes for grouping but they also have memory.
• What is matched inside the parentheses is captured, and can be accessed after the match.
• Use the match variable $1 to access the first sub-match—i.e., what was matched by the first set of parentheses (counting from the left). $2 contains what was matched by the second set of parentheses, and so on.
  • If you need to go higher than $9, you should consider splitting the string up into chunks first (perhaps by using capturing parentheses) and then further processing the chunks using smaller regexes.
  • $1 corresponds to the 0th match (i.e. the match variables are 1-indexed) because $0 means something totally different.
• Example:

  1 my $str = "phone number is 603-432-8696."; 2 my $rgx_phone = qr/(\d{3})-(\d{3})-(\d{4})/; 3 4 print "Searching the string \"$str\"\n"; 5 if ($str =~ $rgx_phone) { 6 print "Found a phone number. Submatches are [$1], [$2], [$3].\n"; 7 } 8 else { 9 print "Didn't find a phone number\n"; 10 }

  outputs:

  Searching the string "phone number is 603-432-8696." Found a phone number. Submatches are [603], [432], [8696].

• If the overall pattern doesn't match, the match variables will not contain meaningful information.
  • Make sure the match succeeded before using the match variables!
• If a match is performed in list context, it returns the match variables in list (e.g. ($1, $2, $3)), or an empty list if the match fails.
  • For example, the code

    my $string = "Julie ate a peach."; my @matches = ($string =~ m/([aeiou])([aeiou])/); print Dumper \@matches;

    outputs

    $VAR1 = [ 'i', 'e' ];

  • This is another time when using array slices on a returned value can be handy:

    1 my $rgx_zip_code = qr/^(\d{5})(-\d{4})?$/; 2 3 my $zip_code = "03060-1234"; 4 my $zip_code_base = ($zip_code =~ m/$rgx_zip_code/)[0]; 5 6 print "base of ZIP code \"$zip_code\" is \"$zip_code_base\"\n";

    outputs

    base of ZIP code "03060-1234" is "03060"

    • Note that $1 corresponds to element zero, $2 to element one, etc.
  • If the /g (global) modifier is used, in list context the match returns all sub-matches in the entire string as it matches repeatedly:

    my $string = "Julie ate a peach."; my @matches = ($string =~ m/([aeiou])([aeiou])/g); print Dumper \@matches;

    outputs

    $VAR1 = [ 'i', 'e', 'e', 'a' ];

  • Finally, if the pattern doesn't contain any capturing parentheses, evaluating the match in list context doesn't make sense, unless it's a global match, in which case it returns all the substrings that matched:

    my $string = "Julie ate a peach."; my @matches = ($string =~ m/[aeiou]{2}/g); print Dumper \@matches;

    outputs

    $VAR1 = [ 'ie', 'ea' ];

• You can also use the match variables in the replacement string.
  • Example:

    1 my $str = 'name = Mr. John Smith'; 2 my $rgx_name = qr/[A-Z][a-z]+\.\s+([A-Z][a-z]+)\s+([A-Z][a-z]+)/; 3 4 print "string before: \"$str\"\n"; 5 $str =~ s/$rgx_name/$2, $1/; 6 print "string after: \"$str\"\n";

    outputs:

    string before: "name = Mr. John Smith" string after: "name = Smith, John"

• Finally, you can even use the sub-matches later in the same pattern by referring to then as \1, \2, etc.
  • Example:

    1 my $text_ok = "This is the hour of our discontent"; 2 my $text_repeat = "This is the the hour of our discontent"; 3 my $rgx_repeat_words = qr/\b([a-z]+)\s+\1/i; 4 5 foreach my $text ($text_ok, $text_repeat) { 6 print "- searching for repeated words in string:\n"; 7 print " \"$text\"\n"; 8 if ($text =~ m/$rgx_repeat_words/) { 9 print " found repeated word \"$1\"\n"; 10 } 11 else { 12 print " no repeated words found\n"; 13 } 14 }

    outputs:

    - searching for repeated words in string: "This is the hour of our discontent" no repeated words found - searching for repeated words in string: "This is the the hour of our discontent" found repeated word "the"

  • This is known as backreferencing.
• If you are using Perl 5.10 or higher, you can also used named capture buffers. This stores the results of captures by name. After your match, you can access the sub-matches via the fantastically-named %+ hash. So the code:

  1 my @strings = qw(800-123-4567 987-6543 90210-1000); 2 my $rgx = qr/^((?<area_code>\d{3})-)?(?<number>\d{3}-\d{4})$/; 3 4 foreach my $str (@strings) { 5 print "parsing \"$str\"\n"; 6 if ($str =~ $rgx) { 7 print " - Number was $+{number}\n"; 8 print " - Area code was " . ($+{area_code} // "not present") . "\n"; 9 } 10 else { 11 print " - Not a phone number!\n"; 12 } 13 }

  outputs

  parsing "800-123-4567" - Number was 123-4567 - Area code was 800 parsing "987-6543" - Number was 987-6543 - Area code was not present parsing "90210-1000" - Not a phone number!

  • To use named buffers as backrefences, use the \g{} notation. For example, a simple regex to detect repeated words in text might look like
       /\b(?<word>[a-z]+) \g{word}\b/
• Named capture buffers can help write more readable code, and if you are using Perl 5.10+ on all of your systems, you should consider using them. However, there are vast swaths of Perl out there that use the numbered buffers, so make sure you understand them as well.
• If you want to group a sub-pattern for clarity or to force precedence, but don't want to capture it (an action also known as clustering), you can use non-capturing parentheses.
  • They are specified as (?: ... ).
  • They group like ordinary parentheses, but their contents are not captured.
  • For example:

    1 my @names = ("Mr. John Smith", "Bill Jackson", "Mrs. White", "Bill"); 2 my $rgx_lastname = qr/^(?:[A-Z][a-z]+\.)?\s*(?:[A-Z][a-z]+)?\s+([A-Z][a-z]+)/; 3 4 foreach my $name (@names) { 5 if ($name =~ m/$rgx_lastname/) { 6 print "For the name \"$name\", got last name \"$1\"\n"; 7 } 8 else { 9 print "The name \"$name\" did not match\n"; 10 } 11 }

    outputs:

    For the name "Mr. John Smith", got last name "Smith" For the name "Bill Jackson", got last name "Jackson" For the name "Mrs. White", got last name "White" The name "Bill" did not match

    • Note in this example, even though there are three sets of parentheses, only one set captures anything. This simplifies subsequent operations.
  • Non-capturing parentheses are usually good to use unless you know you will be capturing.
    • However, they make a regular expression harder to read because they are more complicated. For this reason, these notes (and many perl programmers) use regular parentheses unless it's important for parentheses NOT to capture.
• If you include capturing parentheses in a regex passed to split, what they match will be returned, interspersed with the delimited fields. It's usually best to use non-capturing parentheses in regexes passed to split unless you need to know what the delimiters were.
• What a match variable contains depends on how its corresponding parenthetical expression figures into the overall match.
  • Under most circumstances, the match variable will contains the part of the string its sub-pattern matched in the overall match.
    • Note that if its subpattern contains a quantifier that allows zero matches (e.g. ?, *, or {0,}), the match variable may be defined but empty ("").
  • If a sub-pattern matches multiple times because it (or a sub-match that contains it) is followed by a quantifier, the match variable associated with that expression will contain the last thing that matched.
  • If a sub-pattern is not included in the match (for instance, because it is on one side of an alternation, or is followed by a quantifier that allows zero matches, then the associated match variable will be undefined.
  • For example, the code

    1 # $1 will store the label, if present 2 # | 3 # | $2 will store the LAST set of 4 # | digits followed by a dash matched. 5 # | | 6 # | | $3 will store the LAST set of 7 # | | digits matched 8 # | | | 9 # | | | $4 contains $5 or $6 10 # | | | | 11 my $rgx_digit_cluster_str = qr/ ^ (\w+:)? ( (\d+)-? )* ( (X)+ | (Y+) ) (\.?) $ /x; 12 # | | | 13 # $5 contains one X if any are present | | 14 # | | 15 # $6 contains one or more Ys, if any are present | 16 # | 17 # $7 contains a trailing dot or is defined-but-empty if such is absent 18 # 19 # note the x modifier means spaces are ignored; we will discuss this later. 20 21 my @test_strings = ("123-456-789X", "98765-4321-XXX.", "A:123-456-789Y", 22 "A:98765-4321-YYY.", "A:X", "123--456"); 23 24 foreach my $test_string (@test_strings) { 25 print "TEST STRING: \"$test_string\"... "; 26 if ($test_string =~ m/$rgx_digit_cluster_str/) { 27 print "matches\n"; 28 printf " - \$1 is %-18s", (defined($1) ? "\"$1\"" : "undefined"); 29 printf " - \$4 is %-18s\n", (defined($4) ? "\"$4\"" : "undefined"); 30 printf " - \$2 is %-18s", (defined($2) ? "\"$2\"" : "undefined"); 31 printf " - \$5 is %-18s\n", (defined($5) ? "\"$5\"" : "undefined"); 32 printf " - \$3 is %-18s", (defined($3) ? "\"$3\"" : "undefined"); 33 printf " - \$6 is %-18s\n", (defined($6) ? "\"$6\"" : "undefined"); 34 printf " - \$7 is %-18s\n\n", (defined($7) ? "\"$7\"" : "undefined"); 35 } 36 else { 37 print "doesn't match\n\n"; 38 } 39 }

    outputs:

    TEST STRING: "123-456-789X"... matches - $1 is undefined - $4 is "X" - $2 is "789" - $5 is "X" - $3 is "789" - $6 is undefined - $7 is "" TEST STRING: "98765-4321-XXX."... matches - $1 is undefined - $4 is "XXX" - $2 is "4321-" - $5 is "X" - $3 is "4321" - $6 is undefined - $7 is "." TEST STRING: "A:123-456-789Y"... matches - $1 is "A:" - $4 is "Y" - $2 is "789" - $5 is undefined - $3 is "789" - $6 is "Y" - $7 is "" TEST STRING: "A:98765-4321-YYY."... matches - $1 is "A:" - $4 is "YYY" - $2 is "4321-" - $5 is undefined - $3 is "4321" - $6 is "YYY" - $7 is "." TEST STRING: "A:X"... matches - $1 is "A:" - $4 is "X" - $2 is undefined - $5 is "X" - $3 is undefined - $6 is undefined - $7 is "" TEST STRING: "123--456"... doesn't match

• Match variables persist until the next sucessful pattern match, at which point they are all cleared (i.e. they are undefined), even if the successfully-matched pattern doesn't contain any capturing parentheses!
  • Wait, what??
  • An example is in order. The code:

    1 sub print_match_vars { 2 print "\n --> "; 3 print defined $1 ? "\$1=\"$1\"; " : "\$1 is undefined; "; 4 print defined $2 ? "\$2=\"$2\"; " : "\$2 is undefined; "; 5 print defined $3 ? "\$3=\"$3\"\n" : "\$3 is undefined\n"; 6 } 7 8 my $string = "abcdefghijklmnopqrstuvwxyz"; 9 10 print "The pattern match variables start out undefined:"; 11 print_match_vars(); 12 13 print "After a successful match, those used in the pattern are defined: "; 14 print $string =~ m/a(.+)x(.+)z/ ? "(match)" : "(no match)"; 15 print_match_vars(); 16 17 print "After a failed match, they are unchanged: "; 18 print $string =~ m/([0-9]+)/ ? "(match)" : "(no match)"; 19 print_match_vars(); 20 21 print "After a successful match, they are set again based on the pattern: "; 22 print $string =~ m/([^aeiou])/ ? "(match)" : "(no match)"; 23 print_match_vars(); 24 25 print "After a successful match, they are all undefined: "; 26 print $string =~ m/abc/ ? "(match)" : "(no match)"; 27 print_match_vars();

    outputs

    The pattern match variables start out undefined: --> $1 is undefined; $2 is undefined; $3 is undefined After a successful match, those used in the pattern are defined: (match) --> $1="bcdefghijklmnopqrstuvw"; $2="y"; $3 is undefined After a failed match, they are unchanged: (no match) --> $1="bcdefghijklmnopqrstuvw"; $2="y"; $3 is undefined After a successful match, they are set again based on the pattern: (match) --> $1="b"; $2 is undefined; $3 is undefined After a successful match, they are all undefined: (match) --> $1 is undefined; $2 is undefined; $3 is undefined

    • Notice also that I did not pass $1, $2, etc. to the print_match_vars() subroutine. This serves to illustrate that the pattern match variables are all in the global scope.
    • It is better to pass needed match variables to a subroutine, because otherwise it is very unclear what's going on from the point-of-view of the subroutine (and its maintainer).
  • Because the match variables tend to get reset, it is a good idea to store them in other variables immediately.
  • This has the advantageous side effect of making your code much clearer, especially when passing them around.
  • Example:

    1 # The next line sets $1, $2, and $3. All other match variables are unset. 2 if (my $code =~ m/^(\d+) LET ([A-Z]+) = (.+)$/) { 3 4 my ($line_num, $var_name, $expression) = ($1, $2, $3); 5 6 # The next line obviously sets the match variables 7 # * assuming the match succeeeds 8 if ($expression =~ m/^(\d+|[A-Z]+) ([-+*\/]) (\d+|[A-Z]+)$/) { 9 10 my ($operand1, $operator, $operand2) = ($1, $2, $3); 11 12 # The next line NON-OBVIOUSLY resets the match variables, EVEN THOUGH IT CONTAINS NO PARENS 13 # * assuming the match succeeeds 14 $operand1 = $symtab{$operand1} if ($operand1 =~ m/^[A-Z]+$/); 15 # ditto 16 $operand2 = $symtab{$operand2} if ($operand2 =~ m/^[A-Z]+$/); 17 18 # The next line MIGHT reset the match variables, depending on the condition 19 # Discussion question: when will the match variables be reset and when not? 20 if ($operator eq "/" and $operand2 !~ m/^0+$/) { 21 warn "Division by zero!\n"; 22 } 23 else { 24 # The next line SNEAKILY resets match variables because--and you can't 25 # tell from here!!--the evaluate_expr() function uses a regex 26 my $expr_value = evaluate_expr($operand1, $operand2, $operator); 27 } 28 29 } 30 }

• A final note on the match variables and when they get (re)set: Remember the difference between "not used in the match" and "used in the match but matched no characters"

Interpolating into Regexes

• As regexes are just strings, they interpolate just as a double-quoted string does. Therefore, the regex defined by
     my $str = "abc";
     my $rgx = qr/123$str/;
  is /123abc/.
  • Be aware, however, that any special characters in the string interpolated are treated as special characters. All interpolation is done first and then the final string is interpreted by the regex engine.
  • Therefore, consider the following code:

    1 my $a_plus = "A+"; 2 3 my @students = ("Bill: A", "Jill: B+", "Will: N/A", "Gil: A+", "Phil: F", "Fran: A-"); 4 5 foreach my $student (@students) { 6 if ($student =~ m/$a_plus/) { 7 print "$student\n"; 8 } 9 }

    This outputs

    Bill: A Will: N/A Gil: A+ Fran: A-

  • This is (presumably) not correct. The + in the string $award_grade is interpreted as the "one or more" quantifier, not a literal plus sign.
  • Fortunately, Perl has the \Q\E operator that quotes metacharacters. Use it to surround interpolated strings that you want to be literal text in the regex.
  • Consider the following code:

    1 my $a_plus = "A+"; 2 3 my @students = ("Bill: A", "Jill: B+", "Will: N/A", "Gil: A+", "Phil: F", "Fran: A-"); 4 5 foreach my $student (@students) { 6 if ($student =~ m/\Q$a_plus\E/) { 7 print "$student\n"; 8 } 9 }

    This outputs

    Gil: A+

Building Complex Regexes

• Interpolation is largely useful because it allows you to interpolate sub-regexes defined with qr// into larger regexes. This is very useful for building very complex regexes.

  1 my $rgx_zip = qr/\d{5}(-\d{4})?/; 2 my $rgx_phone = qr/\d{3}-\d{3}-\d{4}/; 3 my $rgx_idnum = qr/[A-Z]{2}\d{5}-[A-Z]{3}/; 4 5 if ($data =~ m/^ZIP=($rgx_zip), PH=($rgx_phone), ID=($rgx_idnum)$/) { 6 print "Data is valid!\n"; 7 }

  • There are two distinct methods for creating large complex regexes from small simple building blocks.
  • Building regexes from the top down means creating one large regex that matches parts of the large string in a very general way. Then parts of this "overview" regex can be refined until the overall regex matches the string with the level of detail needed.
    • Define a complex regex simply by using undefined sub-regexes as placeholders.
    • Then define the sub-regexes, possibly with more sub-regex placeholders if the sub-regex is itself complex.
    • Continue until you have built a complex regular expression much more easily than by doing it all at once.
    • For example, a regular expression to define any possible email address would be highly complex. So instead, start with the following:

      $rgx_email = qr/^($rgx_user)@($rgx_host)$/;

    • Then define each of the two sub-regexes:

      $rgx_host = qr/(?:$rgx_dns_name|$rgx_ipaddr)/; $rgx_user = qr/(?:[a-zA-Z0-9._-]+)/; $rgx_email = qr/^($rgx_user)@($rgx_host)$/;

    • Keep going until you have defined all sub-levels:

      1 $rgx_octet = qr/(?:\d{1,2}|[01]\d{2}|2[0-4]\d|25[0-5])/; 2 $rgx_ipaddr = qr/(?:(?:$rgx_octet\.){3}$rgx_octet)/; 3 $rgx_dns_tld = qr/(?:[a-zA-Z]{2,4})/; 4 $rgx_dns_comp = qr/(?:[a-zA-Z0-9]+[a-zA-Z0-9-]*[a-zA-Z0-9]+|[a-zA-Z0-9]+)/; 5 $rgx_dns_name = qr/(?:(?:$rgx_dns_comp\.)+$rgx_dns_tld)/; 6 $rgx_host = qr/(?:$rgx_dns_name|$rgx_ipaddr)/; 7 $rgx_user = qr/(?:[a-zA-Z0-9._-]+)/; 8 $rgx_email = qr/^($rgx_user)@($rgx_host)$/;

    • In addition to how much easier it was to construct, note how much easier this is to read than if it were all combined into one monstrous regular expression:

      $rgx_email = qr/^([a-zA-Z0-9._-]+)@((?:(?:(?:[a-zA-Z0-9]+[a-zA-Z0-9-]* [a-zA-Z0-9]+|[a-zA-Z0-9]+)\.)+(?:[a-zA-Z]{2,4}))| (?:(?:(?:\d{1,2}|[01]\d{2}|2[0-4]\d|25[0-5])\.){3} (?:\d{1,2}|[01]\d{2}|2[0-4]\d|25[0-5])))$/;

  • Building regexes from the bottom up means creating small regexes that match bits of the string, then combining them into larger regexes until you have one regex that should match the entire string. In other words, you define the pieces you know you will need first, then glue them together.
    • The end result is usually similar; the distinction is in the method of construction.
    • In practice, you will often come at a complex regex from both sides at the same time.
  • There are a couple of caveats to remember when using interpolation to build complex regular expressions:
    • It is often a good idea to surround every sub-regex with (non-capturing) parentheses in order to prevent unexpected interactions between adjacent interpolations. For example, consider

      my $rgx1 = qr/a?b/; my $rgx2 = qr/c|d/; my $rgx_combined = qr/^$rgx1$rgx2$/;

      The overall regex is /^a?bc|d$/, which matches any string that starts with "abc" or "bc" or ends with "d".
      • However, the author probably meant to match "abc", "abd", "bc", or "bd".
      • The regex to do that would be /^(a?b)(c|d)$/
      • Which would have been realized had the sub-regexes been enclosed in (capturing or clustering) parentheses.
    • Remember that positional anchors (beginning and ending of string) are interpreted in the context of the overall regex. Placing positional anchors in sub-regexes will not normally work.
    • Capturing parentheses are numbered left to right in the overall interpolated regex. It is often a good idea to avoid capturing parentheses in sub-regexes that are designed to be interpolated (non-capturing parentheses are fine).

Constructing Regexes Programmatically

• Other string operations such as join and concatenation work on regexes as well.
• This allows you to create very long and cumbersome regexes in code using some data structure as a base (perhaps even using user-supplied data).
• Here is a simple example:

  1 my @months = qw(January February March April May June July 2 August September October November December); 3 4 my $month_abbrs = join "|", map { uc substr $_, 0, 3 } @months; 5 6 my $rgx_date = qr/^([0-9]{4})-($month_abbrs)-([0-9]{2})$/i; 7 8 print "Regex to match dates is /$rgx_date/\n";

  which outputs

  Regex to match dates is /(?i-xsm:^([0-9]{4})-(JAN|FEB|MAR|APR|MAY|JUN|JUL|AUG|SEP|OCT|NOV|DEC)-([0-9]{2})$)/

• Ignore the "(?i-xsm: ... )" for the moment. No worries, we'll come back to it.
• Remember that regexes can be passed to and returned from subroutines.
• Discussion question: what's wrong with the above date-matching regex?

Evaluated Replacements

• s/// supports "evaluated replacement". If the /e modifier is used on the pattern, then the replacement will be executed as Perl code and the string returned will be substituted. For example:

  1 my $rgx_1st_word = qr/^(\w+)\b/; 2 my $sentence = "Call me Ishmael"; 3 4 print "string before modification:\n"; 5 print "$sentence\n\n"; 6 7 my $mod_sentence_without = $sentence; 8 $mod_sentence_without =~ s/$rgx_1st_word/uc $1/; 9 print "without the e modifier:\n"; 10 print "$mod_sentence_without\n\n"; 11 12 my $mod_sentence_with = $sentence; 13 $mod_sentence_with =~ s/$rgx_1st_word/uc $1/e; 14 print "with the e modifier:\n"; 15 print "$mod_sentence_with\n";

  outputs:

  string before modification: Call me Ishmael without the e modifier: uc Call me Ishmael with the e modifier: CALL me Ishmael

Extended Regexes

• By using the /x modifier, you can insert whitespace and comments.
• This makes regular expressions much easier to use.
• For example, consider the regular expression

  $rgx_int = qr/^([-+])?(\d+)(?:[eE](\d{1,3}))?$/;

• Using extended regexes, this regex can be written as

  1 $rgx_int = qr/ 2 ^ # beginning of string 3 4 ( # begin capture $1: sign 5 [-+] # plus or minus 6 ) # end capture $1 7 ? # sign is optional 8 9 ( # capture $2: coefficient 10 \d+ # one or more digits 11 ) # end capture $2 12 13 (?: # group 14 [eE] # literal e or E 15 ( # capture $3: exponent 16 \d{1,3} # one to three digits 17 ) # end capture $3 18 ) 19 ? # exponent is optional 20 21 $ # end of string 22 /x";

• These regular expressions are equivalent, but the second is easier to comprehend.
  • You almost always want to use qr// and store a regex in a variable when using /x; it is very cumbersome (though allowed) to use extended regexes in-place.
• To include literal spaces or hash characters in an extended regex, escape them or include them in a character class.

Greed

• The repetition operators *, +, and ? are greedy.
  • This means they match as much of the string as possible. Example:

    1 my $html_line = 'Here is a <B>bold</B> thing and an <I>italic</I> thing'; 2 print "HTML is: \"$html_line\"\n"; 3 print "(Trying to strip HTML tags)\n"; 4 $html_line =~ s/<.+>//g; 5 print "TEXT is: \"$html_line\"\n";

    outputs:

    HTML is: "Here is a <B>bold</B> thing and an <I>italic</I> thing" (Trying to strip HTML tags) TEXT is: "Here is a thing"

  • Oh no! What happened?
    • The pattern
         /<.+>/
      is three sub-patterns:
      • subpat1: (<) Match a literal "<"
      • subpat2: (.+) Match one or more characters
      • subpat3: (>) Match a literal ">"
    • On the first pass, subpat1 matches the "<" in "<B>"
    • The pattern match continues from that point with subpat2, which is ".+". This is intended to match everything inside the HTML tag (the "B", in the first case, the "/B" in the second, but instead matches as much as possible, including the ">" ending the "<B>" tag, the entire "</B>" tag, the "<I>" tag and the "</I" towards the end, plus all the text in between.
    • Then subpat3 tries to match. It succeeds because there is another ">" after "</I".
    • All three sub-patterns have matched so the overall pattern is a match.
      • However, the text actually matched within the string was: "<B>bold</B> thing and an <I>italic</I>".
      • Therefore when the substitution is done, much more of the string is removed than was intended.
• To avoid greed, use the +?, *?, and ?? minimal match operators instead. They still match one or more repetitions, or zero or more repetitions, but they gobble up the minimum necessary for the pattern to match:

  1 my $html_line = 'Here is a <B>bold</B> thing and an <I>italic</I> thing'; 2 print "HTML is: \"$html_line\"\n"; 3 print "(Trying to strip HTML tags)\n"; 4 $html_line =~ s/<.+?>//g; 5 print "TEXT is: \"$html_line\"\n";

  outputs:

  HTML is: "Here is a <B>bold</B> thing and an <I>italic</I> thing" (Trying to strip HTML tags) TEXT is: "Here is a bold thing and an italic thing"

  • That's what we intended.
  • Another approach is to explicitly exclude the "closing" from the match with a pattern like
       /<[^>]+>/
    This pattern has three sub-patterns like the previous one, but with one critical difference:
    • subpat1: (<) Match a literal "<"
    • subpat2: ([^>]+) Match one or more characters, other than a literal ">"
    • subpat3: (>) Match a literal ">"
  • The use of non-greedy quantifier or an approach like this one is largely a matter of taste, but it does occasionally matter what the middle part consists of.
• Greed is important when thinking about the "longest leftmost" rule.

Transliteration

• Transliteration is not a regex operation, but because it looks like it is, it's always discussed when regexes are.
  • Carrying on the tradition here!
• A transliteration operation simply searches for instances of a list of characters in a string and replaces them with the corresponding character in a second list, using syntax like "$string =~ tr/SRCH/REPL/;".
• For example:

  my $string = "This is some text."; print "Original String: \"$string\"\n"; $string =~ tr/aeiou/AEIOU/; print "Modified String: \"$string\"\n";

  outputs

  Original String: "This is some text." Modified String: "ThIs Is sOmE tExt."

• Like s///, the tr/// operation is destructive.
• Note that if the replacement set has less characters in it than the search set, the last character in the replacement set is replicated as necessary.
  • If the replacement set only contains one character, it will have the effect of replacing all elements in the search set with that character:

    my $string = "This is some text."; print "Original String: \"$string\"\n"; $string =~ tr/aeiou/!/; print "Modified String: \"$string\"\n";

    outputs

    Original String: "This is some text." Modified String: "Th!s !s s!m! t!xt."

    • This is more clearly accomplished with s/[aeiou]/!/g.
• The tr/// construct also accepts modifiers, the most useful of which are:
  • /c means the search set is the complement of the given characters, meaning is it is every other character. This is most useful with a one-character replacement set:

    my $string = "This is some text."; print "Original String: \"$string\"\n"; $string =~ tr/aeiou/?/c; print "Modified String: \"$string\"\n";

    outputs

    Original String: "This is some text." Modified String: "??i??i???o?e??e???"

    • This is more clearly accomplished with s/[^aeiou]/?/g.
  • /s "squashes" runs of replaceable characters with one replacement instead of replacing them one-for-one:

    my $string = "This string has some weird spacing. "; print "Original String: \"$string\"\n"; $string =~ tr/ / /s; # collapse strings of spaces print "Modified String: \"$string\"\n";

    outputs

    Original String: "This string has some weird spacing. " Modified String: "This string has some weird spacing. "

    • This is more easily accomplished with s/ +/ /g;
  • /d deletes characters in the search set but not in the replacement set:

    my $string = "This is some text."; print "Original String: \"$string\"\n"; $string =~ tr/aeioubcdfghjklmnpqrstvwxyz/AEIOU/d; print "Modified String: \"$string\"\n";

    outputs

    Original String: "This is some text." Modified String: "TI I OE E."

    • If the replacement set is empty, this has the effect of deleting all characters in the search set, but this is more clearly accomplished (are you sensing a theme?) with something like s/[aeiou]//g.
• tr returns the number of substitutions made, so it provides an easy way to count the occurrence of a specific character or set of characters, if given a replacement set that is the same as the search set:

  my $string = "This is some text."; my $count = ($string =~ tr/aeiou/aeiou/); print "String \"$string\" contains $count vowels.\n";

  outputs

  String "This is some text." contains 5 vowels.

  • This is easily accomplished with s/([aeiou])/$1/g;
• You can also do something like
     $count = grep { m/[aeiou]/ } split //, $string;
  which has the additional benefit of being non-destructive, so (unlike tr/// or s///) it can be used on constant and literal strings.
• If you need a non-destructive tr/// you can use it en-passant, as with s///.
• For reasons of historical reverence, y/// is a synonym for tr///; the two are identical in every respect.
  • Stick with tr///.
  • Better yet, stick with s///, which can almost always do what you need.
    • I have been a professional Perl programmer since 1999. I have used tr/// exactly zero times.
    • But maybe you will find it useful. Truth is stranger than fiction.

Summary of Pattern Modifiers

Modifier	Effect	m//	s///	qr//	Bad Mnemonic?
/i	match case-Insensitively	yes	yes	yes	no
/s	include \n in the dot class (force string to be handled as a Single line)	yes	yes	yes	yes
/m	^ and $ can match before or after an ebmedded newline (force string to be handled as Multiple lines)	yes	yes	yes	yes
/x	eXtended regex (can include comments and whitespace)	yes	yes	yes	no
/o	only compile pattern Once	yes	yes	–	no
/g	match/substitute Globally	yes	yes	–	no
/c	Continue after a failed global match	with /g	–	–	no
/e	Evaluate replacement as expression	–	yes	–	no

• Remember when we printed out a regex that had been stored with qr// and it looked like
     /(?i-xsm:^([0-9]{4})-(JAN|FEB|MAR|APR|MAY|JUN|JUL|AUG|SEP|OCT|NOV|DEC)-([0-9]{2})$)/
  • The "(?i-xsm: ... )" indicates that for that part of the pattern, the /i modifier is in effect, and the /x, /s, and /m modifiers are not in effect.
  • These are known as cloistered pattern modifiers and it's possible for different parts of the regex to have different modifiers, either by specifiying them literally using this construction, or by interpolating patterns stored with qr// and given different options where they are defined.
  • Whoa, look at the time!

Tips

• Remember the rules for matching:
  • Patterns always try to match the LONGEST LEFTMOST substring.
  • When a pattern is matched against a string, the string must match all sub-patterns IN-ORDER, ADJACENTLY.
• Develop regular expressions one piece at a time, such as from the inside to the outside, and break them up if needed.
  • Use interpolation to make regular expressions easier to understand.
  • Use a top-down or bottom-up method to develop complex regexes.
• If you use match variables, remember that all regular parentheses in a regex memorize things.
  • This is particularly important to remember when interpolating regexes into one another.
• Remember that that *, +, and ? quantifiers are greedy.
• Remember that word boundaries and other anchors match conditions, not characters.
• Don't reinvent the wheel - if you need to use a regex for URLs, email addresses, U.S. Postal addresses, or anything else common but complicated, search the Web for a solution before you develop your own.
• Keep regexes as simple as possible.
• Keep regexes single-functional.
• Give clear names to regular expressions that are stored in variables.
  • Generally they should start with the $rgx prefix or something similar.
  • Give them names based on what they are supposed to do, not how they are supposed to do it.
    • Call it $rgx_zipcode, not $rgx_5digits

When not to use Regular Expressions

• Regexes are not the best solution for every problem involving interpreting a string.
  • They are very powerful, but they are often more difficult to use than some alternatives.
  • The regex engine is highly optimized, but Perl's built-in string functions (like substr and index) are even more highly optimized, so regexes, while very efficient, are not always more efficient than combinations of string functions.
• Therefore, don't use regexes to:
  • Determine if a string is exactly the same as another string (use eq).
  • Determine if a string is exactly the same as another string, disregarding case (use eq with uc).
  • Determine if a string contains a given substring (use index, with uc for case-insensitive search).
  • Determine if a string has a given prefix or suffix (use index/rindex or eq with substr, again perhaps with uc).
  • Chop up a string by length (use substr or unpack)
• In general, if the problem can be solved with one or more string functions, it will probably be faster and easier to do so.
  • But don't use some tortured combination of multiple string functions when a simple regex will work.
  • For example, if you want to find the substring of a string bounded by angle-brackets, you could say
       $start = index $string, "<";
       $stop = index $string, ">", $start;
       $substring = substr $string, $start + 1, $stop - $start - 1;
    But you would be better off saying:
       $substring = ($string =~ m/<(.*?)>/)[0];
    • Although the regex solution looks more complicated at first blush, once you understand regexes it's much simpler.
  • Example 2: suppose you want to determine whether the first three characters of a string after the first space are "abc" (case-insensitive)*. The string-functions version:

    if ("abc" eq lc substr($string, 1+index($string, " "), 3)) { ...

    is probably faster than the functionally equivalent regex version:

    if ($string =~ m/^[^ ]* ABC/i) { ...

    but the regex version is easier to understand (again, once you are familiar with regexes).
    * WHY you would want to do this is beyond the scope of this class...
  • In general, if you need two or more lines, or especially a loop or conditional to parse a string, use a regex.

Good Grief, Make It Stop!

• This information on regular expressions—detailed may it be—is only the beginning. There are many more elements in the regex syntax that we have not covered here:
  • Non-destructive substitution and transliteration (since Perl 5.14)
  • Matches and character classes based on Unicode
  • Additional anchors (\G, \A, \Z, and \z)
  • Recursive regexes
  • Cloistered pattern modifiers
  • Possessive quantifiers
  • Relative backreferences
  • Lookahead and lookbehind assertions
    • Positive and negative
  • Non-backtracking sub-patterns
  • Match-time code evaluation and pattern interpolation
    • Evaluated code to generate sub-patterns
    • Recursive patterns
    • Conditional interpolation
  • Tracing and optimizing the regex engine
  • And more!

Answers to discussion questions:

• What characters are in the class [^\W_]?
  • This class contains everything except "non-word characters and underscores". In otherwords, it contains word characters except underscores
  • So it's equivalent to [A-Za-z0-9]
• What character(s) do the following classes match?
  • [^-] contains everything but a dash
  • [-^] contains a dash and a caret
  • [^-^] contains everything except a dash and a caret
  • [--^] contains characters from a dash to a caret (includes digits, uppercase letters, and more punctuation)
  • [^--^] contains everything except characters from a dash to a caret (including lowercase letters, and some punctuation, and whitespace characters)
  • [^^] contains everything but a caret
  • Please don't use any of these!
• When will the match variables be reset (and when not) when the conditional expression
     $operator eq "/" and $operand2 !~ m/^0+$/
  is evaluated?
  • This expression is a candidate for short-circuit Boolean evaluation: if the first part ($operator eq "/") is true, the second part (the regex match) will be evaluated and the match variables will be reset on a successful match. If the first part is false, the expression is already false and the second part will not be evaluated; hence the pattern match variables will not be reset, even if the match would have succeeded.
• What's wrong with the above date-matching regex:
     /(?i-xsm:^([0-9]{4})-(JAN|FEB|MAR|APR|MAY|JUN|JUL|AUG|SEP|OCT|NOV|DEC)-([0-9]{2})$)/
  • Presumably the case-sensitivity is not what's wrong. The regex was specifically constructed to be case-sensitive and to match uppercase date abbreviations only.
  • The regex is intended to match strings like "2009-JAN-12", but look carefully at the year and day-of-month parts. They just match strings of 4 and 2 digits, respectively.
    • So strings like "0000-JAN-99" will match.
    • This is almost certainly not what was intended.
    • It turns out that a regex to match only valid dates is hilariouly terrible, even if you don't worry about leap years.
      • Pop quiz: was 2000 a leap year?
    • Discussion question: what's a better way to write a date validator?
• What's a better way to write a date validator?
  • Why does this question sound so familiar?
  • There are several modules on CPAN to do this.
  • A "roll-your-own" approach would be to use a "naïve" regex like the one above, and then pass the (appropriately massaged) submatches to the timelocal function (in the Time::Local core library) to see if it dies, in which case it's an invalid date.