30.regular-expressions-in-practice.md

Applications of Regular Expressions

Regular Expression Basics

When writing programs for processing strings, we often meet the need to find strings that match some rules in a piece of text. Regular expression is the tool used to describe these rules. In other words, we can use regular expressions to define the matching pattern of strings, that is, how to check whether a string has a part matching some pattern, or how to extract the part matching the pattern from a string, or replace it.

To give a simple example, if you have ever used wildcards such as * and ? when searching for files in Windows, then regular expressions are a similar kind of text-matching tool. The difference is that regular expressions are far more powerful and can describe your needs much more precisely. Of course, the tradeoff is that writing a regular expression is more complicated than using wildcards.

As another example, suppose we obtain a string from somewhere, perhaps a text file or a news article on the web, and we want to find mobile phone numbers and landline numbers in it. Of course, we could decide that mobile phone numbers are 11 digits long, bearing in mind that not every random 11-digit number is a valid mobile number, and that landlines follow a pattern such as area-code-number. But completing such a task without regular expressions would be rather troublesome. Computers were originally created to perform mathematical calculations, so the information they processed was mostly numeric. Today, however, much of the information we deal with in daily work is text data, and we want computers to recognize and process text that follows certain patterns. That is exactly why regular expressions are so important. Almost every programming language today supports them, and Python provides support through the standard-library re module.

For related knowledge about regular expressions, everyone can read a very famous blog post called Regular Expression 30-Minute Tutorial. After reading this article, you can understand the table below. It is our brief summary of some basic symbols in regular expressions.

Symbol	Meaning	Example	Note
.	Match any character	b.t	Matches bat, but, b#t, b1t, and so on
\w	Match letters, digits, or underscore	b\wt	Matches bat, b1t, b_t, but not b#t
\s	Match whitespace, including \r, \n, \t, and so on	love\syou	Can match love you
\d	Match digits	\d\d	Can match 01, 23, 99, and so on
\b	Match a word boundary	\bThe\b
^	Match the start of a string	^The	Matches strings that begin with The
$	Match the end of a string	.exe$	Matches strings that end with .exe
\W	Match a non-word character	b\Wt	Matches b#t, b@t, but not but, b1t, b_t
\S	Match a non-whitespace character	love\Syou	Matches love#you, but not love you
\D	Match a non-digit	\d\D	Matches 9a, 3#, 0F, and so on
\B	Match a non-word boundary	\Bio\B
[]	Match any one character from a character set	[aeiou]	Matches any vowel
[^]	Match any one character not in the character set	[^aeiou]	Matches any non-vowel
*	Match zero or more times	\w*
+	Match one or more times	\w+
?	Match zero or one time	\w?
{N}	Match exactly N times	\w{3}
{M,}	Match at least M times	\w{3,}
{M,N}	Match at least M and at most N times	\w{3,6}
|	Alternation	foo|bar	Can match foo or bar
(?#)	Comment
(exp)	Match exp and capture it into an automatically named group
(?<name>exp)	Match exp and capture it into a group named name
(?:exp)	Match exp without capturing it
(?=exp)	Match the position before exp	\b\w+(?=ing)	Matches danc in I'm dancing
(?<=exp)	Match the position after exp	(?<=\bdanc)\w+\b	Matches the first ing in I love dancing and reading
(?!exp)	Match a position not followed by exp
(?<!exp)	Match a position not preceded by exp
*?	Repeat any number of times, but as few as possible	a.*b / a.*?b	On aabab, the former matches the whole string, the latter matches aab and ab
+?	Repeat one or more times, but as few as possible
??	Repeat zero or one time, but as few as possible
{M,N}?	Repeat M to N times, but as few as possible
{M,}?	Repeat at least M times, but as few as possible

Note: If the character you want to match is itself a special character in regular expressions, you can escape it with \. For example, if you want to match a literal period, write \.. Writing . directly would match any character. Similarly, to match parentheses literally, you must write \( and \), otherwise they will be treated as grouping syntax.

Python Support for Regular Expressions

Python provides the re module to support operations related to regular expressions. The core functions in re are listed below.

Function	Description
compile(pattern, flags=0)	Compile a regular expression and return a pattern object
match(pattern, string, flags=0)	Match a string with a regular expression; return a match object on success, otherwise None
search(pattern, string, flags=0)	Search for the first occurrence matching the regular expression; return a match object on success, otherwise None
split(pattern, string, maxsplit=0, flags=0)	Split a string using the regex as the separator and return a list
sub(pattern, repl, string, count=0, flags=0)	Replace content matching the regex with the specified string; count controls the number of replacements
fullmatch(pattern, string, flags=0)	A full-string version of match, requiring the whole string to match
findall(pattern, string, flags=0)	Find all matches and return them as a list of strings
finditer(pattern, string, flags=0)	Find all matches and return an iterator
purge()	Clear the cache of implicitly compiled regular expressions
re.I / re.IGNORECASE	Ignore-case matching flag
re.M / re.MULTILINE	Multiline matching flag

Note: The re module functions listed above can also be replaced in actual development with methods on a compiled regular-expression object, a Pattern object. If a regular expression needs to be reused, compiling it first with compile and then using the resulting pattern object is clearly the wiser choice.

The examples below show how to use regular expressions in Python.

Example 1: Validate a username and a QQ number and give corresponding prompts

"""
Requirement: the username must consist of letters, digits, or underscores and be 6 to 20 characters long.
The QQ number must be 5 to 12 digits long and cannot start with 0.
"""
import re

username = input('请输入用户名: ')
qq = input('请输入QQ号: ')
# The first argument to match is the regex string or regex object.
# The second argument is the string to be matched.
m1 = re.match(r'^[0-9a-zA-Z_]{6,20}$', username)
if not m1:
    print('请输入有效的用户名.')
# fullmatch requires the entire string to match the regex,
# so start and end anchors are unnecessary here.
m2 = re.fullmatch(r'[1-9]\d{4,11}', qq)
if not m2:
    print('请输入有效的QQ号.')
if m1 and m2:
    print('你输入的信息是有效的!')

Tip: The regular expressions above are written as raw strings, meaning a string literal prefixed with r. A raw string means every character in the string keeps its original meaning, with no escape processing. Since regular expressions contain many metacharacters and many backslashes, using raw strings avoids having to write \\ everywhere. For example, \d would otherwise have to be written as \\d, which is inconvenient to write and hard to read.

Example 2: Extract domestic mobile phone numbers from a piece of text

The picture below shows the mobile phone number segments launched by the three domestic carriers by the end of 2017.

import re

# Create a regex object. Lookbehind and lookahead are used to make sure digits do not continue on either side.
pattern = re.compile(r'(?<=\D)1[34578]\d{9}(?=\D)')
sentence = '''重要的事情说8130123456789遍，我的手机号是13512346789这个靓号，
不是15600998765，也不是110或119，王大锤的手机号才是15600998765。'''
# Method 1: find all matches and store them in a list
tels_list = re.findall(pattern, sentence)
for tel in tels_list:
    print(tel)
print('--------华丽的分隔线--------')

# Method 2: get match objects through an iterator
for temp in pattern.finditer(sentence):
    print(temp.group())
print('--------华丽的分隔线--------')

# Method 3: repeatedly call search with a starting position
m = pattern.search(sentence)
while m:
    print(m.group())
    m = pattern.search(sentence, m.end())

Note: The regular expression above for domestic Chinese mobile numbers is not ideal, because numbers starting with 14 are only valid for certain prefixes such as 145 or 147, and that is not considered above. A better regex for domestic Chinese mobile numbers would be (?<=\D)(1[38]\d{9}|14[57]\d{8}|15[0-35-9]\d{8}|17[678]\d{8})(?=\D). China also seems to have numbers starting with 19 and 16 now, but we will not consider those here for the moment.

Example 3: Replace undesirable content in a string

import re

sentence = 'Oh, shit! 你是傻逼吗? Fuck you.'
purified = re.sub('fuck|shit|[傻煞沙][比笔逼叉缺吊碉雕]',
                  '*', sentence, flags=re.IGNORECASE)
print(purified)  # Oh, *! 你是*吗? * you.

Note: The regex-related functions in the re module all have a flags parameter. It represents regex matching flags, which can be used to control whether matching ignores case, whether multiline matching is enabled, whether debug information is shown, and so on. If you need to specify multiple values for flags, you can combine them with the bitwise OR operator, such as flags=re.I | re.M.

Example 4: Split a long string

import re

poem = '窗前明月光，疑是地上霜。举头望明月，低头思故乡。'
sentences_list = re.split(r'[，。]', poem)
sentences_list = [sentence for sentence in sentences_list if sentence]
for sentence in sentences_list:
    print(sentence)

Summary

Regular expressions are really very powerful in string processing and matching. Through the examples above, I believe everyone has already felt the charm of regular expressions. Of course, writing a regular expression is not so easy for beginners, but many things become easy after you get familiar with them. Just try boldly. There is an online regular expression testing tool that I believe can help everyone to some degree.