Python re.compile: How to Speed Up Regex in a Loop

The slow version

import re

def parse_lines(lines):
    out = []
    for line in lines:
        m = re.match(r'^(\S+)\s+(\w+)\s+(.*)$', line)   # runs for every line
        if m:
            out.append({'ts': m.group(1), 'level': m.group(2), 'msg': m.group(3)})
    return out

re.match(pattern, string) is a convenience function. On each call it takes the pattern string, finds the compiled pattern (Python keeps a small internal cache of recently used patterns, so it does not always compile from scratch), and then runs the match. For one call this cost does not matter. For millions of log lines, the repeated lookup and extra function calls add up - and if your program uses many different patterns, they can fall out of the cache and get compiled again.

The fast version

import re

# Compile once, when the module is imported
LINE_RE = re.compile(r'^(\S+)\s+(\w+)\s+(.*)$')

def parse_lines(lines):
    out = []
    for line in lines:
        m = LINE_RE.match(line)          # use the compiled object
        if m:
            out.append({'ts': m.group(1), 'level': m.group(2), 'msg': m.group(3)})
    return out

The pattern never changes between loop passes, so there is no reason to handle it inside the loop. This is called hoisting an invariant out of the loop.

Measure it - do not guess

Time the function on a realistic number of lines before and after the change:

python3 -c "
import time
from parser import parse_lines
lines = ['2026-05-25T12:34:56Z INFO user logged in'] * 200_000
t = time.perf_counter()
parse_lines(lines)
print(f'{(time.perf_counter() - t) * 1000:.0f} ms')
"

For a more careful number, use timeit:

python3 -m timeit -s "from parser import parse_lines; L=['2026-05-25T12:34:56Z INFO x']*10000" "parse_lines(L)"

Also check the output is the same. A faster function that returns different results is a bug:

assert old_parse(sample) == parse_lines(sample)

Find hot loops with a profiler

On a real codebase you may not know which loop is slow. Profile it:

python3 -m cProfile -s cumtime parser.py | head -20

If re.match, re._compile, or re.compile appears near the top with a very high call count, a pattern is being handled inside a loop. py-spy top --pid <pid> does the same for a running process without stopping it.

More regex speed tips

• Anchor the pattern with ^ when the match must start at the beginning of the line. It lets the engine fail fast on lines that do not match.
• Use str methods first when they are enough. line.startswith('ERROR') or 'timeout' in line is faster than a regex.
• Avoid nested quantifiers like (a+)+. They can cause very slow backtracking on some inputs.
• Use named groups ((?P<level>\w+)) for readability. They do not slow matching.

The same pattern beyond regex

The rule is general: if a value is the same on every loop pass, create it once outside the loop. Other common examples:

• Reuse one requests.Session() or httpx.Client() instead of opening a new connection per request.
• Build lookup dicts and sets once, not inside the loop.
• Read config once, not on every iteration.

Read the loop body and ask: "does this need to happen on every pass?"