The Treacherous Power of Extended Regular Expressions

I wanted to filter out lines containing the word "line" or a double quote from a 1GB file. This can be easily specified as an extended regular expression, but it turns out that I got more than I bargained for.

What I wanted to do was to see all the unique error messages in a huge SQL error log. The log contained lines like

"INSERT INTO FILECOPIES VALUES(2,5127)"
Integrity constraint violation - no parent SYS_FK_123 table: FILES
SQL Error at 'stdin' line 15276525:

egrep -v '"|line' sql.err |
sort -u |
more

I reasoned that the problem was the slowness introduced by the | regular expression operator. This is typically implemented using a nondeterministic finite state machine, which uses inefficient backtracking and can even exhibit exponential running times on certain expressions. I therefore rewrote the pipeline using two fgrep invocations:

fgrep -v line sql.err |
fgrep -v \" |
sort -u |
more

Comments   Toot! Tweet Share