Gettin' Greppy Wit It

Jeff Atwood -

We're currently supporting a third party application that, in addition to producing some truly impressive WTFs, generates incredibly verbose log files with zillions of 'error messages' that aren't really errors. This makes diagnosing problems in their server code* very difficult.

It is, however, a great use for a GREP tool-- basically, regular expressions applied to the filesystem. Let's filter this set of log files to the real errors:

This is PowerGREP, from the same guy who wrote RegexBuddy. I haven't used any GUI GREP tools before, so I can't say how PowerGREP compares, but it's certainly of the same high quality I've come to expect from using RegexBuddy (which also has, not coincidentally, a mini-GREP mode in the latest version).

GREP is originally a UNIX command, so of course there's a command line GREP. There are a million command line params, which is expected. Let me save you some time. First, use the --V command to make sure you're using GREP version 2.5.1 or later. Here's the syntax you'll likely want if you're used to standard (Perl-style) .NET regular expressions :

Show only the actual text that matches this pattern, case insensitive, against a single file. This is great for summarizing stuff: 

grep -P -i -o "d{2}:d{2}:d{2}|>[^<>]+" xyz.txt

Attempt to match the pattern contained in the file myregex.txt to every file in the current folder. If a match is found, output the entire line that matches including the line number. You end up with a nice little file of matches and line numbers (note that filename will be included if the matches span more than one file, although this can be disabled). 

grep -P -i -n --file myregex.txt *.* >output.txt

When you start typing complicated regular expressions at the command line, beware truly mind-melting command line escaping rules! And that's on top of the regex escaping rules. I found it easier to put complex regexes in a file and use the --file command, or wrap the whole thing in a tiny .bat file and use CALL to execute it.

After I developed the regex to eliminate all the spurious errors (as pictured), I set up a batch file that:

  1. command-line greps that regex on the log folder to a temp file
  2. grep the temp file using a simpler regex, filtering out just the times and messages without all the surrounding pseudo-XML cruft
  3. use blat to email the file output to me
  4. placed batch file in a scheduled task

Using only three lines in a batch file (GREP, GREP, BLAT), I get a list of relevant errors generated by this system every morning. It's enough to make you almost appreciate UNIX.

Almost.

* It's sort of assumed that third party vendors are writing code of higher quality than we do internally. That is, after all, why they're in the software business, and we're not. If using third party software instills a strong desire to rewrite the application, that's a damning indictment. If I wanted slow, buggy, and crash-prone, I would have written it myself.

This app jumped the shark when I forwarded RFC821 to their engineering department. I diagnosed a "we're not getting email" ticket with the sniffer, and saw their app wasn't sending the domain name with the HELO command. Sendmail was (rightly) rejecting them. Way to adhere to the very first command in a standard written in 1982, people!