Today’s lesson: It pays to be in the habit of checking your log files.
I make it a habit to check log files once a week. I set an alarm for every Sunday morning to remind me to check my log files.
Geeky, I know. Maybe it would be less geeky on Monday morning. But whenever you do it, the important thing is to make a habit of it.
One Month, Tops
I first took up the habit after an overhaul of my site. I knew there would be some bugs, and I thought after a month of Sundays, I would have my PHP errors log down to a very slow trickle.
How wrong I was.
It seems as fast as I can fix bugs, I can introduce new ones. That’s not to say I’m a terrible coder. Just that fixing one bug often reveals another. And since I’m always tinkering with my various sites, introducing a new feature often introduces new bugs.
A Thousand Uses
A recent example: I created a sitemap for one of my big sites. It turns out I forgot to test a key component of the sitemap generator, and my original sitemap contained incorrect links. Not being a consumer of my own XML sitemap, I never would have spotted my mistake, except that my log file was showing me lots of requests for URLs that were badly formed. Tracing them back to my own sitemap was embarrassing, but very useful. I’m still finding bad URLs even after 3 weeks. Apparently some web search engines cached my sitemap for a long time.
Checking the log files is also a good way to see what the hackers are trying to do. Often I’ll see badly-formed URLs in my error logs where someone is trying to replace a valid query string value with ?var=http://malicious.example.com/. I also see people looking for the URL SITE_NAME_cross_links, apparently a common page on other web sites. I usually try their URLs on my own to see whether they got any interesting information. So far, so good.
The other reason the PHP error logs still haven’t shrunk to a trickle is that I’ve learned to write debug info to the log file. If I can’t figure out why some framework file is misbehaving, I can at least add some more descriptive logging for next time. Want to know whether it’s a bad link on some obscure page instead of a hacker? Write the PHP referrer to the log file. Want to know what variable was passed instead of what your framework was expecting? Write the variable to the log file. Want to know what the query string was when the page broke? Write the query string to the log file.
Yes, my original goal was to eventually get the log file down to the slowest trickle of errors and warnings possible. But experience has made me change my weekly goal, which is to fix one thing (or more) per week. If I can’t fix something after 20 or 30 minutes, I add some more logging info and call it good until the following Sunday. Incremental improvement is the name of the game.
The important thing is not to fix everything – you’ll never achieve it because people will find new ways to hack, link to, or mis-type your URLs.
The important thing is to be in the habit of looking at log files, and to make slow, small, but steady improvements over time. And to be clued in to how your site is working.