The Silver Lining of 404 Errors

The Double-Edged Sword of Custom 404 Pages

In a previous discussion, we explored the merits of implementing a custom 404 error page on your website, along with a step-by-step guide on how to create one. These pages can be a boon, helping you salvage traffic that might otherwise be lost. However, they do come with a downside that can make site maintenance more challenging.

The issue lies in the fact that when a server encounters a 404 error, it typically logs the details of the event. For instance, if your log shows multiple 404 errors due to a missing "abutme.html" page, you can infer that you've mistakenly linked to "abutme.html" instead of the correct "aboutme.html". This functionality of log files is invaluable in identifying and rectifying such errors, keeping your site relatively free of broken links.

However, if you've replaced the standard 404 error with your own HTML 404 error page, this useful feature becomes inaccessible. You'll still see that a 404 error has occurred, but you won't receive any further details that could help you pinpoint the exact issue. This makes finding and fixing these errors nearly impossible, leading to an accumulation of errors over time and potentially tarnishing the professional image of your site.

There's Always a Solution

Despite the challenges outlined above, don't despair. Every problem has a solution, and this one is no exception. If you want to maintain your custom 404 page and still receive notifications when you create a broken link, there's a way to do it. However, to implement this fix, your host needs to be running Apache with support for .htaccess files, Server Side Includes (SSI), and CGI. You can contact your technical support to verify whether you have access to these features.

Getting Started with the CGI Script

The first step is to acquire a CGI script that will log the errors and notify you about them. There are several options available, but a personal favorite is Matrix Vault's free 404 Helper, which can be downloaded here. Once you've downloaded the source code, open it in a text editor. You can find free editors online, or you can use the MS-DOS Edit that comes with most versions of Windows.

Before you start editing the file, you need to know where your host has installed the Perl interpreter and Sendmail. Check if the paths used in the CGI script match those used by your host. The Perl interpreter's location is set to /usr/bin/perl in the first row, and the location of Sendmail is set to /usr/lib/sendmail in the 21st row. Make changes as necessary.

After ensuring that the paths are correct, modify the rest of the script to suit your needs. Replace the email address in the $email field with the one you want the error report to be sent to. You might also want to use a smaller value in the $mailon field than the default of 10, as it can take a while for a small site to generate enough 404 errors to fill up a 10K log. A value of 1 or 2 is recommended initially, which can be increased later if you find that you're receiving error reports too frequently.

Once you're done with the CGI script, save it as "404helper.cgi". However, there's still more work to be done.

Editing Your Custom 404 Page and .htaccess File

Having the script alone isn't enough. It needs to be executed when an error is encountered, which is where the SSI's come into play. Open your 404 error page in a text editor and add the following line:

The best place for this line is at the bottom of your 404 error page, but before the tag. After everything is in place, save the file with a ".shtml" extension instead of the usual ".htm" or ".html". This is crucial, as the SSI tag might not work if you don't use the correct extension.

Next, you'll need to edit the .htaccess file you created when you built your custom 404 page. If it currently only contains "ErrorDocument 404 http://www.yourdomain.com/404page.html", modify it to include the following:

Options Includes ExecCGI AddType application/x-httpd-cgi .cgi .pl AddType text/html .shtml AddHandler server-parsed .shtml ErrorDocument 404 http://www.yourdomain.com/404page.shtml

These new lines will enable Server Side Includes and CGI so that your script will work. Don't forget to change the ErrorDocument 404 line to point to the new .shtml page instead of your old .html version. Save your .htaccess file once you're done.

Upload, Set Permissions, and Launch!

Connect to your host with an FTP program and upload the .shtml version of your 404 page and your new .htaccess file into your root directory. Then, go to the directory you've reserved for CGI programs and upload the 404helper.cgi file there. Make sure to upload in ASCII, not Binary mode.

The final step is to set permissions so that the CGI program can be executed. You'll need to CHMOD 755 both the CGI file and the directory it's in. The steps to do this depend on the software you're using, but instructions for WS_FTP, a popular Windows FTP program, can be found here.

With that, you're done! You can now enjoy the luxury of receiving an email report on all 404 errors, allowing you to quickly fix any broken links and improve the quality of your site. Congratulations!