A website bites the hand of its creator

Ah the joys of development. You spend hours crafting crafting code into a functional website, you run as many tests as possible to iron out those frustrating bugs, finally you “go live” and release your creation into the world.

That’s when it really starts to test your patience. New bugs, Gotchas, missing links, misbehaving dynamic content you name it, it will happen. Then when you think you’ve fixed all those, something else will come along to kick you.

Today’s rant from me is because of a site that I thought was running perfectly. Orders were coming in (albeit slowly), traffic was reasonable and life was good…. until I checked my trusty Google Webmaster tools and discovered over 750 page not found 404 errors in my site for pages that I thought were working perfectly well for the user.

So, here’s the situation. A URL that worked absolutely fine from any web browser was returning a 404 page not found error to search engines. Suddenly Google and Yahoo think that my site has disappeared into vapour and those pages that have taken months to creep up the search results drop like a stone.

The reason for this was that I’d tried to be clever by integrating Wordpress as a Content Management System (CMS) into an existing site.

I’m a BIG fan of Wordpress, not only as a blogging platform at which it excels but also as a generic CMS that I can use to create dynamic content for my sites. Out of the box it includes a fantastic admin system, hundred of plugins, great php based API, user management, commenting system and of course Google loves content and Google loves blogs which really helps with my SEO efforts. I’ve used Wordpress successfully now to add regularly updated content to a number of site - for example, I use the Amazon Web Services API to query the Amazon database and create website that sell specific products such as Jewellery (Jewelry for US readers!). By using Wordpress as a CMS I can dynamically bring in articles relevant to specific sections of the Jewellery site - for example, an article about the latest deals on Watches in the Watches department.

Back to my problem… what I done in my site is use an .htaccess file to allow pretty URLs to be used. For example: the URL /products/department/watches would actually go to the file /products.php?department/watches.

To integrate Wordpress into a site, you add the following code:

<?php define('WP_USE_THEMES', false); ?>
<?php require('./blog/wp-blog-header.php');?>

Wordpress doesn’t seem to like me using an .htaccess file that isn’t in sync with it’s own Permalink structure, so when one of my pretty Wordpress URLs was passed to wp-blog-header.php, it couldn’t work out where the page was and returned a 404 error … but only to search engines!

I haven’t figured out why a 404 was being returned for search engines but users could quite easily see the pages. That’s the a real mystery that I’m hoping readers of this site may have the answer to.