Lighttpd and ruby on rails were making me Google problems

Wow, it took me a really long time to notice, and I feel almost stupid now.

I didn’t pay any attention but Google never brought any traffic to the Apparent Software website. But somehow it didn’t bother me as much, I was busy developing. A few days ago I noticed this and started to look into the source.

Looking at the server log files I noticed that all my responses for pages were HTTP status 404 (page not found). Apparently, Google doesn’t index requests that return code 404 (and it really shoudn’t).

I use Ruby On Rails (RoR) for my Apparent Software website and lighttpd with Fast-CGI to serve it. The easiest way to serve with Fast-CGI and RoR is setting the webapp to be the 404 error handler in lighttpd, using the following line of code:

server.error-handler-404 = "/dispatch.fcgi"

In the specific version of lighttpd web server that I was using (1.4.16) there was this nasty bug that stripped HTTP response header from the app in this case. So today I upgraded to the latest stable 1.4.19 and, voila, the response code is back to normal (200). 

Now, hopefully, Google and its lesser brothers will catch up, crawl the site, index and divert some search traffic into Apparent Software and another site I had with the same issue. I wonder how long it will take. I’ll keep updated. 

Pin It on Pinterest