What a day

I had plans for today. I was going to do some coding.. add the new Google +1 thinger to SearchTempest, update the FAQ, port some stuff from that FAQ over to the AutoTempest FAQ, tweak the ads on SearchTempest, even work on a new feature or two! … my server had other ideas. I had an email waiting for me this morning from someone who couldn’t find an ad they had posted using SearchTempest. These are fairly frequent, and 98% of the time it’s for one of the reasons covered here. When the emailer claims to have already read the FAQ, that number drops to about 97%. 🙂 This case was no different. What was different was that when I pulled up SearchTempest to verify its omniscience, it took about 47 seconds for the homepage to come up. I pay way too much for screaming fast internet, so this was definitely not normal. Took a look at the apache logs, and noticed a bunch of entries like this:
mod_fcgid: can’t apply process slot for [filename]
mod_fcgid is a fastcgi module for apache, which allows apache (the web server) to talk to things like php without having to load them inside the server itself. This is great for a bunch of reasons that I won’t get into here. What wasn’t great was that it was out of processes. The server is configured such that it should handle many times the current peak load of the sites (just in case I get slashdotted or what have you.. 🙂 ), so that shouldn’t happen. At this point I’m kind of freaking out because a) all you fine people can’t access SearchTempest, b) the other fine people who probably aren’t reading this because they’re new users are now thinking that SearchTempest is crap and doesn’t work!, and c) I’m nearing the end of my very limited LAMP (Linux, Apache, MySQL, PHP) debugging abilities. I sent off a somewhat panicked message to the support folks at the company I lease the server from, then, as is my way, proceeded to get in their way trying to debug the problem myself at the same time. Surprisingly, I was actually pretty successful with that, notwithstanding shooting myself in the foot a couple times. I’ll spare you all the details, but what I figured out was that the site wasn’t slow because of running out of php processes; rather, it was running out of processes because they were too slow. Specifically, the script that grabs the ebay results was timing out because there was something broken out in the internet between searchtempest.com and ebay.com. Because that script was using the default timeout of 60 seconds, each of those accesses to grab ebay results that would normally take less than a second were taking 60 seconds, depleting the supply of processes pretty quickly! (And yes, this IS me sparing you the boring details. 😉 ) Anyway, from there it was standard. I only had to solve one impossible problem, and did one very stupid thing – par for the course.
  • Impossible problem: Reduced the timeout from 60 to 5 seconds, which should have let everything work again while the path to ebay was down. Nothing changed. The script in question was still taking a minute to timeout. (Which was of course impossible. Facepalm.) Turns out the timeout I was setting only applied to the transfer of data *after* a connection was made, so never got to take effect. Facepalm x2.
  • Very Stupid Thing: To test whether I could get data from ebay to the server, I used the handy linux command wget. Tried to wget some ebay results and failed. Then wgot yahoo.com, just to make sure it worked. That was fine.About 15 minutes later, was informed that SearchTempest had been replaced by yahoo.com. Freaked out. … then realized that I had used wget to grab the yahoo homepage while in the root directory for SearchTempest, thereby replacing the ST homepage with theirs. (I should edit the wikipedia entry on knowing just enough to be dangerous and add my picture.)
Anyway, all’s well now. I survived the day and even learned a few things. If something like this happens again, SearchTempest will have sensible timeouts in place and will be impervious to harm. My brain is now mush-like, but from past experience should be better after sleep. I’m sure this blog post reflects that, but I’ll let it stand as further evidence of the effects of the day. The important thing is SearchTempest is now up, and the people rejoice. 🙂