Category Archives: Web Development

Faster to re-calculate than retrieve from database

As you know, SearchTempest allows you to search craigslist by distance from your zip code. To do this, we keep database tables (MySQL) of zip codes and craigslist cities, each with latitude and longitude.

In the past, we also had one large table containing the distance between every single zip code and every craigslist city. Although this table takes a long time to put together, the logic is it only needs to be done once. When you actually do your search, all the distances for your zip code have already been calculated, so the server just has to grab them from the database.

I’ve had a suspicion for a while and just decided to test it. Long story short, it turns out it’s actually faster to recalculate these 500-odd distances on the fly, than it is to simply pull them out of a database table on the hard drive. Basically, processors are so fast compared to hard drives that minimizing hard drive access is practically the only thing that matters. Pretty cool!

(Yes, if the database were on an SSD the result might be different, but at the moment it takes approximately 0.01s to pull up all the distances, so that’s probably an unnecessary investment at this point! 🙂 )

Dynamic Bookmarks Idea

Have you ever discovered a cool webcomic, blog, youtube channel, etc., and wanted to read it from the beginning? Noticed that there isn’t really a good way to do that?

RSS feeds are great for keeping up to date with something, but (as far as I know), there’s no way to ‘go back to the beginning’, unless the site is specifically designed to allow it. Of course, you can just read a few then bookmark where you left off in your browser (or ‘favorite’ it if you’re using IE), but then every time you read a few more, you have to manually update the bookmark.

I was thinking, wouldn’t it be cool if there was a way – a Firefox extension maybe – to create dynamic bookmarks, which automatically updates to the last page you were on?

So you’d open up the bookmark, read a few comics, blog posts, etc., and it would automatically be updated to the last page you viewed before you moved on to another site or closed the browser tab. Next time you open that bookmark, you could just pick up where you left off.

If you know of something like this that already exists, please let me know about it in the comments! And if you have experience with browser extensions and would like to create it, I would be happy to promote it on the links page at!

How to undo bumps (or ‘unbump’) in phpbb

As you may have noticed, the SearchTempest forums use phpBB. Generally problems I’ve had with it are due to my own ignorance and today was no exception!

I was messing with the forum permissions, and I accidentally gave anonymous guest users the power to ‘bump’ topics. In phpbb, there’s a special function to bump a topic without having to post to it. Essentially it automatically updates the date of the latest post in that topic to the current time, so it appears as if it was just posted, and the topic moves to the top of the forum. (A little ‘Last bumped by [username]’ line appears at the bottom of the post, but I didn’t notice it at first.)

The problem is, apparently there are bots out there that will just go around bumping your posts like crazy if you let them. I’m not sure what they have to gain by this.. possibly just anarchy. Anyway, my temporarily lax user permissions resulted in a whole pile of topics from circa 2008 being bumped to present times. And since the bumping process actually changes the dates of the topic and its most recent post, there’s no easy way to just ‘unbump’ them. I didn’t relish the idea of manually going through my database editing the timestamps of tens or hundreds of posts, soooo I came up with a better solution.

(It probably took me just as long, but I learned some MySQL subtleties along the way, and now I can pass it along, in case anyone else runs into the same problem!) Without more ado, the code to automatically ‘unbump’ phpbb topics:

Continue reading

Transferring Large MySQL Databases with both MyISAM and InnoDB Tables

While setting up a development environment on my laptop, Nathan and I came across a bit of a problem: How do you transfer large databases in a way that a) works for databases with a mix of MyISAM and InnoDB tables, b) doesn’t take an eternity, and c) won’t wipe out any existing data on your server? After scouring forums and blog posts, I mixed a number of methods and came up with my own super awesome method! It’s pretty easy, and is much faster and more reliable than a straight MySQL dump:
  1. Copy database files from the old server to the new one. On the both servers, find your MySQL data directories. In WAMP, that directory is located at ‘WAMP\bin\mysql\mysql[version]\data\’. In MAMP, that directory is located at ‘MAMP/db/mysql/’ In LAMP, that directory is located at ‘LAMP/var/lib/mysql/’ Continue reading

iPhone/iPad/Android Awesomeness

After way too much effort, SearchTempest is now working quite well on all those platforms! For smartphones, it defaults to the Google-powered ‘All Cities Together’ tab on the results page, since this format requires less processing power. The iPad default is the same as in standard browsers though, where you get each craigslist city in a separate window. That said, both modes are now fully supported on phones and tablets running iOS or ‘droid though, so feel free to try ’em both! You’ll notice that instead of scrolling each city, as you would in a standard browser, you will just have to scroll the whole window. (If you like that, great! If you hate it.. sorry!) Basically, since these devices use touchscreens instead of mice and keyboards, their makers figured it would be difficult to scroll frames inside of windows. So instead they expand the frames to fit their contents, allowing you to just scroll the main site. (Of course, if you happen to be a web developer that can be kind of a pain in the butt, since 1) each phone and browser tends to do it in a slightly different way, and 2) frames expanding to fit contents you can’t control tends to wreck havoc on the design of your page in general… But fortunately only I have to worry about that. 😉 ) So! Enjoy the improved mobile compatibility, and of course, please comment or fire me an email if you notice any problems!

The Ultimate Website Deployment Solution

Most basic websites tend to be deployed something like this:
  1. Write code
  2. Copy code to website hosting via FTP
  3. Refresh page to see latest changes
And.. that works. But it leaves much to be desired, both in terms of performance and convenience, for you and for your users. I’ll go over some of those issues here, then dive into a little more technical detail in an upcoming post (or posts..) So, here are some of the things we would like to add to the deployment process:
  1. Version Control! If you haven’t used version control before, go check out Subversion. I would also recommend the windows client TortoiseSVN. Honestly, version control has the potential to save your ass by allowing you to revert bad changes. The ability to compare current and past versions of files also makes it an invaluable development tool. Plus a million more things.

    As far as the auto-deployment utopia though, version control is key because it allows you to automatically pull the latest versions of files (or the latest stable versions if you prefer) directly to your production server. No hunting around for changed files and uploading via ftp. Plus, in the worst case, if the update breaks the live version of the site, you can easily revert to how it was before.

  2. Compression. One of the goals of good web development is of course to have pages load as quickly as possible. Yahoo offers a fantastic tool call YUI Compressor, which will compress (or minify) CSS and Javascript files to a fraction of their initial size, while maintaining full functionality. It does this by eliminating unnecessary whitespace, optionally renaming variables (in a consistent manner of course), cutting out redundant braces, etc. You obviously wouldn’t want to actually work on the files in this format, but doing it before making them live helps page loads and essentially costs nothing.

    Continue reading

Common CSS Problem – Header background cut off when page scrolled horizontally

Actually it can be any element, but it’s usually seen with header divs. For example, head over to facebook. Shrink your page down until a horizontal scrollbar appears, then scroll to the right. Notice that the blue background in the header is cut off. I’ve been noticing this on a few websites lately, but just today thought to check SearchTempest – and found that it had the same problem! Now fixed. The problem is that by default (or even by explicit css) these divs fill 100% of the width of their container. (For example, the page body.) This is good, because no matter how big the browser window is stretched, the div will stretch to fill it. Unfortunately, the 100% refers to the window width, so if the window is shrunk to less than the size of the content, it will be cut off when the page is scrolled horizontally. The solution? Fortunately, it’s simple. Just use the min-width property, and to set it equal to the fixed-width size of the content. My content is 960px wide, so by setting min-width=960px on the header div, I ensure that it is always at least that wide, even if the window is narrower. Of course, it still extended to 100% of the window size if that is wider than 960px. (If your content is variable width, just set the min-width of the header equal to the greatest min-width within the content – ie to the width at which the horizontal scrollbar appears.) Have a better solution? I’d love to hear about it in the comments!