SearchTempest Update

As mentioned in recent posts, craigslist has been making some changes lately that we have had to adapt to. (We have no affiliation with craigslist at all, so we don’t get any warning or help from their end when they change something; we just have to figure it out.)

In the past, we allowed you to open results from multiple craigslist cities at once, all on the same page using ‘iframes’. A couple months back, CL added some code to their site to prevent their pages being opened in iframes, so we had to find a new way. As a result we developed two new results modes:

  • Separate Cities – Similar in appearance to our old setup, but inside each city ‘frame’, instead of a results page straight from craigslist, we showed results from a third-party provider called 3taps. 3taps is an ‘exchange data API’, which got craigslist listings from the google cache, so they did not have to actually scrape craigslist.

  • Direct Results – The goal of this mode was to preserve the straight-from-craigslist feature of the frames. Since we could no longer link to craigslist results pages in frames, instead we created an interface whereby they could be conveniently opened in new windows. Essentially we open up a small navigation window on the left, with links to craigslist results that open in a larger window on the right. Since both remain visible at the same time, it lets you conveniently flip through results from multiple cities.

Recently there has been another development. craigslist has apparently instructed google to stop caching their results. (Note, that doesn’t mean their results have been removed from google, just that google doesn’t store copies of entire craigslist pages.) This effectively cuts off 3taps’ source for craigslist data. For us, it meant we needed to find yet another new way to do the Separate Cities mode.

So, what we’re doing now is powering that mode with a Google Custom Search. Basically this allows us to customize a Google search so it only returns results from certain sites. So we take your search at SearchTempest, then send a request to google with your keywords, along with a set of sites to search based on the zip code, distance, and category you entered. Google returns a bunch of results fitting those parameters, which we display for you. When you click on one of the results links, it your browser opens that posting right at craigslist in a new window.

There are a couple downsides to this google-powered approach. The main ones are that it does not support filtering by price or displaying prices in the results, and that you can’t filter by ads with pictures. Also, preview images only work if the ad was created with embedded html images, not the standard craigslist pics. If you need those features, we suggest using the Direct Results mode mentioned above, which should do everything you need!

So in summary, the Separate Cities mode is now powered by Google. It’s probably the most convenient option for most people. If you find yourself missing those price or image filters though, or if you just want to get your results straight from the source, rather than via google, you should give the Direct Results mode a try.

Next thing on our agenda, thanks to user feedback, is simplifying the results page. We’re going to be removing a bunch of tabs, consolidating the Separate Cities and All Cities results modes into one (since they’re both now powered by Google), and just generally making everything a lot more straightforward. Hope to have that ready for you soon!

Code Compression

Was thinking recently about how any good code base tends to go through a continuous cycle of expansion and compression. (This thinking may have been inspired by the recent frantic development work on SearchTempest in the wake of craigslist blocking framing…) The ‘expansion’ part is the standard stuff. Building something new, adding features, even fixing bugs most of the time tends to involve writing more code, causing the code base to expand.

However, if you only expand and never compress, eventually you will inevitably end up with a giant pile of pasta.

It’s critical to occasionally go through your code and simplify. Trace through the logic and figure out how it can be improved. Look at places where procedural stuff could be made object-oriented. Even just strip out legacy crap that isn’t used anymore.

Of course, this can be difficult to justify. There are always higher priorities, and it’s tough to put a bunch of time into a project that, in the best case, has no immediate visible effect. (Especially if you happen to answer to a manager who hasn’t personally done much/any coding.) And that’s the best case. It’s ironic, but this process of cleaning up code can very well introduce new bugs. After all, it may involve making fairly significant structural changes to a code base that is by all appearances working just fine. (And those bugs tend to make people… irate. Don’t do something like this then go away for the weekend.)

So, why do we bother? Here’s a similar issue: why do we bother researching sources of renewable energy? It has always -so far- been cheaper to just stick with fossil fuels. That may well continue to be the case nearly until they run out, since the incremental cost of extracting a barrel of oil does not increase linearly with its scarcity. But if we wait until we run out of oil, or until we destroy our atmosphere burning coal, it’s too late.

Of course, the consequences of ugly code are rather less dramatic, but the analogy holds. If you wait long enough, eventually you will be forced to clean up your spaghetti code because you’ll get to the point where adding one more hack will break the camel’s back. You’ll have a feature to add, or a bug to fix, and it simply won’t be possible to shoehorn it into the existing morass. When that day comes, it is NOT a fun day to contemplate redesigning your whole code base. Especially if the bug you’re trying to fix happens to be a critical one.

On the other hand, when such a fateful day rolls around, a clean code base can be a truly beautiful thing. A little irony: the best thing about nice clean code is that it makes it really quick and easy to slap on an ugly hack. And when your site’s down and the hysterical emails are rolling in, that ugly hack that gets you running again can be a beautiful thing too.

Or more generally, by periodically cleaning up your code, you make your job a lot easier the rest of the time. Of course, you could try to just ‘do things right the first time’. But even without deadlines (which takes us into imaginary-land), it’s pretty difficult to always keep the entire big picture in mind while solving a specific problem. Of course you should still try to write nice, clean, extensible code whenever possible. Sometimes though, you still have to take a step back.

In the real world, the cleanup of a given module will likely be spurred by some other development, which is okay. For me at least, there’s no motivation to sit down for the express purpose of prettifying code. But when you’ve already dug into something a bit and you start to see avenues for improvement -and you’re not in an absolutely critical time crunch- go for it! (If you’re always in a critical time crunch, you’re doing it wrong. Or someone is.) It’s just like cleaning in real life actually. Ever go to pick up a dirty sock and end up doing all the laundry then cleaning the entire house? Do that with code! (If not, go vacuum something. I bet your partner/roommate/cat will appreciate it.)

Nice clean code is a beautiful thing, but it’s elusive; you can’t aim straight for it. What you can do is write fairly decent code, then occasionally compress it. Channel your inner Superman and squeeze that code coal into a precious diamond. (Ha! Tied those analogies together!)

If it ain’t broke, now’s a great time to fix it!

Performance Improvements

We’ve just finished some pretty major changes behind the scenes to improve performance from the new 3taps-powered ‘Separate Cities’ results. If all goes well, you shouldn’t see any difference… except for a much snappier site. If you do run into any new problems, please feel free to let us know, and we’ll take care of them asap!

Recent Changes

Craigslist Change

Craigslist (with whom we have no affiliation) recently made a change that prevents their pages from being loaded in ‘frames’. Frames are basically sub-windows within another page, like our default results mode at SearchTempest used to use. This caused several other search sites to simply shut down, but we made some changes and are still going strong.

Problems?

If you are having issues, please try the debugging steps in our FAQ. Then read on!

Changes to Separate Cities Mode

First, you will notice there are now three tabs near the top of the results page. The first, ‘Separate Cities’ is basically our default results mode from before, except that we have switched to getting the links to the craigslist results from a service called 3taps. It works essentially the same as before for now, although in the future it will allow us to do some cool new things like filtering by date and consolidating everything into one list rather than windows within windows.

The only issue at the moment is that 3taps didn’t quite have the capacity to handle all our traffic, so things can be a bit sluggish at times, especially between 5-7pm pacific time. They are working hard to ramp up, and this should be dealt with very soon. Also, in order to help with the slowdowns in the short term, currently only results from the past seven days are shown in this mode.

Update: 3taps added some new servers today, so we’re hoping this will be resolved. We may still have to do some tuning, but timeouts should now be seen much less frequently.

New Direct Results Mode

Now, for those of you who liked getting your results straight from craigslist, just more conveniently, we have also rushed to develop a new alternative that will let you keep doing just that! You’ll find it under the ‘Direct Results’ tab on the results page. Since we can no longer use frames, it basically opens a craigslist results window along with a little control window with links to easily flip between cities. I’m pretty pleased with how it turned out in such a short time!

And of course, we still have the ‘All Cities Together’ mode, which remains a good option for quick searches across wide areas.

Summary

So basically, craigslist made a change, so we reacted to keep the site (hopefully!) as useful as ever. We are still dealing with that performance issue in the ‘Separate Cities’ results mode, but it should be 100% very soon. For now you might want to try searching outside the peak hours of 5-7 pacific if you find things are running slowly.

We also recommend you give all three results modes a try, using those tabs near the top of the results page: ‘Separate Cities’, ‘All Cities Together’, and ‘Direct Results’. We’ve found that different modes are more convenient for different types of searches, so there’s no right answer for everyone. Fortunately, it’s easy to quickly flip between them!

Feedback

Since we got all this out in such a rush, please do let us know about any difficulties you might be having. We will work to address them ASAP. Even if everything’s working great (and we certainly hope it is!) we want to hear your first impressions, and what you’d like to see in the future!

If you are having problems though, first try out these basic debugging steps. If those don’t work, please do contact us, but please be as specific as possible about what isn’t working. What results mode are you using, what’s the URL of your results page, etc. We need specific info about the problem to try and fix it!

You can comment here, check out the forum thread, or even just fire me an email. Thanks!

 

Browser Debugging

To the web developers: ever get an obscure error in one browser, so run the code in a different browser to see if it’s more helpful? Try it next time you get a head-scratcher. 🙂 Just now Firefox was giving me “NetworkError 596”. To which I said, “Oh, of course!” :roll: So I fired up Chrome, and it gave a much more helpful console message, helping me realize that I was accidentally trying to make a cross-domain JSON request instead of JSONp. Oops.

Anyway, I’ve found this technique handy for a bunch of things. Sometimes Firebug’s message is better. Even IE sometimes gives the most verbose errors. Just depends on what obscure problem you’ve run into!

Scavenger Hunt Update: craigslist is being changed to eBay

I know you’re all excited with our Scavenger Hunt coming up in a couple days. Initially the idea was that you guys would search for craigslist ads for the hunt, but after a letter from craigslist’s lawyers, we’ll be switching the hunt to eBay. You can still use SearchTempest to hunt for ads, but you can just go straight to eBay as well.

One of the benefits of doing the hunt on eBay is, since eBay archives ads forever, a screenshot of the ad will no longer be needed, just a link to the ad! That makes things a lot easier for all of us.

Other than those small changes, everything will be going ahead as scheduled. You can check out the revised Scavenger Hunt page and make sure to follow our Facebook for the category being posted at noon, PST, on May 1st!

$500 in prizes with SearchTempest Scavenger Hunts!

We’ve all found some weird things using SearchTempest. Over on Facebook, we’ve heard about people finding llamas, universal dentures, a case of rubber nipples… there’s a lot of weird stuff out there. But there’s also a ton of really cool stuff on there too! And that got us thinking…

Could we do a scavenger hunt for cool stuff using SearchTempest? Short answer: Of course we can.

On May 1st, we’re going to post on Facebook the category for our first ever SEARCHTEMPEST SCAVENGER HUNT! And of course there will be prizes, even if you’re just a voter.

Basically, we’ll post a category for the hunt, like “The coolest _____” or “The weirdest _____”. Then when you guys find something on eBay* that fits in that category, you’ll post it to Facebook and everyone votes for their favorites! There will be three different prizes given out:

First Prize of $250 goes to the person who posts the ad that gets the most likes. If you want, we’ll also put your name and link in the sidebar of the SearchTempest results page for a month!

Honorable Mention of $150 goes to the person who posts the best ad, as judged by the impeccable taste of the Tempest staff.

Bonus Prize of $100 goes to a random “liker” on the post we choose for Honorable Mention, since agreeing with us clearly deserves a reward. Just don’t go spamming all the posts in hopes of winning. Picking everything isn’t good taste. Keep it to the ones you actually like, if you want to win this prize.

Check out our Scavenger Hunt page for more on the rules and how to make a submission.

We’ll be hosting these regularly, and the prizes will vary with each draw.

The topic of our first Scavenger Hunt will be released at noon, PST, May 1st, and submissions will be accepted until midnight, May 21st. Even though you have three weeks to get your submission in, remember: all listings must be posted to eBay BEFORE the start of the competition, so it’s best to start searching early. Also, the longer your submission is up there, the longer it has to get votes!

And that’s it! Stay tuned to our Facebook and Scavenger Hunt page for more info.

Tell your friends, and happy searching!

*Note: this is a hunt for eBay ads – which SearchTempest also searches – not craigslist as you might expect.

Now you can save your searches in AutoTempest!

If you’ve been using SearchTempest for a while, you’re probably familiar with its My Searches feature. It automatically saves your recent searches so you can access them later without having to fill out the entire search form again. It also lets you tag your searches as favorites, so those searches will be saved forever (or at least until you don’t want them anymore). It’s handy if you run the same search pretty frequently, or if you are jumping back and forth between searches.

If this is the first you’re hearing about this feature, then you should totally try it out. You can access it by clicking that little yellow star just above the search button on the SearchTempest home page.

Why am I bringing this up? Because we just brought that same feature over to AutoTempest, that’s why. You can find it in the navigation bar at the far right. So now your favorite used car search just got that much more awesome.

Eventually, we’d like to add to both SearchTempest and AutoTempest the ability to login to those sites and save your searches so you can access them from a different computer too. Keep an eye out for that in the not so distant future!

If you’re new to AutoTempest and would like to know about it, keep reading after the jump.
Read More

Faster to re-calculate than retrieve from database

As you know, SearchTempest allows you to search craigslist by distance from your zip code. To do this, we keep database tables (MySQL) of zip codes and craigslist cities, each with latitude and longitude.

In the past, we also had one large table containing the distance between every single zip code and every craigslist city. Although this table takes a long time to put together, the logic is it only needs to be done once. When you actually do your search, all the distances for your zip code have already been calculated, so the server just has to grab them from the database.

I’ve had a suspicion for a while and just decided to test it. Long story short, it turns out it’s actually faster to recalculate these 500-odd distances on the fly, than it is to simply pull them out of a database table on the hard drive. Basically, processors are so fast compared to hard drives that minimizing hard drive access is practically the only thing that matters. Pretty cool!

(Yes, if the database were on an SSD the result might be different, but at the moment it takes approximately 0.01s to pull up all the distances, so that’s probably an unnecessary investment at this point! 🙂 )

More AutoTempest Improvements – Making things even easier for Canadian car buyers

As you know, we’re on a mission to make used car buying the easiest, fastest most efficient process possible with our used car search engine AutoTempest.

With that said, constant improvements are the name of the game!

As someone who lives in Canada, ever feel like our friends south of the border have it WAY easier when it comes to finding a used car? We certainly thought so, and we’re pleased to say, no more!

Canadian car buyers – AutoTempest is here for you. Here are the 2 major improvements we’ve just made…

A new car classifieds addition

AutoTempest already searched Craigslist and eBay north of the 49th, but we’ve now added the #1 used car classifieds site in Canada, AutoTrader.ca to the AutoTempest car search platform as well.

When you search used cars online with AutoTempest, you’ll now get an even greater volume of listings in Canada.

Improved search results for Canadian postal codes

But what if you’re planning to look across the border too? As we all know, there are some great deals to be had by buying a used car in the States. Well, AutoTempest now supports Canadian postal codes, even for sites like Cars.com and CarsDirect that only operate on the US side of the border.

No more wasting time figuring out and looking up nearby zip codes in the US… perhaps while cursing the fact that the only ZIP code you can seem to remember is 90210! AutoTempest will now automatically find the zip code of the border crossing nearest to you and use that for any sites that require a zip. So if you’re like many, curious about the incredible car deals you hear about down in the States, AutoTempest will give you the straight goods, as always, all in one place.

We’d love to hear your thoughts on the Canadian improvements… let us know in the comments.