Category Archives: SearchTempest

The EASY way to search craigslist

Google not indexing craigslist – SearchTempest switches to Bing

As of February 28, Google has stopped indexing new craigslist posts. Or more specifically, every day between about 5pm and midnight PST, they index them as usual. Then at midnight, they throw them all away. So anyone searching Google for craigslist posts over the past couple weeks has been faced with a giant gap since the beginning of March.

SearchTempest has no affiliation with craigslist, so until recently, we used Google to power our searches. Since Google is no longer getting the job done though, we’ve switched to Bing!

To be honest, Bing’s API doesn’t hold a candle to Google Custom Search. You can’t sort by date, specify a list of urls to search (Google’s ‘annotations’), or even reliably search within the url at all. (Bing does have a semi-hidden option, instreamset:(url):{text}, which is similar to Google’s inurl:{text}, but we’ve found it to be unreliable.)

That said, through some clever manipulation of query strings and a mess of hard-coded special cases, we’ve managed to come up with a Bing-powered craigslist search that’s quite functional. If you’re frustrated by not being able to search craigslist through Google like before, give it a try!

We Want Your Input on the New SearchTempest.com Results Page!

Oct. 29th/2012 UPDATE: After a few minor tweaks based on your suggestions, we have pushed these changes to the live site. Please let us know what you think via comment on this post or posting to our Facebook page.
—

Based on the feedback we have received from you, our lovely users, we have re-done the SearchTempest.com results page! We have made things simpler, easier to use, and more intuitive. You can still do everything you could on the old site, with some new stuff too, it’s just packaged now in a easier to read and understand, and nicer to look at, package.

The new site is not live yet and that’s because we wanted to get your feedback on it so we can make it the best it can be before we make it live. You can start playing around with it at Beta.SearchTempest.com and you can learn more about the changes made below. Have thoughts on what could be improved or just want to tell us how awesome you think it is? Leave us a comment on our Facebook page.

Changes

  • The first thing you will probably notice is that the “Separate Cities”, “All Cities Together”, and “Direct Results” tabs are gone. Instead, we have settled on two modes: “Combined Results” and “Direct Results”. “Direct Results” looks and acts just like it used to. Since “Separate Cities” and “All Cities Together” provided the same information, just displayed differently (the former sorting results by cities, the latter by individual results), we have put those two in the same mode, which we call “Combined Results”. You can switch between the old “Separate Cities” and “All Cities Together” modes by choosing how you want to sort the results from the “Sort by:” dropdown. Select “Distance”, “State”, or “City Size” to sort by city (“Separate Cities”); choose “Most Recent” or “Best Match” to sort by individual results (“All Cities Together”).
  • You will also notice that we got rid of the upper tabs as well. “Modify Search” can now be called from the “Full Search” button in the search bar and “Display Options” has become the “Preferences” link in the results page.
  • We have generally cleaned up the page to make it less cluttered and easier to take in the information you need and leave out the stuff you don’t.

Date Filter

We have also added a new feature that we have had many requests for, and that is the ability to filter the results by the dates they were posted. You will see a new field with the label “Within the last # days:”. Pretty self-explanatory, but if you put the number 2 in there, you will only get results that are max 2 days old. Pretty awesome, right?

Please check out the new site and let us know how we can make it better for you!

SearchTempest Update

As mentioned in recent posts, craigslist has been making some changes lately that we have had to adapt to. (We have no affiliation with craigslist at all, so we don’t get any warning or help from their end when they change something; we just have to figure it out.)

In the past, we allowed you to open results from multiple craigslist cities at once, all on the same page using ‘iframes’. A couple months back, CL added some code to their site to prevent their pages being opened in iframes, so we had to find a new way. As a result we developed two new results modes:

  • Separate Cities – Similar in appearance to our old setup, but inside each city ‘frame’, instead of a results page straight from craigslist, we showed results from a third-party provider called 3taps. 3taps is an ‘exchange data API’, which got craigslist listings from the google cache, so they did not have to actually scrape craigslist.

  • Direct Results – The goal of this mode was to preserve the straight-from-craigslist feature of the frames. Since we could no longer link to craigslist results pages in frames, instead we created an interface whereby they could be conveniently opened in new windows. Essentially we open up a small navigation window on the left, with links to craigslist results that open in a larger window on the right. Since both remain visible at the same time, it lets you conveniently flip through results from multiple cities.

Recently there has been another development. craigslist has apparently instructed google to stop caching their results. (Note, that doesn’t mean their results have been removed from google, just that google doesn’t store copies of entire craigslist pages.) This effectively cuts off 3taps’ source for craigslist data. For us, it meant we needed to find yet another new way to do the Separate Cities mode.

So, what we’re doing now is powering that mode with a Google Custom Search. Basically this allows us to customize a Google search so it only returns results from certain sites. So we take your search at SearchTempest, then send a request to google with your keywords, along with a set of sites to search based on the zip code, distance, and category you entered. Google returns a bunch of results fitting those parameters, which we display for you. When you click on one of the results links, it your browser opens that posting right at craigslist in a new window.

There are a couple downsides to this google-powered approach. The main ones are that it does not support filtering by price or displaying prices in the results, and that you can’t filter by ads with pictures. Also, preview images only work if the ad was created with embedded html images, not the standard craigslist pics. If you need those features, we suggest using the Direct Results mode mentioned above, which should do everything you need!

So in summary, the Separate Cities mode is now powered by Google. It’s probably the most convenient option for most people. If you find yourself missing those price or image filters though, or if you just want to get your results straight from the source, rather than via google, you should give the Direct Results mode a try.

Next thing on our agenda, thanks to user feedback, is simplifying the results page. We’re going to be removing a bunch of tabs, consolidating the Separate Cities and All Cities results modes into one (since they’re both now powered by Google), and just generally making everything a lot more straightforward. Hope to have that ready for you soon!

Performance Improvements

We’ve just finished some pretty major changes behind the scenes to improve performance from the new 3taps-powered ‘Separate Cities’ results. If all goes well, you shouldn’t see any difference… except for a much snappier site. If you do run into any new problems, please feel free to let us know, and we’ll take care of them asap!

Recent Changes

Craigslist Change

Craigslist (with whom we have no affiliation) recently made a change that prevents their pages from being loaded in ‘frames’. Frames are basically sub-windows within another page, like our default results mode at SearchTempest used to use. This caused several other search sites to simply shut down, but we made some changes and are still going strong.

Problems?

If you are having issues, please try the debugging steps in our FAQ. Then read on!

Changes to Separate Cities Mode

First, you will notice there are now three tabs near the top of the results page. The first, ‘Separate Cities’ is basically our default results mode from before, except that we have switched to getting the links to the craigslist results from a service called 3taps. It works essentially the same as before for now, although in the future it will allow us to do some cool new things like filtering by date and consolidating everything into one list rather than windows within windows.

The only issue at the moment is that 3taps didn’t quite have the capacity to handle all our traffic, so things can be a bit sluggish at times, especially between 5-7pm pacific time. They are working hard to ramp up, and this should be dealt with very soon. Also, in order to help with the slowdowns in the short term, currently only results from the past seven days are shown in this mode.

Update: 3taps added some new servers today, so we’re hoping this will be resolved. We may still have to do some tuning, but timeouts should now be seen much less frequently.

New Direct Results Mode

Now, for those of you who liked getting your results straight from craigslist, just more conveniently, we have also rushed to develop a new alternative that will let you keep doing just that! You’ll find it under the ‘Direct Results’ tab on the results page. Since we can no longer use frames, it basically opens a craigslist results window along with a little control window with links to easily flip between cities. I’m pretty pleased with how it turned out in such a short time!

And of course, we still have the ‘All Cities Together’ mode, which remains a good option for quick searches across wide areas.

Summary

So basically, craigslist made a change, so we reacted to keep the site (hopefully!) as useful as ever. We are still dealing with that performance issue in the ‘Separate Cities’ results mode, but it should be 100% very soon. For now you might want to try searching outside the peak hours of 5-7 pacific if you find things are running slowly.

We also recommend you give all three results modes a try, using those tabs near the top of the results page: ‘Separate Cities’, ‘All Cities Together’, and ‘Direct Results’. We’ve found that different modes are more convenient for different types of searches, so there’s no right answer for everyone. Fortunately, it’s easy to quickly flip between them!

Feedback

Since we got all this out in such a rush, please do let us know about any difficulties you might be having. We will work to address them ASAP. Even if everything’s working great (and we certainly hope it is!) we want to hear your first impressions, and what you’d like to see in the future!

If you are having problems though, first try out these basic debugging steps. If those don’t work, please do contact us, but please be as specific as possible about what isn’t working. What results mode are you using, what’s the URL of your results page, etc. We need specific info about the problem to try and fix it!

You can comment here, check out the forum thread, or even just fire me an email. Thanks!

 

Scavenger Hunt Update: craigslist is being changed to eBay

I know you’re all excited with our Scavenger Hunt coming up in a couple days. Initially the idea was that you guys would search for craigslist ads for the hunt, but after a letter from craigslist’s lawyers, we’ll be switching the hunt to eBay. You can still use SearchTempest to hunt for ads, but you can just go straight to eBay as well.

One of the benefits of doing the hunt on eBay is, since eBay archives ads forever, a screenshot of the ad will no longer be needed, just a link to the ad! That makes things a lot easier for all of us.

Other than those small changes, everything will be going ahead as scheduled. You can check out the revised Scavenger Hunt page and make sure to follow our Facebook for the category being posted at noon, PST, on May 1st!

$500 in prizes with SearchTempest Scavenger Hunts!

We’ve all found some weird things using SearchTempest. Over on Facebook, we’ve heard about people finding llamas, universal dentures, a case of rubber nipples… there’s a lot of weird stuff out there. But there’s also a ton of really cool stuff on there too! And that got us thinking…

Could we do a scavenger hunt for cool stuff using SearchTempest? Short answer: Of course we can.

On May 1st, we’re going to post on Facebook the category for our first ever SEARCHTEMPEST SCAVENGER HUNT! And of course there will be prizes, even if you’re just a voter.

Basically, we’ll post a category for the hunt, like “The coolest _____” or “The weirdest _____”. Then when you guys find something on eBay* that fits in that category, you’ll post it to Facebook and everyone votes for their favorites! There will be three different prizes given out:

First Prize of $250 goes to the person who posts the ad that gets the most likes. If you want, we’ll also put your name and link in the sidebar of the SearchTempest results page for a month!

Honorable Mention of $150 goes to the person who posts the best ad, as judged by the impeccable taste of the Tempest staff.

Bonus Prize of $100 goes to a random “liker” on the post we choose for Honorable Mention, since agreeing with us clearly deserves a reward. Just don’t go spamming all the posts in hopes of winning. Picking everything isn’t good taste. Keep it to the ones you actually like, if you want to win this prize.

Check out our Scavenger Hunt page for more on the rules and how to make a submission.

We’ll be hosting these regularly, and the prizes will vary with each draw.

The topic of our first Scavenger Hunt will be released at noon, PST, May 1st, and submissions will be accepted until midnight, May 21st. Even though you have three weeks to get your submission in, remember: all listings must be posted to eBay BEFORE the start of the competition, so it’s best to start searching early. Also, the longer your submission is up there, the longer it has to get votes!

And that’s it! Stay tuned to our Facebook and Scavenger Hunt page for more info.

Tell your friends, and happy searching!

*Note: this is a hunt for eBay ads – which SearchTempest also searches – not craigslist as you might expect.

Faster to re-calculate than retrieve from database

As you know, SearchTempest allows you to search craigslist by distance from your zip code. To do this, we keep database tables (MySQL) of zip codes and craigslist cities, each with latitude and longitude.

In the past, we also had one large table containing the distance between every single zip code and every craigslist city. Although this table takes a long time to put together, the logic is it only needs to be done once. When you actually do your search, all the distances for your zip code have already been calculated, so the server just has to grab them from the database.

I’ve had a suspicion for a while and just decided to test it. Long story short, it turns out it’s actually faster to recalculate these 500-odd distances on the fly, than it is to simply pull them out of a database table on the hard drive. Basically, processors are so fast compared to hard drives that minimizing hard drive access is practically the only thing that matters. Pretty cool!

(Yes, if the database were on an SSD the result might be different, but at the moment it takes approximately 0.01s to pull up all the distances, so that’s probably an unnecessary investment at this point! 🙂 )

Google Mode Awesome Again

Google has fixed the bug on their end, so the ‘All Cities Together’ tab on the results page is 100% awesome again. Options to show or hide previews and to control number of results per page are once again available under the Display Options tab. Enjoy!

iPhone/iPad/Android Awesomeness

After way too much effort, SearchTempest is now working quite well on all those platforms! For smartphones, it defaults to the Google-powered ‘All Cities Together’ tab on the results page, since this format requires less processing power. The iPad default is the same as in standard browsers though, where you get each craigslist city in a separate window. That said, both modes are now fully supported on phones and tablets running iOS or ‘droid though, so feel free to try ’em both! You’ll notice that instead of scrolling each city, as you would in a standard browser, you will just have to scroll the whole window. (If you like that, great! If you hate it.. sorry!) Basically, since these devices use touchscreens instead of mice and keyboards, their makers figured it would be difficult to scroll frames inside of windows. So instead they expand the frames to fit their contents, allowing you to just scroll the main site. (Of course, if you happen to be a web developer that can be kind of a pain in the butt, since 1) each phone and browser tends to do it in a slightly different way, and 2) frames expanding to fit contents you can’t control tends to wreck havoc on the design of your page in general… But fortunately only I have to worry about that. 😉 ) So! Enjoy the improved mobile compatibility, and of course, please comment or fire me an email if you notice any problems!