Category Archives: SearchTempest

The EASY way to search craigslist

How to Follow Tempest

As you may notice scanning the dates below, we’re not blogging here very often anymore! But we’re certainly still very active working on AutoTempest.com and SearchTempest.com.

We blog about AutoTempest and car buying in general at the AutoTempest Blog.

You can give us feedback and follow our progress on AutoTempest here, and on SearchTempest here.

And here are some of our other social channels:

So, while we may not be blogging here too often, rest assured we’re still working hard to make the sites as good as they can be, and please feel free to contact us via any of those other channels, or by email!

Where did the Separate Cities results go?

As you may know, the SearchTempest.com results are powered by Google and Bing.  Until recently we used a “Google Custom Search” to show results separated by city.  Unfortunately it appears that Google has now changed their policies, and permit only a single custom search per page, which means we are no longer able to show results for each city separately.  Unfortunately it also appears that when multiple cities are searched together, Google tends to miss more matching results.

There is some good news though!  We have just released a major update to our Direct Results mode, which allows you to get results directly from the source.  As well as giving you several options for how to group the results (by large areas, by state, or by individual city,) this also avoids the main problem with using Google/Bing results: delayed or missing listings.  Since the results come straight from the source to your browser, they are always complete and up to date.

In the past, Direct Results required you to click through each city individually, but we now combine multiple cities into each link, making it significant faster and easier to use.  (Although the old per-city Direct Results are still available if you want.)  You can learn more about Direct Results here.  In the coming days we will be making it new default view.  To give it a try now though, just flip the “Single List Results / Direct Results” toggle above the list of results to “Direct Results”.  Also, once we make it the default, you will still be able to switch back to the Google-powered Single List results using that same toggle.

We hope you like the new Direct Results!

Cities stuck “Retrieving Results from Google”

Update: Fix released

We have released an update that should make this significantly less of a hassle.  We now have built-in timers on in SearchTempest to avoid hitting this Google limit, so instead of being timed out and having everything break for five minutes, the worst you’ll experience is a few seconds delay (with a message explaining why).  This still might be an annoyance for larger searches, so we do have a couple other options for you as well.  Please see here for more info: http://support.searchtempest.com/hc/en-us/articles/226378827-Limit-on-number-of-cities-that-can-be-searched-in-a-short-time.

Original post

Some of you have probably run across this recently: view a few pages of results fine, and then all the cities just get stuck “Retrieving Results from Google”.  This issue just came up a few days ago, but what we’ve been able to determine so far is that Google have added a new limit to how many queries can be sent to their Custom Search API (which powers our results) in a short period of time.  We’re in the process of determining what those limits are, so that we can put our own limits in place to avoid hitting them.

For the moment, as long as you’re taking the time to read each page of results before moving on to the next, it’s unlikely you’ll have this problem.  Where it is more likely is if you’re searching for something rare, so a bunch of cities are skipped through quickly.  For now we suggest just taking your time with those searches.  Loading a chunk of 50 or so cities (including ones with no results), waiting 30 seconds, then loading the next chunk.  That should avoid hitting any limits.

Another option is to switch your sort order (using the “Sort” drop down on the results page) to one of the single list options.  That way a separate query doesn’t need to be sent to Google for each city, which will guarantee no limits are hit.

Again, we’re still actively working on this and hope to  have better work-arounds in place soon.  You can find more info here: http://support.searchtempest.com/hc/en-us/articles/205166587-The-results-load-slowly-

Default Results Sort Change

Update: There has been… significant… reaction in favor of sticking with Stacked Cities as the default, so we’re going to switch back, and look at ways to improve that view.  As always though, the sort order is entirely your choice, and can be changed using the “Sort” dropdown on the results page, just below the search bar.

—

SearchTempest is powered by a Google Custom Search.  Basically Custom Search is an API that Google offers, allowing other websites to run customized Google searches.  Like any search engine though, Google sometimes misses results, especially new ones.  We don’t like missing results, so recently we added Bing search as a backup option, to try to catch anything Google misses.

Unfortunately, it’s not possible to search each city separately with Bing like we do with Google.  We can search all the cities in your search area at once though, and show the results in a single list.  And we have found that for most searches, this Google + Bing combination returns better results than the “Separate Cities” searches we do with Google.  Therefore, we’re switching the default sort to a single list, including Google and Bing.

The “Separate Cities” sort options aren’t going away though!  If you preferred that way of viewing results, changing back is as easy as clicking the “Sort” dropdown box on the results page, just below the search bar.  We’re also looking at creating a sort of hybrid view that could offer the best of both worlds.

Finally, we also still do offer our Direct Results view, which doesn’t miss anything, so that might be of interest to some of you, especially if you like having the results separated by city.  You can learn more about Direct Results here.

Craigslist blocking Feedly again

It looks like craigslist is once again blocking Feedly from accessing its RSS feeds. This happened a few months ago when Feedly traffic apparently got high enough to hit an automated block on craigslist’s end. They made some changes to reduce their traffic, and things started working again… until now. As far as I know there’s no official word from craigslist, but it seems likely that Feedly has simply grown to the point where they’re hitting the block again.

Unfortunately this means that at the moment RSS feeds from craigslist (like the ones from our RSS Feeds Tool) are not updating on Feedly. Most likely they will get this sorted out in the next few weeks, but if you don’t want to wait, there are a couple options. First off, other popular readers, such as NewsBlur and TheOldReader appear to be working for now, although as people move over from Feedly it likely won’t be long before they’re having the same problems.

A long-term solution, although with a bit more effort, would be to host your own RSS server. If you like NewsBlur, you can actually self-host it on your own server (or home computer), which will not only avoid getting caught up in these kinds of blocks, it will also save you the annual subscription fee. Code an installation instructions can be found on Github here.

Another popular option with a bit more detailed installation instructions is TinyTinyRSS. There’s a primer on MakeUseOf here. Whichever reader you choose, self-hosted or not, you should be able to import the OPML files generated by SearchTempest’s RSS Feeds Tool. Let us know how it goes in the comments!

Edit: It sounds like a number of people have been having trouble lately with self-hosted RSS as well. This thread at TheOldReader might shed some light. Apparently craigslist recently made a change to how they redirect RSS urls, which TheOldReader says isn’t supported by many other readers. So if your RSS reader isn’t picking up craigslist feeds, you might want to ask them to look into that.

Duplicate and distant results in RSS feeds

If you use craigslist RSS feeds (either directly or imported with one of our OPML files), you may have noticed an increase in duplicate results recently, as well as results that are outside your specified search area.

This is second-hand info, but it appears the problem is that craigslist has recently started including their ‘results from NEARBY cities’ in their RSS feeds. There are a couple problems with this. First, they are mixed in with the results from the main city, so even if you’re only in searching one location, you end up with a bunch of irrelevant results mixed in. If you have feeds from multiple cities (like with SearchTempest) though, it’s worse. Many different cities could include the same ‘nearby’ results, so you could end up with numerous duplicates of the same posts.

I’ve been told that craigslist is looking into how to fix this, so presumably it was an unintentional change, and hopefully it’ll be fixed soon. We have no actual affiliation with or inside knowledge of craigslist though, so really your guess is as good as ours.

If you get tired of waiting, or just like to tinker, it should be possible to de-duplicate your feeds yourself using Yahoo Pipes. Basically you would need a pipe that takes the OPML file from SearchTempest as an input, combines all the feeds within it, and then runs a sort and a unique filter. Here’s one that might be a good start. I haven’t tried it, but it looks like you could feed in the link to the SearchTempest OPML file (by right-clicking on our get rss button and saving the link instead of downloading the file). Then you would just need to add the Unique operator after the sort block. If you’ve never used Pipes before, here’s a tutorial that might help. It doesn’t do exactly what we want, but it should be enough to get the basic idea.

If anyone decides to try the Pipes solution, let us know in the comments! Otherwise, hopefully this bug gets squashed on craigslist’s end soon!

What’s the best way to search craigslist?

*Note that the Tempest Blog and SearchTempest.com are in no way affiliated with or endorsed by craigslist.

Craigslist clearly has a fantastic wealth of classified ads available in every conceivable category. However, their search capabilities are relatively limited. Fortunately, there are a number of useful tools out there to help with searching craigslist. Which one is best really depends on what you’re hoping to find. Here’s our completely biased rundown:

For the Locals

First off, if you’re really just interested in your own city, and you’re not shopping for an apartment or a car, your best bet is probably just to search craigslist directly. (And you’re probably not actually reading this…) In case you are reading though, when searching directly, keep their advanced search syntax options in mind. They can often come in handy, allowing you to combine multiple searches into one.

Globetrotters

On the opposite side of the spectrum, say you want to find something on craigslist, but you don’t care where in the world it is, or what category it’s in. In that case, your best bet is simply to use Google’s ‘site’ operator:

your search terms site:craigslist.org

One downside is that in many countries craigslist uses the country’s own top level domain instead of .org. (For example, craigslist uses .ca in Canada and .co.uk in the UK.) With the query above, Google won’t catch those. Of course, you can add them to your query, like this

your search terms site:craigslist.org OR site:craigslist.ca OR site:craigslist.co.uk

but that quickly becomes tedious if you really want to search everywhere. And of course, this method doesn’t let you narrow your search either; you’re basically stuck with results from (almost) everywhere and from every category, even if you’re only looking for, say, a motorcycle in your home state. That said, it can’t be beat for speed or simplicity.

Power Searchers

One option to have a bit more control over where you’re searching is the tool we came up with: SearchTempest.com. With SearchTempest, you can choose exactly which craigslist cities you want to search, in a couple of different ways. The easiest option is to enter your zip (or postal) code and the distance that you’re willing to travel. However, if you prefer you can also search by state, or even choose specific cities. You can select the category and subcategory to search, as well as only show results since a given date. And finally you can get results from eBay, Amazon, and Oodle (Facebook Marketplace) at the same time (but only if you want them).

Perfectionists

SearchTempest isn’t perfect though (although we certainly try!) Its default mode is powered by Google, so it should offer basically the same results as the Google method, just with a bit more flexibility in terms of where to search. One downside to this though, is that Google does occasionally miss craigslist posts, or at least takes a while to find some. (Most posts are found within minutes, but some take hours or longer to show up, especially in less popular cities and categories.)

If you really want to be sure to catch everything, especially if you’re searching for something obscure, using craigslist directly might be a better option. However, if you still want to check multiple cities, SearchTempest has an alternate mode called Direct Results that we hope offers the best of both worlds.

Basically Direct Results mode allows you to open results pages directly at craigslist.org, but adds a little navigation window to quickly flip through cities. So you’re getting results straight from the source, just as if you’d searched directly, but you save the effort of manually typing your search into each site. It still takes a bit longer than the Google-powered option, but you’re seeing exactly the same results as if you searched each city manually.

Serial Searchers

Now, if you tend to run the same searches repeatedly, you have a couple options to speed up that process. First of all, if you’re only checking a handful of cities, one great technique is simply to bookmark the craigslist results page for each city, and put all the bookmarks in a folder. Then you just use your browser’s option to open all bookmarks in that folder in tabs, and you’ll have the newest results for all the cities in your search.

If you prefer a bit more automation, craigslist supports RSS feeds, which are built for this sort of thing. Each craigslist results page has an orange RSS link, which you can click to add the feed to your reader of choice. (If you don’t have a favorite, we recommend Feedly. No affiliation – we’ve just found it to be the best free option so far.) You can create a folder for each search, and add the results feeds for all the cities you’re interested in. Then just click on that folder to view the new results from all of them in one place. It keeps track of which posts you’ve already seen too, so time is wasted retreading the same ground.

If you’d like to speed up the setup a bit, SearchTempest can help with that too. Just run a search at SearchTempest.com, then click the “Feeds for this search” link at the top-right. It will generate a file that you can import into Feedly (or your reader of choice) with the matching feeds for every city in your search, saving you the effort of adding them one by one.

Car Buffs and Vagabonds

While all the options above will work regardless of what you’re seeking, for some searches there are better alternatives. For apartments, Padmapper is a cool site that will show you results from craigslist and several other sites, all on a map as the name implies. Craigslist has also recently added a map option, but it’s not as slick as Padmapper (yet, anyway), and obviously lacks results from other sites. (We’re not affiliated with Padmapper; I just think it’s cool.)

If you’re looking for a car (or truck, or van…) we’d like to think AutoTempest.com (our other site) is worth a look. It ties into SearchTempest, so you get everything mentioned above, plus it adds results from all the other top car classifieds sites: AutoTrader (no affiliation), Cars.com, CarsDirect, Oodle, eBay Motors, and more. Also, it simplifies things like specifying model years. It is possible to specify a range of years using craigslist’s advanced keyword syntax, like this

1998|1999|2000|2001|2002|2003|2004

but with AutoTempest you don’t have to bother; we do it for you.

In Summary

  • Searching a single city? Just use craigslist.
  • Don’t care where, just want results fast? Google’s site: operator should do the trick.
  • Want to choose where to search? Try SearchTempest.com.
  • Want to get every result? Try RSS feeds if you’re running the same search a lot, or Direct Results mode otherwise.
  • Searching for a car? Try AutoTempest.com. An apartment? Padmapper.

Did I miss anything? Let us know which tools you use to search for classifieds in the comments!

How to make your craigslist For Rent or For Sale post not suck

There are a truly surprising number of really terrible housing for-rent and for-sale ads out there, on craigslist and other sites like it. Don’t make yours one of them! Here are some of the top things to avoid:

Bad Pictures (Or none at all!)

Aside from the title (which should be long and descriptive), the pictures are the first thing people will notice about your posting. Don’t post pictures that suck.

One picture (or two or three) is not enough!

If your ad contains one lonely picture (often of the laundry room or outside or something useless), I’ll pass. If you can, post pictures of every room! And the yard (front and back), plus shots of the exterior and views. But mostly the rooms. If you don’t have a shot of the kitchen, I’m going to assume it looks like this.

Show the house in the best light, literally.

Take pictures in the daytime, preferably on a sunny day. Use a nice camera. If you suck at taking pictures, have a photographer friend do it. If you’re selling your house or it’s an executive rental, hire a photographer to take pictures. Or at least listen to their advice.

Oh, and post the pictures at a decent resolution, and right-side-up for Pete’s sake. If you don’t know how to do that, get someone to help you. Here’s a rule of thumb: if you have to squint to make out the details in your pictures, they suck.

Non-descriptive Descriptions

Don’t post a one-line description! Here’s a post I just read on usedeverywhere.com: “amazing 4 bedroom, 3 bath, large garage great for workshop or storage, large lot”. (It had one picture of what looked like an attic.) That’s not even close to the worst description I’ve read, but even so, it’s one line. If the house is so ‘amazing’, why don’t you tell me something about it?!

Really though, it’s not enough to just tell something about it. You want to tell everything about the house/apartment/shed/whatever-it-is. At a bare minimum, number of bedrooms and bathrooms, square footage, garage size, appliances, yard size, any selling features like air conditioning, pool, recent renos, whatever, and a link to a Google Map of the location. But don’t stop there. Describe the house in detail. Give people enough information to really picture what living in the house would be like! You want them to already be planning who would sleep in what room and how their furniture would be laid out. The better someone can picture living there, the more they will want to live there. (Unless your place itself sucks. But even then, the people who do contact you will be legitimately interested and you won’t be wasting both your time.)

Price Confusion

First of all, state the price for crying out loud. Why do people leave that out? Gah! If you think it’s going to help your bargaining position or something, forget it. You’re losing half your potential buyers/renters right off the bat.

Second, put the price, and only the price, in the price field. If you have two rooms for rent, for $500 and $600, make two separate ads. Do not enter $500600 in the price field. It boggles my mind when people do this. Not only is it annoying, it also removes your ad from pretty much everyone’s search results, since most people are not looking for a five hundred thousand dollar room for rent. Plus, with two ads you get twice the exposure!

Also, don’t put $1 as the price, unless you are selling your house for one dollar. That’s little better than the people who spam a bunch of unrelated keywords at the bottom of their ad. Presumably you realize that if you don’t post a price at all, your ad won’t show up when people specify a maximum. (So you’re one step ahead of $500600 guy…) But you also know the price you’re asking is exorbitant (presumably, or you’d just post the actual price, right?) So you figure you’ll just game the system. Well, don’t. Most likely you won’t be able to rent (sell) your place while still keeping the price a secret, so you might as well spill the beans now. (And if you can’t post the price because you’re trying to list multiple things in one ad, see above.)

Going into Hiding

Be reachable. Put a phone number and an email address in the ad. Include your first name so people know who they’re calling or writing to. You could even write something like, “Feel free to call, text, or email whenever.” You want people to contact you. Make it as easy as possible!

A few more “Don’ts” for Good Measure

Don’t…

  • Write things like “No deadbeats” or “No loud noise after 10” or even “Serious inquiries only” in your ad.
  • Post an ad with nothing but a link to a post somewhere else. If you’re too busy to copy and paste your post, I’m too busy to read it. Linking to more or higher resolution pictures is fine if the site you’re posting on limits you there, of course.
  • Use abbreviations, like “p/m” for “per month”, or “w&d” for “washer and drier”. These aren’t old school newspaper classifieds where you’re paying by the word. Don’t make people decrypt your ad; write it out in plain English.

Summary

And there you have it. Follow those few simple guidelines and at least people won’t be cursing your name as they attempt to decipher your ad or avoid getting eye strain from your photos. That may not rent your apartment or sell your house on its own, but it’s certainly a start.

How to prevent Google hammering server for old linked CSE specifications

Google’s Linked CSE is a fantastic tool. It allows you to dynamically generate a custom search engine for each of your users, or even for each individual visit, based on any parameters available to your application. This functionality has been invaluable for SearchTempest.com as we use custom search engines to provide customized multi-city searches of craigslist (no affiliation).

The problem with this approach is that when you create a Google Custom Search Engine (CSE) with a linked specification file on your server, Google’s “FeedFetcher-Google-CoOp” bot requests that file in order to build the CSE. It then continues to regularly request the file, repeatedly, for at least a matter of months afterward, even if it is never again used by an actual user.

In our case, it got to the point where the majority of all requests for files from our web server were for useless, outdated Google CSE specification files. Unfortunately, once this is happening, it appears there is no way to stop it. The best you can do is to add a rule either the web server or, ideally, the firewall level to block these requests. (Currently we return a 410 ‘gone’ response in as few bytes as possible.)

However, there is a way to avoid getting into this situation in the first place. In short, Google CSE specification files should be served from disposable subdomains. For example, create a subdomain called gcrefs1. For convenience you can point it at the same directory as your main (www) site. In your CSE setup, tell Google to access the file at http://gcrefs1.example.com/filename. Then, after a period of time (once Google’s Feedfetcher bot is making too many requests to the file for your liking), simply create a new subdomain (say, gcrefs2), update your references to point to the new domain, and then remove the DNS entries for the old one.

Of course, it’d be nice if Google’s feedfetcher just respected robots.txt, or reacted properly to 410 responses, but given the usefulness of Custom Search Engines in general, I’ll take what I can get.

Update: It appears that Google ignores 410 responses, but not 301 responses. So by 301 redirecting an outdated cref file to null.html (for example), you should be able to convince them to stop requesting it. (Although the bot will run through each of its saved sets of request arguments one last time, since it sees each as a completely separate file.)