Category: ST – Features and News

SearchTempest news, plans, and thoughts

Where did the Separate Cities results go?

As you may know, the SearchTempest.com results are powered by Google and Bing.  Until recently we used a “Google Custom Search” to show results separated by city.  Unfortunately it appears that Google has now changed their policies, and permit only a single custom search per page, which means we are no longer able to show results for each city separately.  Unfortunately it also appears that when multiple cities are searched together, Google tends to miss more matching results.

There is some good news though!  We have just released a major update to our Direct Results mode, which allows you to get results directly from the source.  As well as giving you several options for how to group the results (by large areas, by state, or by individual city,) this also avoids the main problem with using Google/Bing results: delayed or missing listings.

In the past, Direct Results required you to click through each city individually, but we now combine multiple cities into each link, making it significant faster and easier to use.  (Although the old per-city Direct Results are still available if you want.)  You can learn more about Direct Results here.  In the coming days we will be making it new default view.  To give it a try now though, just flip the “Single List Results / Direct Results” toggle above the list of results to “Direct Results”.  Also, once we make it the default, you will still be able to switch back to the Google-powered Single List results using that same toggle.

We hope you like the new Direct Results!

Cities stuck “Retrieving Results from Google”

Update: Fix released

We have released an update that should make this significantly less of a hassle.  We now have built-in timers on in SearchTempest to avoid hitting this Google limit, so instead of being timed out and having everything break for five minutes, the worst you’ll experience is a few seconds delay (with a message explaining why).  This still might be an annoyance for larger searches, so we do have a couple other options for you as well.  Please see here for more info: http://support.searchtempest.com/hc/en-us/articles/226378827-Limit-on-number-of-cities-that-can-be-searched-in-a-short-time.

Original post

Some of you have probably run across this recently: view a few pages of results fine, and then all the cities just get stuck “Retrieving Results from Google”.  This issue just came up a few days ago, but what we’ve been able to determine so far is that Google have added a new limit to how many queries can be sent to their Custom Search API (which powers our results) in a short period of time.  We’re in the process of determining what those limits are, so that we can put our own limits in place to avoid hitting them.

For the moment, as long as you’re taking the time to read each page of results before moving on to the next, it’s unlikely you’ll have this problem.  Where it is more likely is if you’re searching for something rare, so a bunch of cities are skipped through quickly.  For now we suggest just taking your time with those searches.  Loading a chunk of 50 or so cities (including ones with no results), waiting 30 seconds, then loading the next chunk.  That should avoid hitting any limits.

Another option is to switch your sort order (using the “Sort” drop down on the results page) to one of the single list options.  That way a separate query doesn’t need to be sent to Google for each city, which will guarantee no limits are hit.

Again, we’re still actively working on this and hope to  have better work-arounds in place soon.  You can find more info here: http://support.searchtempest.com/hc/en-us/articles/205166587-The-results-load-slowly-

Default Results Sort Change

Update: There has been… significant… reaction in favor of sticking with Stacked Cities as the default, so we’re going to switch back, and look at ways to improve that view.  As always though, the sort order is entirely your choice, and can be changed using the “Sort” dropdown on the results page, just below the search bar.

SearchTempest is powered by a Google Custom Search.  Basically Custom Search is an API that Google offers, allowing other websites to run customized Google searches.  Like any search engine though, Google sometimes misses results, especially new ones.  We don’t like missing results, so recently we added Bing search as a backup option, to try to catch anything Google misses.

Unfortunately, it’s not possible to search each city separately with Bing like we do with Google.  We can search all the cities in your search area at once though, and show the results in a single list.  And we have found that for most searches, this Google + Bing combination returns better results than the “Separate Cities” searches we do with Google.  Therefore, we’re switching the default sort to a single list, including Google and Bing.

The “Separate Cities” sort options aren’t going away though!  If you preferred that way of viewing results, changing back is as easy as clicking the “Sort” dropdown box on the results page, just below the search bar.  We’re also looking at creating a sort of hybrid view that could offer the best of both worlds.

Finally, we also still do offer our Direct Results view, which doesn’t miss anything, so that might be of interest to some of you, especially if you like having the results separated by city.  You can learn more about Direct Results here.

Craigslist blocking Feedly again

It looks like craigslist is once again blocking Feedly from accessing its RSS feeds. This happened a few months ago when Feedly traffic apparently got high enough to hit an automated block on craigslist’s end. They made some changes to reduce their traffic, and things started working again… until now. As far as I know there’s no official word from craigslist, but it seems likely that Feedly has simply grown to the point where they’re hitting the block again.

Unfortunately this means that at the moment RSS feeds from craigslist (like the ones from our RSS Feeds Tool) are not updating on Feedly. Most likely they will get this sorted out in the next few weeks, but if you don’t want to wait, there are a couple options. First off, other popular readers, such as NewsBlur and TheOldReader appear to be working for now, although as people move over from Feedly it likely won’t be long before they’re having the same problems.

A long-term solution, although with a bit more effort, would be to host your own RSS server. If you like NewsBlur, you can actually self-host it on your own server (or home computer), which will not only avoid getting caught up in these kinds of blocks, it will also save you the annual subscription fee. Code an installation instructions can be found on Github here.

Another popular option with a bit more detailed installation instructions is TinyTinyRSS. There’s a primer on MakeUseOf here. Whichever reader you choose, self-hosted or not, you should be able to import the OPML files generated by SearchTempest’s RSS Feeds Tool. Let us know how it goes in the comments!

Edit: It sounds like a number of people have been having trouble lately with self-hosted RSS as well. This thread at TheOldReader might shed some light. Apparently craigslist recently made a change to how they redirect RSS urls, which TheOldReader says isn’t supported by many other readers. So if your RSS reader isn’t picking up craigslist feeds, you might want to ask them to look into that.

Duplicate and distant results in RSS feeds

If you use craigslist RSS feeds (either directly or imported with one of our OPML files), you may have noticed an increase in duplicate results recently, as well as results that are outside your specified search area.

This is second-hand info, but it appears the problem is that craigslist has recently started including their ‘results from NEARBY cities’ in their RSS feeds. There are a couple problems with this. First, they are mixed in with the results from the main city, so even if you’re only in searching one location, you end up with a bunch of irrelevant results mixed in. If you have feeds from multiple cities (like with SearchTempest) though, it’s worse. Many different cities could include the same ‘nearby’ results, so you could end up with numerous duplicates of the same posts.

I’ve been told that craigslist is looking into how to fix this, so presumably it was an unintentional change, and hopefully it’ll be fixed soon. We have no actual affiliation with or inside knowledge of craigslist though, so really your guess is as good as ours.

If you get tired of waiting, or just like to tinker, it should be possible to de-duplicate your feeds yourself using Yahoo Pipes. Basically you would need a pipe that takes the OPML file from SearchTempest as an input, combines all the feeds within it, and then runs a sort and a unique filter. Here’s one that might be a good start. I haven’t tried it, but it looks like you could feed in the link to the SearchTempest OPML file (by right-clicking on our get rss button and saving the link instead of downloading the file). Then you would just need to add the Unique operator after the sort block. If you’ve never used Pipes before, here’s a tutorial that might help. It doesn’t do exactly what we want, but it should be enough to get the basic idea.

If anyone decides to try the Pipes solution, let us know in the comments! Otherwise, hopefully this bug gets squashed on craigslist’s end soon!

Google not indexing craigslist – SearchTempest switches to Bing

As of February 28, Google has stopped indexing new craigslist posts. Or more specifically, every day between about 5pm and midnight PST, they index them as usual. Then at midnight, they throw them all away. So anyone searching Google for craigslist posts over the past couple weeks has been faced with a giant gap since the beginning of March.

SearchTempest has no affiliation with craigslist, so until recently, we used Google to power our searches. Since Google is no longer getting the job done though, we’ve switched to Bing!

To be honest, Bing’s API doesn’t hold a candle to Google Custom Search. You can’t sort by date, specify a list of urls to search (Google’s ‘annotations’), or even reliably search within the url at all. (Bing does have a semi-hidden option, instreamset:(url):{text}, which is similar to Google’s inurl:{text}, but we’ve found it to be unreliable.)

That said, through some clever manipulation of query strings and a mess of hard-coded special cases, we’ve managed to come up with a Bing-powered craigslist search that’s quite functional. If you’re frustrated by not being able to search craigslist through Google like before, give it a try!

We Want Your Input on the New SearchTempest.com Results Page!

Oct. 29th/2012 UPDATE: After a few minor tweaks based on your suggestions, we have pushed these changes to the live site. Please let us know what you think via comment on this post or posting to our Facebook page.

Based on the feedback we have received from you, our lovely users, we have re-done the SearchTempest.com results page! We have made things simpler, easier to use, and more intuitive. You can still do everything you could on the old site, with some new stuff too, it’s just packaged now in a easier to read and understand, and nicer to look at, package.

The new site is not live yet and that’s because we wanted to get your feedback on it so we can make it the best it can be before we make it live. You can start playing around with it at Beta.SearchTempest.com and you can learn more about the changes made below. Have thoughts on what could be improved or just want to tell us how awesome you think it is? Leave us a comment on our Facebook page.

Changes

  • The first thing you will probably notice is that the “Separate Cities”, “All Cities Together”, and “Direct Results” tabs are gone. Instead, we have settled on two modes: “Combined Results” and “Direct Results”. “Direct Results” looks and acts just like it used to. Since “Separate Cities” and “All Cities Together” provided the same information, just displayed differently (the former sorting results by cities, the latter by individual results), we have put those two in the same mode, which we call “Combined Results”. You can switch between the old “Separate Cities” and “All Cities Together” modes by choosing how you want to sort the results from the “Sort by:” dropdown. Select “Distance”, “State”, or “City Size” to sort by city (“Separate Cities”); choose “Most Recent” or “Best Match” to sort by individual results (“All Cities Together”).
  • You will also notice that we got rid of the upper tabs as well. “Modify Search” can now be called from the “Full Search” button in the search bar and “Display Options” has become the “Preferences” link in the results page.
  • We have generally cleaned up the page to make it less cluttered and easier to take in the information you need and leave out the stuff you don’t.

Date Filter

We have also added a new feature that we have had many requests for, and that is the ability to filter the results by the dates they were posted. You will see a new field with the label “Within the last # days:”. Pretty self-explanatory, but if you put the number 2 in there, you will only get results that are max 2 days old. Pretty awesome, right?

Please check out the new site and let us know how we can make it better for you!

SearchTempest Update

As mentioned in recent posts, craigslist has been making some changes lately that we have had to adapt to. (We have no affiliation with craigslist at all, so we don’t get any warning or help from their end when they change something; we just have to figure it out.)

In the past, we allowed you to open results from multiple craigslist cities at once, all on the same page using ‘iframes’. A couple months back, CL added some code to their site to prevent their pages being opened in iframes, so we had to find a new way. As a result we developed two new results modes:

  • Separate Cities – Similar in appearance to our old setup, but inside each city ‘frame’, instead of a results page straight from craigslist, we showed results from a third-party provider called 3taps. 3taps is an ‘exchange data API’, which got craigslist listings from the google cache, so they did not have to actually scrape craigslist.

  • Direct Results – The goal of this mode was to preserve the straight-from-craigslist feature of the frames. Since we could no longer link to craigslist results pages in frames, instead we created an interface whereby they could be conveniently opened in new windows. Essentially we open up a small navigation window on the left, with links to craigslist results that open in a larger window on the right. Since both remain visible at the same time, it lets you conveniently flip through results from multiple cities.

Recently there has been another development. craigslist has apparently instructed google to stop caching their results. (Note, that doesn’t mean their results have been removed from google, just that google doesn’t store copies of entire craigslist pages.) This effectively cuts off 3taps’ source for craigslist data. For us, it meant we needed to find yet another new way to do the Separate Cities mode.

So, what we’re doing now is powering that mode with a Google Custom Search. Basically this allows us to customize a Google search so it only returns results from certain sites. So we take your search at SearchTempest, then send a request to google with your keywords, along with a set of sites to search based on the zip code, distance, and category you entered. Google returns a bunch of results fitting those parameters, which we display for you. When you click on one of the results links, it your browser opens that posting right at craigslist in a new window.

There are a couple downsides to this google-powered approach. The main ones are that it does not support filtering by price or displaying prices in the results, and that you can’t filter by ads with pictures. Also, preview images only work if the ad was created with embedded html images, not the standard craigslist pics. If you need those features, we suggest using the Direct Results mode mentioned above, which should do everything you need!

So in summary, the Separate Cities mode is now powered by Google. It’s probably the most convenient option for most people. If you find yourself missing those price or image filters though, or if you just want to get your results straight from the source, rather than via google, you should give the Direct Results mode a try.

Next thing on our agenda, thanks to user feedback, is simplifying the results page. We’re going to be removing a bunch of tabs, consolidating the Separate Cities and All Cities results modes into one (since they’re both now powered by Google), and just generally making everything a lot more straightforward. Hope to have that ready for you soon!

Performance Improvements

We’ve just finished some pretty major changes behind the scenes to improve performance from the new 3taps-powered ‘Separate Cities’ results. If all goes well, you shouldn’t see any difference… except for a much snappier site. If you do run into any new problems, please feel free to let us know, and we’ll take care of them asap!