Duplicate and distant results in RSS feeds

If you use craigslist RSS feeds (either directly or imported with one of our OPML files), you may have noticed an increase in duplicate results recently, as well as results that are outside your specified search area.

This is second-hand info, but it appears the problem is that craigslist has recently started including their ‘results from NEARBY cities’ in their RSS feeds. There are a couple problems with this. First, they are mixed in with the results from the main city, so even if you’re only in searching one location, you end up with a bunch of irrelevant results mixed in. If you have feeds from multiple cities (like with SearchTempest) though, it’s worse. Many different cities could include the same ‘nearby’ results, so you could end up with numerous duplicates of the same posts.

I’ve been told that craigslist is looking into how to fix this, so presumably it was an unintentional change, and hopefully it’ll be fixed soon. We have no actual affiliation with or inside knowledge of craigslist though, so really your guess is as good as ours.

If you get tired of waiting, or just like to tinker, it should be possible to de-duplicate your feeds yourself using Yahoo Pipes. Basically you would need a pipe that takes the OPML file from SearchTempest as an input, combines all the feeds within it, and then runs a sort and a unique filter. Here’s one that might be a good start. I haven’t tried it, but it looks like you could feed in the link to the SearchTempest OPML file (by right-clicking on our get rss button and saving the link instead of downloading the file). Then you would just need to add the Unique operator after the sort block. If you’ve never used Pipes before, here’s a tutorial that might help. It doesn’t do exactly what we want, but it should be enough to get the basic idea.

If anyone decides to try the Pipes solution, let us know in the comments! Otherwise, hopefully this bug gets squashed on craigslist’s end soon!