Of all the titles I hope to never have in my life, “Hiring Manager” is right up there with “Sanitation Engineer”. So if you’re applying to our job posting, please, just call me Nathan. Or use our company name if you prefer. But please, no “Dear Hiring Manager.”
It looks like craigslist is once again blocking Feedly from accessing its RSS feeds. This happened a few months ago when Feedly traffic apparently got high enough to hit an automated block on craigslist’s end. They made some changes to reduce their traffic, and things started working again… until now. As far as I know there’s no official word from craigslist, but it seems likely that Feedly has simply grown to the point where they’re hitting the block again.
Unfortunately this means that at the moment RSS feeds from craigslist (like the ones from our RSS Feeds Tool) are not updating on Feedly. Most likely they will get this sorted out in the next few weeks, but if you don’t want to wait, there are a couple options. First off, other popular readers, such as NewsBlur and TheOldReader appear to be working for now, although as people move over from Feedly it likely won’t be long before they’re having the same problems.
A long-term solution, although with a bit more effort, would be to host your own RSS server. If you like NewsBlur, you can actually self-host it on your own server (or home computer), which will not only avoid getting caught up in these kinds of blocks, it will also save you the annual subscription fee. Code an installation instructions can be found on Github here.
Another popular option with a bit more detailed installation instructions is TinyTinyRSS. There’s a primer on MakeUseOf here. Whichever reader you choose, self-hosted or not, you should be able to import the OPML files generated by SearchTempest’s RSS Feeds Tool. Let us know how it goes in the comments!
Edit: It sounds like a number of people have been having trouble lately with self-hosted RSS as well. This thread at TheOldReader might shed some light. Apparently craigslist recently made a change to how they redirect RSS urls, which TheOldReader says isn’t supported by many other readers. So if your RSS reader isn’t picking up craigslist feeds, you might want to ask them to look into that.
Quick tip for other businesses out there. Regardless of what business you’re in, or what kind of website you have, if your site goes down for “maintenance” every night, you’re doing it wrong.
Oh, and if your website is “closed” during non-business hours, you’re really doing it wrong, although it appears the standard culprits there are government agencies.
(Sheesh, don’t people know 2am is a prime working hour for us entrepreneurs? )
Heartbleed is making me realize how much the internet needs a standardized way to change your password on websites. Right now if you want to change your password on a given site, the process looks something like
Go to homepage > Search for login option > search for account/settings/profile/options section > search for password change prompt > oh, looks like “account” was wrong… try “profile” > hmm.. nope… maybe it’s in a submenu somewhere > gah.. maybe I can google where their password change page is…….
How great would it be if there was a standard like www.example.com/password, the place to change your password on any website?
Until yesterday, searchtempest.com had used namecheap.com for both domain registry and DNS. They are one of the least expensive registrars out there that isn’t named GoDaddy, and generally have a good reputation, so this seemed like a reasonable choice. And DNS came free with domain registration, so we didn’t see any need to look elsewhere.
That was until several recent complete outages of their DNS servers. Now we don’t blame namecheap for that. Their business isn’t distributed DNS, and they certainly didn’t DDoS themselves. However, it did demonstrate our need for a more robust solution.
We settled on DNS Made Easy. They appear to provide a very robust, globally distributed, fast, user-friendly, and inexpensive solution. But this post isn’t about them. It’s about what happened when we tried to switch from namecheap’s internal DNS servers to the ones from DNS Made Easy.
The right way to transfer DNS is pretty straightforward, but it’s important that it be followed to avoid apparent downtime. Generally nameserver records (the locations of the nameservers themselves) are cached for 24 hours. So, when you want to change your nameservers without downtime, you just follow these steps:
- Configure the new nameservers with all necessary records.
- Point the domain at the new nameservers.
- Wait 48 hours* for the cache period to expire.
- Remove the records from the old nameservers.
*or whatever the TTL of the NS records is
This process is explained pretty succinctly in the first section here, for example:
But pay attention to the fact, that the NS records of your parent DNS servers are usually cached for 48 hours. Thus you should keep your old nameservers online for at least 48 hours after making the changes to your NS records.
The problem is, at namecheap, when we performed step #2, they immediately did #4: removing our records from their DNS. That means anyone who has accessed the site within 48 hours suddenly has a stale cache and is unable to get there again, unless they know to flush their dns, or wait 48 hours. (And if it’s their ISP that cached the DNS info, they have no choice but to wait.)
I immediately contacted Namecheap support, hoping that they could reinstate our records for the remainder of the 24 hour period, but they repeatedly gave me the canned (and incorrect) response that downtime is inevitable with DNS transfers, and I should simply wait 24 hours (apparently oblivious to the fact that a 24 hour outage of a busy website is kind of a big deal, and the fact that their NS records actually had a 48 hour TTL).
Eventually, after two fruitless rounds with namecheap “tech” support, I was able to establish that they should have preserved our records, and that it is in fact their policy to do so for a period of 5 days. However I now couldn’t convince them that this had not, in fact, happened.
Finally, with a bit of help from DNS Made Easy (which appears to have very competent tech support), we figured out the problem. Namecheap has two sets of nameservers, which they call “DNS v1″ and “DNS v2″. The problems we had a couple weeks ago were with v2, so we switched to v1 at that point, while we sought out a more permanent solution. However, when we transferred yesterday, they preserved our records on their v2 servers (which we haven’t even been using for weeks!), but not on v1 where they need to be. I was finally able to explain this to the third namecheap tech I spoke to, who told me that the v1 servers are controlled by a separate provider, and there must be a problem on their end. She apparently sent them a ticket.
That was now 13 hours ago, with no resolution. I apologize profusely for the inconvenience users of searchtempest.com are suffering. Hopefully it’s some consolation that I’m at least as frustrated myself. If you’re unable to access www.searchtempest.com, you could try flushing your DNS cache. The easiest way to do that is to restart your computer. If that doesn’t help, unfortunately the only options are to call your ISP and ask them to flush the nameserver records for searchtempest.com from their cache, or to wait until the cache expires – potentially until tomorrow afternoon.
Otherwise, all I can do at this point is warn others to avoid the same pitfall. Go ahead and use namecheap for domain registration, but switch to an external DNS immediately, before your website has traffic. It is easily worth a few bucks a month to avoid these kinds of problems. If you’re already using their DNS services, it should be possible to transfer out without downtime, but make sure you’re on v2 before transferring out. And good luck.
I just followed up with Namecheap tech support for the fourth time, to ask why our records still haven’t been restored on their partner’s web servers. Unfortunately it sounds like the response they got from their partner was almost identical to the canned response they repeatedly gave me:
When you change nameservers for a domain name, these changes are not accepted instantly all over the world. It may take up to 24 hours (in rare cases more) for local ISPs to update their DNS cache, so that everyone can see your website. Since the caching time varies between ISPs, it takes time for DNS changes to be totally in effect. Unfortunately this process cannot be influenced or sped up because of its automated nature.
Once again ignoring the real problem. We know DNS propagation is not instantaneous. But if they leave the records on the old nameservers until the TTL (time to live) of the old NS (nameserver) records has passed, everyone will still be able to access the site while the propagation takes place. What’s more, according to at least one of the tech support reps I spoke with, that is in fact their policy. It’s becoming clear though that the cache period will have expired long before I will be able to find someone willing and able to make the 10 second change that would fix this problem.
If you use craigslist RSS feeds (either directly or imported with one of our OPML files), you may have noticed an increase in duplicate results recently, as well as results that are outside your specified search area.
This is second-hand info, but it appears the problem is that craigslist has recently started including their ‘results from NEARBY cities’ in their RSS feeds. There are a couple problems with this. First, they are mixed in with the results from the main city, so even if you’re only in searching one location, you end up with a bunch of irrelevant results mixed in. If you have feeds from multiple cities (like with SearchTempest) though, it’s worse. Many different cities could include the same ‘nearby’ results, so you could end up with numerous duplicates of the same posts.
I’ve been told that craigslist is looking into how to fix this, so presumably it was an unintentional change, and hopefully it’ll be fixed soon. We have no actual affiliation with or inside knowledge of craigslist though, so really your guess is as good as ours.
If you get tired of waiting, or just like to tinker, it should be possible to de-duplicate your feeds yourself using Yahoo Pipes. Basically you would need a pipe that takes the OPML file from SearchTempest as an input, combines all the feeds within it, and then runs a sort and a unique filter. Here’s one that might be a good start. I haven’t tried it, but it looks like you could feed in the link to the SearchTempest OPML file (by right-clicking on our get rss button and saving the link instead of downloading the file). Then you would just need to add the Unique operator after the sort block. If you’ve never used Pipes before, here’s a tutorial that might help. It doesn’t do exactly what we want, but it should be enough to get the basic idea.
If anyone decides to try the Pipes solution, let us know in the comments! Otherwise, hopefully this bug gets squashed on craigslist’s end soon!
I’ve been having some troubles with Firefox lately, particularly when trying to use gmail. It’s been painfully slow (10-15 seconds just to change folders), and locking up the whole browser in the process. I decided to finally take my own advice and go through the troubleshooting steps we suggest people use when they’re having similar problems with SearchTempest.com or AutoTempest.com.
I found that if I restarted Firefox in safe mode (with all add-ons disabled), things were once again nice and snappy. Went from 10s+ to switch labels in gmail to around one second. Sweet. At first I figured the culprit must be something gmail-related, like PowerBot or Gmelius. (And I was pretty choked. Losing Gmelius wouldn’t be a big deal, but PowerBot is a huge productivity boost for me.) Fortunately though, the real culprit was an add-on I never use and didn’t even install: Skype Click to Call. It appears that this add-on is installed automatically when you install Skype. It’s probably an opt-in of some sort, and I imagine I must’ve figured it would be useful at the time, but for some reason, they failed to mention how it would make the browser an order of magnitude slower…
Anyway, it’s gone now, and gmail (and everything else) is fast once again!
In case anyone’s interested in a small glimpse into my life, I just finished the little details of setting up my new workspace (just moved a few weeks ago) and figured I’d share.
OK, honestly, it’s because it’s the first time I’ve ever done proper cable management. Here’s how it looked before. And yes, it’s always that clean now. … Yep. At least, when I take pictures of it, it is. Just don’t pan to the right or left at all. Or down. But otherwise, veeeery tidy.
The desk is a NextDesk Terra, which I don’t love everything about, but the huge surface and sit/stand adjustment are great. Monitors are BenQ BL2710pt, which I do love everything about. Except the OSD I suppose, but you pretty much use that once ever, so I can live with it. Great review of it here if you’re interested. PC is a Sandy Bridge home build from about three years ago in a Fractal Design Define case, which has been great. (I like things quiet, so probably spent more time picking out the case, fans, and PSU than the primary components…)
Oh, and the kid on the calendar and the Amazon gift thing on the desk is my daughter, Lila. In fact, that one deserves a closeup.
The toys you’d see on the floor if you had panned down before are (mostly) hers too.
And that’s about it! Welcome to my little world.
We made a change today that should result in far fewer expired posts showing up. There will likely still be some for reasons described here, but it should be significantly better than before.
*Note that the Tempest Blog and SearchTempest.com are in no way affiliated with or endorsed by craigslist.
Craigslist clearly has a fantastic wealth of classified ads available in every conceivable category. However, their search capabilities are relatively limited. Fortunately, there are a number of useful tools out there to help with searching craigslist. Which one is best really depends on what you’re hoping to find. Here’s our completely biased rundown:
For the Locals
First off, if you’re really just interested in your own city, and you’re not shopping for an apartment or a car, your best bet is probably just to search craigslist directly. (And you’re probably not actually reading this…) In case you are reading though, when searching directly, keep their advanced search syntax options in mind. They can often come in handy, allowing you to combine multiple searches into one.
On the opposite side of the spectrum, say you want to find something on craigslist, but you don’t care where in the world it is, or what category it’s in. In that case, your best bet is simply to use Google’s ‘site’ operator:
your search terms site:craigslist.org
One downside is that in many countries craigslist uses the country’s own top level domain instead of .org. (For example, craigslist uses .ca in Canada and .co.uk in the UK.) With the query above, Google won’t catch those. Of course, you can add them to your query, like this
your search terms site:craigslist.org OR site:craigslist.ca OR site:craigslist.co.uk
but that quickly becomes tedious if you really want to search everywhere. And of course, this method doesn’t let you narrow your search either; you’re basically stuck with results from (almost) everywhere and from every category, even if you’re only looking for, say, a motorcycle in your home state. That said, it can’t be beat for speed or simplicity.
One option to have a bit more control over where you’re searching is the tool we came up with: SearchTempest.com. With SearchTempest, you can choose exactly which craigslist cities you want to search, in a couple of different ways. The easiest option is to enter your zip (or postal) code and the distance that you’re willing to travel. However, if you prefer you can also search by state, or even choose specific cities. You can select the category and subcategory to search, as well as only show results since a given date. And finally you can get results from eBay, Amazon, and Oodle (Facebook Marketplace) at the same time (but only if you want them).
SearchTempest isn’t perfect though (although we certainly try!) Its default mode is powered by Google, so it should offer basically the same results as the Google method, just with a bit more flexibility in terms of where to search. One downside to this though, is that Google does occasionally miss craigslist posts, or at least takes a while to find some. (Most posts are found within minutes, but some take hours or longer to show up, especially in less popular cities and categories.)
If you really want to be sure to catch everything, especially if you’re searching for something obscure, using craigslist directly might be a better option. However, if you still want to check multiple cities, SearchTempest has an alternate mode called Direct Results that we hope offers the best of both worlds.
Basically Direct Results mode allows you to open results pages directly at craigslist.org, but adds a little navigation window to quickly flip through cities. So you’re getting results straight from the source, just as if you’d searched directly, but you save the effort of manually typing your search into each site. It still takes a bit longer than the Google-powered option, but you’re seeing exactly the same results as if you searched each city manually.
Now, if you tend to run the same searches repeatedly, you have a couple options to speed up that process. First of all, if you’re only checking a handful of cities, one great technique is simply to bookmark the craigslist results page for each city, and put all the bookmarks in a folder. Then you just use your browser’s option to open all bookmarks in that folder in tabs, and you’ll have the newest results for all the cities in your search.
If you prefer a bit more automation, craigslist supports RSS feeds, which are built for this sort of thing. Each craigslist results page has an orange RSS link, which you can click to add the feed to your reader of choice. (If you don’t have a favorite, we recommend Feedly. No affiliation – we’ve just found it to be the best free option so far.) You can create a folder for each search, and add the results feeds for all the cities you’re interested in. Then just click on that folder to view the new results from all of them in one place. It keeps track of which posts you’ve already seen too, so time is wasted retreading the same ground.
If you’d like to speed up the setup a bit, SearchTempest can help with that too. Just run a search at SearchTempest.com, then click the “Feeds for this search” link at the top-right. It will generate a file that you can import into Feedly (or your reader of choice) with the matching feeds for every city in your search, saving you the effort of adding them one by one.
Car Buffs and Vagabonds
While all the options above will work regardless of what you’re seeking, for some searches there are better alternatives. For apartments, Padmapper is a cool site that will show you results from craigslist and several other sites, all on a map as the name implies. Craigslist has also recently added a map option, but it’s not as slick as Padmapper (yet, anyway), and obviously lacks results from other sites. (We’re not affiliated with Padmapper; I just think it’s cool.)
If you’re looking for a car (or truck, or van…) we’d like to think AutoTempest.com (our other site) is worth a look. It ties into SearchTempest, so you get everything mentioned above, plus it adds results from all the other top car classifieds sites: AutoTrader (no affiliation), Cars.com, CarsDirect, Oodle, eBay Motors, and more. Also, it simplifies things like specifying model years. It is possible to specify a range of years using craigslist’s advanced keyword syntax, like this
but with AutoTempest you don’t have to bother; we do it for you.
- Searching a single city? Just use craigslist.
- Don’t care where, just want results fast? Google’s site: operator should do the trick.
- Want to choose where to search? Try SearchTempest.com.
- Want to get every result? Try RSS feeds if you’re running the same search a lot, or Direct Results mode otherwise.
- Searching for a car? Try AutoTempest.com. An apartment? Padmapper.
Did I miss anything? Let us know which tools you use to search for classifieds in the comments!