Standardized password-change path

Heartbleed is making me realize how much the internet needs a standardized way to change your password on websites. Right now if you want to change your password on a given site, the process looks something like

Go to homepage > Search for login option > search for account/settings/profile/options section > search for password change prompt > oh, looks like “account” was wrong… try “profile” > hmm.. nope… maybe it’s in a submenu somewhere > gah.. maybe I can google where their password change page is…….

How great would it be if there was a standard like www.example.com/password, the place to change your password on any website?

 

How Namecheap is preventing thousands from reaching our site

Until yesterday, searchtempest.com had used namecheap.com for both domain registry and DNS. They are one of the least expensive registrars out there that isn’t named GoDaddy, and generally have a good reputation, so this seemed like a reasonable choice. And DNS came free with domain registration, so we didn’t see any need to look elsewhere.

That was until several recent complete outages of their DNS servers. Now we don’t blame namecheap for that. Their business isn’t distributed DNS, and they certainly didn’t DDoS themselves. However, it did demonstrate our need for a more robust solution.

We settled on DNS Made Easy. They appear to provide a very robust, globally distributed, fast, user-friendly, and inexpensive solution. But this post isn’t about them. It’s about what happened when we tried to switch from namecheap’s internal DNS servers to the ones from DNS Made Easy.

The right way to transfer DNS is pretty straightforward, but it’s important that it be followed to avoid apparent downtime. Generally nameserver records (the locations of the nameservers themselves) are cached for 24 hours. So, when you want to change your nameservers without downtime, you just follow these steps:

  1. Configure the new nameservers with all necessary records.
  2. Point the domain at the new nameservers.
  3. Wait 48 hours* for the cache period to expire.
  4. Remove the records from the old nameservers.

*or whatever the TTL of the NS records is

This process is explained pretty succinctly in the first section here, for example:

But pay attention to the fact, that the NS records of your parent DNS servers are usually cached for 48 hours. Thus you should keep your old nameservers online for at least 48 hours after making the changes to your NS records.

The problem is, at namecheap, when we performed step #2, they immediately did #4: removing our records from their DNS. That means anyone who has accessed the site within 48 hours suddenly has a stale cache and is unable to get there again, unless they know to flush their dns, or wait 48 hours. (And if it’s their ISP that cached the DNS info, they have no choice but to wait.)

I immediately contacted Namecheap support, hoping that they could reinstate our records for the remainder of the 24 hour period, but they repeatedly gave me the canned (and incorrect) response that downtime is inevitable with DNS transfers, and I should simply wait 24 hours (apparently oblivious to the fact that a 24 hour outage of a busy website is kind of a big deal, and the fact that their NS records actually had a 48 hour TTL).

Eventually, after two fruitless rounds with namecheap “tech” support, I was able to establish that they should have preserved our records, and that it is in fact their policy to do so for a period of 5 days. However I now couldn’t convince them that this had not, in fact, happened.

Finally, with a bit of help from DNS Made Easy (which appears to have very competent tech support), we figured out the problem. Namecheap has two sets of nameservers, which they call “DNS v1” and “DNS v2”. The problems we had a couple weeks ago were with v2, so we switched to v1 at that point, while we sought out a more permanent solution. However, when we transferred yesterday, they preserved our records on their v2 servers (which we haven’t even been using for weeks!), but not on v1 where they need to be. I was finally able to explain this to the third namecheap tech I spoke to, who told me that the v1 servers are controlled by a separate provider, and there must be a problem on their end. She apparently sent them a ticket.

That was now 13 hours ago, with no resolution. I apologize profusely for the inconvenience users of searchtempest.com are suffering. Hopefully it’s some consolation that I’m at least as frustrated myself. If you’re unable to access www.searchtempest.com, you could try flushing your DNS cache. The easiest way to do that is to restart your computer. If that doesn’t help, unfortunately the only options are to call your ISP and ask them to flush the nameserver records for searchtempest.com from their cache, or to wait until the cache expires – potentially until tomorrow afternoon.

Otherwise, all I can do at this point is warn others to avoid the same pitfall. Go ahead and use namecheap for domain registration, but switch to an external DNS immediately, before your website has traffic. It is easily worth a few bucks a month to avoid these kinds of problems. If you’re already using their DNS services, it should be possible to transfer out without downtime, but make sure you’re on v2 before transferring out. And good luck.

Update:

I just followed up with Namecheap tech support for the fourth time, to ask why our records still haven’t been restored on their partner’s web servers. Unfortunately it sounds like the response they got from their partner was almost identical to the canned response they repeatedly gave me:

When you change nameservers for a domain name, these changes are not accepted instantly all over the world. It may take up to 24 hours (in rare cases more) for local ISPs to update their DNS cache, so that everyone can see your website. Since the caching time varies between ISPs, it takes time for DNS changes to be totally in effect. Unfortunately this process cannot be influenced or sped up because of its automated nature.

Once again ignoring the real problem. We know DNS propagation is not instantaneous. But if they leave the records on the old nameservers until the TTL (time to live) of the old NS (nameserver) records has passed, everyone will still be able to access the site while the propagation takes place. What’s more, according to at least one of the tech support reps I spoke with, that is in fact their policy. It’s becoming clear though that the cache period will have expired long before I will be able to find someone willing and able to make the 10 second change that would fix this problem.

Duplicate and distant results in RSS feeds

If you use craigslist RSS feeds (either directly or imported with one of our OPML files), you may have noticed an increase in duplicate results recently, as well as results that are outside your specified search area.

This is second-hand info, but it appears the problem is that craigslist has recently started including their ‘results from NEARBY cities’ in their RSS feeds. There are a couple problems with this. First, they are mixed in with the results from the main city, so even if you’re only in searching one location, you end up with a bunch of irrelevant results mixed in. If you have feeds from multiple cities (like with SearchTempest) though, it’s worse. Many different cities could include the same ‘nearby’ results, so you could end up with numerous duplicates of the same posts.

I’ve been told that craigslist is looking into how to fix this, so presumably it was an unintentional change, and hopefully it’ll be fixed soon. We have no actual affiliation with or inside knowledge of craigslist though, so really your guess is as good as ours.

If you get tired of waiting, or just like to tinker, it should be possible to de-duplicate your feeds yourself using Yahoo Pipes. Basically you would need a pipe that takes the OPML file from SearchTempest as an input, combines all the feeds within it, and then runs a sort and a unique filter. Here’s one that might be a good start. I haven’t tried it, but it looks like you could feed in the link to the SearchTempest OPML file (by right-clicking on our get rss button and saving the link instead of downloading the file). Then you would just need to add the Unique operator after the sort block. If you’ve never used Pipes before, here’s a tutorial that might help. It doesn’t do exactly what we want, but it should be enough to get the basic idea.

If anyone decides to try the Pipes solution, let us know in the comments! Otherwise, hopefully this bug gets squashed on craigslist’s end soon!

Skype Click to Call makes Firefox painfully slow

I’ve been having some troubles with Firefox lately, particularly when trying to use gmail. It’s been painfully slow (10-15 seconds just to change folders), and locking up the whole browser in the process. I decided to finally take my own advice and go through the troubleshooting steps we suggest people use when they’re having similar problems with SearchTempest.com or AutoTempest.com.

I found that if I restarted Firefox in safe mode (with all add-ons disabled), things were once again nice and snappy. Went from 10s+ to switch labels in gmail to around one second. Sweet. At first I figured the culprit must be something gmail-related, like PowerBot or Gmelius. (And I was pretty choked. Losing Gmelius wouldn’t be a big deal, but PowerBot is a huge productivity boost for me.) Fortunately though, the real culprit was an add-on I never use and didn’t even install: Skype Click to Call. It appears that this add-on is installed automatically when you install Skype. It’s probably an opt-in of some sort, and I imagine I must’ve figured it would be useful at the time, but for some reason, they failed to mention how it would make the browser an order of magnitude slower…

Anyway, it’s gone now, and gmail (and everything else) is fast once again!

My new workspace

In case anyone’s interested in a small glimpse into my life, I just finished the little details of setting up my new workspace (just moved a few weeks ago) and figured I’d share.

My Desk

OK, honestly, it’s because it’s the first time I’ve ever done proper cable management. Here’s how it looked before. 🙂 And yes, it’s always that clean now. … Yep. At least, when I take pictures of it, it is. Just don’t pan to the right or left at all. Or down. But otherwise, veeeery tidy.

The desk is a NextDesk Terra, which I don’t love everything about, but the huge surface and sit/stand adjustment are great. Monitors are BenQ BL2710pt, which I do love everything about. Except the OSD I suppose, but you pretty much use that once ever, so I can live with it. 😉 Great review of it here if you’re interested. PC is a Sandy Bridge home build from about three years ago in a Fractal Design Define case, which has been great. (I like things quiet, so probably spent more time picking out the case, fans, and PSU than the primary components…)

Oh, and the kid on the calendar and the Amazon gift thing on the desk is my daughter, Lila. In fact, that one deserves a closeup. 🙂

Christmas Elf

The toys you’d see on the floor if you had panned down before are (mostly) hers too.

And that’s about it! Welcome to my little world.

 

What’s the best way to search craigslist?

*Note that the Tempest Blog and SearchTempest.com are in no way affiliated with or endorsed by craigslist.

Craigslist clearly has a fantastic wealth of classified ads available in every conceivable category. However, their search capabilities are relatively limited. Fortunately, there are a number of useful tools out there to help with searching craigslist. Which one is best really depends on what you’re hoping to find. Here’s our completely biased rundown:

For the Locals

First off, if you’re really just interested in your own city, and you’re not shopping for an apartment or a car, your best bet is probably just to search craigslist directly. (And you’re probably not actually reading this…) In case you are reading though, when searching directly, keep their advanced search syntax options in mind. They can often come in handy, allowing you to combine multiple searches into one.

Globetrotters

On the opposite side of the spectrum, say you want to find something on craigslist, but you don’t care where in the world it is, or what category it’s in. In that case, your best bet is simply to use Google’s ‘site’ operator:

your search terms site:craigslist.org

One downside is that in many countries craigslist uses the country’s own top level domain instead of .org. (For example, craigslist uses .ca in Canada and .co.uk in the UK.) With the query above, Google won’t catch those. Of course, you can add them to your query, like this

your search terms site:craigslist.org OR site:craigslist.ca OR site:craigslist.co.uk

but that quickly becomes tedious if you really want to search everywhere. And of course, this method doesn’t let you narrow your search either; you’re basically stuck with results from (almost) everywhere and from every category, even if you’re only looking for, say, a motorcycle in your home state. That said, it can’t be beat for speed or simplicity.

Power Searchers

One option to have a bit more control over where you’re searching is the tool we came up with: SearchTempest.com. With SearchTempest, you can choose exactly which craigslist cities you want to search, in a couple of different ways. The easiest option is to enter your zip (or postal) code and the distance that you’re willing to travel. However, if you prefer you can also search by state, or even choose specific cities. You can select the category and subcategory to search, as well as only show results since a given date. And finally you can get results from eBay, Amazon, and Oodle (Facebook Marketplace) at the same time (but only if you want them).

Perfectionists

SearchTempest isn’t perfect though (although we certainly try!) Its default mode is powered by Google, so it should offer basically the same results as the Google method, just with a bit more flexibility in terms of where to search. One downside to this though, is that Google does occasionally miss craigslist posts, or at least takes a while to find some. (Most posts are found within minutes, but some take hours or longer to show up, especially in less popular cities and categories.)

If you really want to be sure to catch everything, especially if you’re searching for something obscure, using craigslist directly might be a better option. However, if you still want to check multiple cities, SearchTempest has an alternate mode called Direct Results that we hope offers the best of both worlds.

Basically Direct Results mode allows you to open results pages directly at craigslist.org, but adds a little navigation window to quickly flip through cities. So you’re getting results straight from the source, just as if you’d searched directly, but you save the effort of manually typing your search into each site. It still takes a bit longer than the Google-powered option, but you’re seeing exactly the same results as if you searched each city manually.

Serial Searchers

Now, if you tend to run the same searches repeatedly, you have a couple options to speed up that process. First of all, if you’re only checking a handful of cities, one great technique is simply to bookmark the craigslist results page for each city, and put all the bookmarks in a folder. Then you just use your browser’s option to open all bookmarks in that folder in tabs, and you’ll have the newest results for all the cities in your search.

If you prefer a bit more automation, craigslist supports RSS feeds, which are built for this sort of thing. Each craigslist results page has an orange RSS link, which you can click to add the feed to your reader of choice. (If you don’t have a favorite, we recommend Feedly. No affiliation – we’ve just found it to be the best free option so far.) You can create a folder for each search, and add the results feeds for all the cities you’re interested in. Then just click on that folder to view the new results from all of them in one place. It keeps track of which posts you’ve already seen too, so time is wasted retreading the same ground.

If you’d like to speed up the setup a bit, SearchTempest can help with that too. Just run a search at SearchTempest.com, then click the “Feeds for this search” link at the top-right. It will generate a file that you can import into Feedly (or your reader of choice) with the matching feeds for every city in your search, saving you the effort of adding them one by one.

Car Buffs and Vagabonds

While all the options above will work regardless of what you’re seeking, for some searches there are better alternatives. For apartments, Padmapper is a cool site that will show you results from craigslist and several other sites, all on a map as the name implies. Craigslist has also recently added a map option, but it’s not as slick as Padmapper (yet, anyway), and obviously lacks results from other sites. (We’re not affiliated with Padmapper; I just think it’s cool.)

If you’re looking for a car (or truck, or van…) we’d like to think AutoTempest.com (our other site) is worth a look. It ties into SearchTempest, so you get everything mentioned above, plus it adds results from all the other top car classifieds sites: AutoTrader (no affiliation), Cars.com, CarsDirect, Oodle, eBay Motors, and more. Also, it simplifies things like specifying model years. It is possible to specify a range of years using craigslist’s advanced keyword syntax, like this

1998|1999|2000|2001|2002|2003|2004

but with AutoTempest you don’t have to bother; we do it for you.

In Summary

  • Searching a single city? Just use craigslist.
  • Don’t care where, just want results fast? Google’s site: operator should do the trick.
  • Want to choose where to search? Try SearchTempest.com.
  • Want to get every result? Try RSS feeds if you’re running the same search a lot, or Direct Results mode otherwise.
  • Searching for a car? Try AutoTempest.com. An apartment? Padmapper.

Did I miss anything? Let us know which tools you use to search for classifieds in the comments!

How to make your craigslist For Rent or For Sale post not suck

There are a truly surprising number of really terrible housing for-rent and for-sale ads out there, on craigslist and other sites like it. Don’t make yours one of them! Here are some of the top things to avoid:

Bad Pictures (Or none at all!)

Aside from the title (which should be long and descriptive), the pictures are the first thing people will notice about your posting. Don’t post pictures that suck.

One picture (or two or three) is not enough!

If your ad contains one lonely picture (often of the laundry room or outside or something useless), I’ll pass. If you can, post pictures of every room! And the yard (front and back), plus shots of the exterior and views. But mostly the rooms. If you don’t have a shot of the kitchen, I’m going to assume it looks like this.

Show the house in the best light, literally.

Take pictures in the daytime, preferably on a sunny day. Use a nice camera. If you suck at taking pictures, have a photographer friend do it. If you’re selling your house or it’s an executive rental, hire a photographer to take pictures. Or at least listen to their advice.

Oh, and post the pictures at a decent resolution, and right-side-up for Pete’s sake. If you don’t know how to do that, get someone to help you. Here’s a rule of thumb: if you have to squint to make out the details in your pictures, they suck.

Non-descriptive Descriptions

Don’t post a one-line description! Here’s a post I just read on usedeverywhere.com: “amazing 4 bedroom, 3 bath, large garage great for workshop or storage, large lot”. (It had one picture of what looked like an attic.) That’s not even close to the worst description I’ve read, but even so, it’s one line. If the house is so ‘amazing’, why don’t you tell me something about it?!

Really though, it’s not enough to just tell something about it. You want to tell everything about the house/apartment/shed/whatever-it-is. At a bare minimum, number of bedrooms and bathrooms, square footage, garage size, appliances, yard size, any selling features like air conditioning, pool, recent renos, whatever, and a link to a Google Map of the location. But don’t stop there. Describe the house in detail. Give people enough information to really picture what living in the house would be like! You want them to already be planning who would sleep in what room and how their furniture would be laid out. The better someone can picture living there, the more they will want to live there. (Unless your place itself sucks. But even then, the people who do contact you will be legitimately interested and you won’t be wasting both your time.)

Price Confusion

First of all, state the price for crying out loud. Why do people leave that out? Gah! If you think it’s going to help your bargaining position or something, forget it. You’re losing half your potential buyers/renters right off the bat.

Second, put the price, and only the price, in the price field. If you have two rooms for rent, for $500 and $600, make two separate ads. Do not enter $500600 in the price field. It boggles my mind when people do this. Not only is it annoying, it also removes your ad from pretty much everyone’s search results, since most people are not looking for a five hundred thousand dollar room for rent. Plus, with two ads you get twice the exposure!

Also, don’t put $1 as the price, unless you are selling your house for one dollar. That’s little better than the people who spam a bunch of unrelated keywords at the bottom of their ad. Presumably you realize that if you don’t post a price at all, your ad won’t show up when people specify a maximum. (So you’re one step ahead of $500600 guy…) But you also know the price you’re asking is exorbitant (presumably, or you’d just post the actual price, right?) So you figure you’ll just game the system. Well, don’t. Most likely you won’t be able to rent (sell) your place while still keeping the price a secret, so you might as well spill the beans now. (And if you can’t post the price because you’re trying to list multiple things in one ad, see above.)

Going into Hiding

Be reachable. Put a phone number and an email address in the ad. Include your first name so people know who they’re calling or writing to. You could even write something like, “Feel free to call, text, or email whenever.” You want people to contact you. Make it as easy as possible!

A few more “Don’ts” for Good Measure

Don’t…

  • Write things like “No deadbeats” or “No loud noise after 10” or even “Serious inquiries only” in your ad.
  • Post an ad with nothing but a link to a post somewhere else. If you’re too busy to copy and paste your post, I’m too busy to read it. Linking to more or higher resolution pictures is fine if the site you’re posting on limits you there, of course.
  • Use abbreviations, like “p/m” for “per month”, or “w&d” for “washer and drier”. These aren’t old school newspaper classifieds where you’re paying by the word. Don’t make people decrypt your ad; write it out in plain English.

Summary

And there you have it. Follow those few simple guidelines and at least people won’t be cursing your name as they attempt to decipher your ad or avoid getting eye strain from your photos. That may not rent your apartment or sell your house on its own, but it’s certainly a start.

How to prevent Google hammering server for old linked CSE specifications

Google’s Linked CSE is a fantastic tool. It allows you to dynamically generate a custom search engine for each of your users, or even for each individual visit, based on any parameters available to your application. This functionality has been invaluable for SearchTempest.com as we use custom search engines to provide customized multi-city searches of craigslist (no affiliation).

The problem with this approach is that when you create a Google Custom Search Engine (CSE) with a linked specification file on your server, Google’s “FeedFetcher-Google-CoOp” bot requests that file in order to build the CSE. It then continues to regularly request the file, repeatedly, for at least a matter of months afterward, even if it is never again used by an actual user.

In our case, it got to the point where the majority of all requests for files from our web server were for useless, outdated Google CSE specification files. Unfortunately, once this is happening, it appears there is no way to stop it. The best you can do is to add a rule either the web server or, ideally, the firewall level to block these requests. (Currently we return a 410 ‘gone’ response in as few bytes as possible.)

However, there is a way to avoid getting into this situation in the first place. In short, Google CSE specification files should be served from disposable subdomains. For example, create a subdomain called gcrefs1. For convenience you can point it at the same directory as your main (www) site. In your CSE setup, tell Google to access the file at http://gcrefs1.example.com/filename. Then, after a period of time (once Google’s Feedfetcher bot is making too many requests to the file for your liking), simply create a new subdomain (say, gcrefs2), update your references to point to the new domain, and then remove the DNS entries for the old one.

Of course, it’d be nice if Google’s feedfetcher just respected robots.txt, or reacted properly to 410 responses, but given the usefulness of Custom Search Engines in general, I’ll take what I can get.

Update: It appears that Google ignores 410 responses, but not 301 responses. So by 301 redirecting an outdated cref file to null.html (for example), you should be able to convince them to stop requesting it. (Although the bot will run through each of its saved sets of request arguments one last time, since it sees each as a completely separate file.)

Google not indexing craigslist – SearchTempest switches to Bing

As of February 28, Google has stopped indexing new craigslist posts. Or more specifically, every day between about 5pm and midnight PST, they index them as usual. Then at midnight, they throw them all away. So anyone searching Google for craigslist posts over the past couple weeks has been faced with a giant gap since the beginning of March.

SearchTempest has no affiliation with craigslist, so until recently, we used Google to power our searches. Since Google is no longer getting the job done though, we’ve switched to Bing!

To be honest, Bing’s API doesn’t hold a candle to Google Custom Search. You can’t sort by date, specify a list of urls to search (Google’s ‘annotations’), or even reliably search within the url at all. (Bing does have a semi-hidden option, instreamset:(url):{text}, which is similar to Google’s inurl:{text}, but we’ve found it to be unreliable.)

That said, through some clever manipulation of query strings and a mess of hard-coded special cases, we’ve managed to come up with a Bing-powered craigslist search that’s quite functional. If you’re frustrated by not being able to search craigslist through Google like before, give it a try!