As I continue a scan of public library websites, normally using Bing to find them, I sometimes run into this “description” under the sitename and URL:
We would like to show you a description here, but the site you’re looking at won’t allow us.
A nicely nontechnical way of putting it.
Here’s how Google deals with the same site:
A description for this result is not available because of this site’s robots.txt – learn more.
DuckDuckGo? Well…it doesn’t show the obvious first result for the search (first in both Bing and Google)–at least not within the first 50 results. (Is it possible that DDG doesn’t index sites with that robots.txt setting?)
Blekko, which does show the library’s homepage as the first result, has precisely the same text as Bing.
So does Yahoo! (same first result, same text), but since Yahoo! uses Bing as a search engine, that’s not surprising.
All of this is just curious–and I find myself curious about two things:
- Why doesn’t DuckDuckGo return the library’s homepage anywhere within the first (very long) results page?
- Why do (a few) public library homepages set robots.txt so as to prevent descriptions?
Update July 6, 2013: I believe I know what’s going on with most of these, and it’s reminiscent of The Price is Right. See followup post.
