User Comments

I used to block index pages over at Gadgetopia and I still believe in this practice.

BUT — ad revenues were sucking wind and we get 90% of our traffic from search engines. We sold out.

Very interesting to see some of the big players in your blog roll haven’t cottoned on to this strategy yet.

If I remember correctly, I originally stumbled across this website some months back through a google search for something that I subsequently couldn’t find in your page content. However I liked the blog so much that I’ve been coming back ever since.

Good to know that you’ve recognised the problem and done something about it. I hope that like me, some of your readers were mis-directed to the site from a search engine but decided to bookmark it anyway. Occasionally these little Internet idiosyncrasies are blessings in disguise.

Funny,

I remember reading about this before, but hadn’t done anything about it. I would like the homepage of our blog to be indexed, so I didn’t bother ‘no-index’ing.

Now I think I may try some form of cloaking, perhaps leaving out the posts from the various bots but include a site description and perhaps the permalinks… As an experiment….

BTW - I’m still having trouble posting comments if I don’t pass a referer…

I used to block index pages over at Gadgetopia and I still believe in this practice.

BUT - ad revenues were sucking wind and we get 90% of our traffic from search engines. We sold out.

I don’t understand why your ad revenues would be affected by preventing the indexing of indices! Could you explain this a little further Deane?

Very interesting to see some of the big players in your blog roll haven’t cottoned on to this strategy yet.

Yes. It was for me too. I’m especially amazed that many of the websites that promote best standards, accessibility and search-engine “friendliness” actually fail in this most basic of principles.

I originally stumbled across this website some months back through a google search for something that I subsequently couldn’t find in your page content. However I liked the blog so much that I’ve been coming back ever since.

I’d suggest that you represent the exception rather than the rule James. Something you found encouraged you to stay awhile and, fortunately for me, you became a repeat visitor.

I am working under the assumption that the majority of visitors would react differently - that they’d be frustrated, perhaps even annoyed, that my website contradicted the search-engine results.

I think it’s safe to presume that a “frustrated” or “annoyed” visitor is not going to stay, they’re going to click “back” and be lost forever.

I would like the homepage of our blog to be indexed

Why?

Now I think I may try some form of cloaking, perhaps leaving out the posts from the various bots but include a site description and perhaps the permalinks.

You don’t have to do anything as complicated as cloaking Mike. Remember, the search-engine will “follow” the links to permanent content and then “index” that content at its permanent URI. Isn’t this what you want?

I’m still having trouble posting comments if I don’t pass a referer

Yes, sorry about this. I haven’t had time to work on the software at all. I’m probably not going to be able to make any coding changes now until after I return from France.

Hey dude, nice article. Makes me rush to my blog to see if i have the same problem :-)

Why? Don’t ask such hard questions ;-]

I went for the cloaking to avoid ‘letting users down’. If I cloak away the changing content I could avoid this.

Kind of ironic that Google prefers ‘changing content’ or frequently updated pages to more sedantary ones. It (I’m told) deems them as fresher and therefore more relevant. So on a relatively tightly themed blog, lets say about CSS, it would give the nod to the more regularily updated blog over, http://www.positioniseverything.net, all other things being more or less equal.

Google prefers ‘changing content’

This is the bit that I don’t get. I agree that Google seems to prefer fresh content. But I don’t see why proper use of the “noindex” tag is perceived as detrimental.

Google follows all the rules when it comes to “robot” control. Therefore, the search-engine will find the fresh content via the “follow” directive. Using “noindex” should have no effect on PageRank or positioning with regards to Google. However, it’s too early to tell if my recent adoption of the directive has altered Google’s perception of my website, so I can’t definitively state that there is not a detrimental affect.

Does anybody have any real stats on this? Anyone know for sure which method google ‘prefers’?

I don’t know for sure yet Jim. I expect it’ll be at least a couple of weeks before I’ll be able to plot some trends.

I do know that Google is still indexing even the deepest content of my website since I am still recording numerous hits from “crawlerXX.googlebot.com”.

Time will tell I guess.

Why not create a more user-friendly way of handling homepage visitors coming from search engines?

With a little PHP, you could easily identify search referrers, and add a preamble to the homepage for those visitors - something like:

“Were you looking for my post about ‘peanuts’? Try this blog entry from two weeks ago.”

Much more useful for visitors, plus you retain the benefit of Google thinking you have great frequently updated content.

That’s a really neat and elegant way of dealing with the problem Matthew.

The big problem is that there are several dynamic components that can affect search results and handling all the permutations in the way you describe would be tedious. Consider the following list of dynamic elements that have all been responsible for false-positive matches on my website (because they’ve changed after indexing):

• Weblog summaries (the obvious one)
• “Fresh Meat” (linkblog)
• blogroll
• “Currently Hearing” (iTunes “Now Playing” plug-in)
• “Currently Reading”

How would you deal with such a diverse range of possible matches?

the benefit of Google thinking you have great frequently updated content

I think this is a non-issue. Google can still “see” the new content (and the frequency of change) because it’s following the links from the dynamic index pages.

As I wrote in a previous comment, only time will tell if my changes have had a negative impact. Fortunately I am in the happy position of not caring a toss what the search-engines “think” of my website, so I can afford to experiment a little.

I tried it for a while, but then Google didn’t return the homepage of my site as the most relevant link for the search of my name, so I switched back to indexing. This is an example of needing search engines to pick up indexes, if the relevant piece of information is the index itself.

I guess that it all depends on your priorities and aims for your website.

• I do not care about PageRank
• I do not care whether searches for “Jonathan Hollin” yield this website or not
• I do not care whether http://blog.urbanmainframe.com/ (or http://urbanmainframe.com/folders/blog/) are linked to in search-engine results

I do care about my visitors. I absolutely do not want a visitor to follow a search-engine result to this website and then not see the content he expected.

Some external vindication: Help the Googlebot Understand Your Website…

And another one: Noindex!

NOTE: This comment has been edited by its author! Additions are shown in italics, deletions with a strike-through.

Update:

Well it’s been over a month since I posted this article and Google traffic to my website has not diminished at all. Just the opposite in fact, I’m getting more Google referrals than ever before - however, I do appreciate that this could simply be because I have more content now than I did then, so there is a greater possibility of a keyword match.

Conclusion:

Thus I can only conclude that the “noindex” tag on my indices has not had a detrimental effect on my search-engine positioning.

Case closed?