Google indexing hidden pages

Not that long ago, it used to take a bit of work and time to get indexed by Google.  Not anymore, it seems.

Like many web development companies, we use a development server to build our sites, then when pages are ready we 'stage' them on a client's site so they can see what they look like.  

The staging usually just involves putting the page(s) in a location that is not linked to by any of the publicly available pages.  That way, the theory goes, the search engines will not know about the new pages - only the client will.

This approach has been turned on it's head by Google, though.  We have had a situation where pages under development were indexed by Google.

How could this have happened?  Well, Google aren't saying, but the general industry wisdom is that it may have happened because of information collected by Google from the Google toolbar, the Chrome browser or even the use of Gmail.   Yahoo and Bing of course have not indexed any of the hidden pages.

So the lesson is, if you're staging new pages on a public site you should protect them from view.  That can be done in a number of ways, with varying levels of protection, for example:

• .htaccess access protection
• robots.txt file
• noindex