In reply to Dan.

Thanks for the tip although people should rarely be doing things that ignore the robots.txt file.