Robots txt disallow all
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:
The User-agent: * means this section applies to all robots. The Disallow: / tells the robot that it should not visit any pages on the site.
There are two important considerations when using /robots.txt: