A text file placed in the root directory of a Web site that prohibits search engine spiders from indexing all or specific pages of the site. The Robots Exclusion protocol provides a format for designating which directories and files are off limits to the spidering program. See spider.

Download Computer Desktop Encyclopedia to your PC, iPhone or Android.

THIS DEFINITION IS FOR PERSONAL USE ONLY.
All other reproduction is strictly prohibited without permission from the publisher.
© 1981-2012 The Computer Language Company Inc.  All rights reserved.