standard for robot exclusion

World-Wide Web A proposal to try to prevent the havoc

wreaked by many of the early World-Wide Web robots when

they retrieved documents too rapidly or retrieved documents

that had side effects (such as voting). The proposed standard

for robot exclusion offers a solution to these problems.

.

(1996-03-23)