R is for Robots, Spiders and Crawlers
Search engines, such
as Google, Yahoo, and MSN, use special programs to access your website,
evaluate it, and index it. These are variously known as robots, crawlers
and spiders. The metaphorical allusions tell us that it's an automatic
process and that it makes it way endlessly across the WWW.
Google's programme is called Googlebot. It makes it way through the
web searching out links (HREF and SCR links).
Yahoo doesn't seem to provide the equivalent information. I've had
to go to third parties, i.e. other websites, to find info about Yahoo's
ways of doing things.
MSN follows Google's example and info for website owners is easy to
get to from its frontpage - MSNBot.