By Letter: Non-alphabet | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z
  Email this page to a friend


Spider




(Or "robot", "crawler") A program that automatically explores the World-Wide Web by retrieving a document and recursively retrieving some or all the documents that are referenced in it.

This is in contrast with a normal web browser operated by a human that doesn't automatically follow links other than inline images and URL redirection.

The algorithm used to pick which references to follow strongly depends on the program's purpose.

Index-building spiders usually retrieve a significant proportion of the references.

The other extreme is spiders that try to validate the references in a set of documents; these usually do not retrieve any of the links apart from redirections.

The standard for robot exclusion is designed to avoid some problems with spiders.

Early examples were Lycos and WebCrawler.

Home (http://info.webcrawler.com/mak/projects/robots/robots.html).



< Previous Terms Terms Containing spider Next Terms >
SPG
SPGA
SPI
Spice Lisp
SPID
asterisk
bot
robot
spider
WEB
Spiderweb
spiffy
spike
spill
spin


Web Standards & Support:

Link to and support eLook.org Powered by LoadedWeb Web Hosting
Valid XHTML 1.0!Valid CSS!eLook.org FireFox Extensions