What is an “Internet Spider” anyway?

The website  netlingo.com described Spider as

Synonymous with a crawler, this is a program that searches the Internet and attempts to locate new, publicly accessible resources, such as WWW documents, files available in public FTP archives, and Gopher documents. 

Also called wanderers or bots, spiders contribute their discoveries to a database that Internet users can search by using a search engine. Spider technology is necessary because the rate at which people are creating new Internet documents greatly exceeds any manual indexing capacity (although search directories prefer the manual approach).

But what does a “Spider” actually do and what part of a Website is it crawling around on anyway? The answer to the first part of this question is that a “Spider” looks over the code and content of a Webpage so that when a user goes to Google, Bing or any other Search Engine, results can be returned for the search term enquired.

The Second part, needs to go into a bit more detail of something called Meta Data (found in a tag) to partly understand.  Meta data is information that tells the Spider what the Website covers, where it’s information is aimed (after all you would not want to be sent to a service in Vancouver WA if you were looking for it in Vancouver BC) 

So you see, while this is a very basic explanation of Spiders, in the case of the World Wide Web, Spiders are your friends