What exactly is a web crawler program?

Question

can someone explain to me what a web crawler is and how it is developed??

Answer 1

ahh well that makes a lot of sense now thank you

Answer 2

a crawler is a program, script or whatever that browse in the world wide wide in a automated way, a example of crawler is the google bot, the google bot browse in the world wide web for create a copy of the web pages to be processed for the google search engine later.
For example a spammer can use a web crawler for search e-mails in the WWW and later automatically send message for these!
PS: sorry for my bad english, i'm brazilian!

Answer 3

A web crawler is a piece of code written in a particular programming language (c, java, Phyton, php).
The main task performed by a web crawler is to extract the information from a single web page or multiple web pages. The information could be the title of the web page or the links contained by that particular web page or any other associated with that web page.
Google is a good example which uses web crawler.