How to program python web crawler?

Answer 1

http://www.netinstructions.com/how-to-make-a-web-crawler-in-under-50-lines-of-JUMP_LINK__&&__python__&&__JUMP_LINK-code/

Here's another one.

http://code.activestate.com/recipes/576551/

I didn't verify these much but they look plausible.

Answer 2

i found scrapy framework

https://scrapy.org/

Answer 3

Cool. Scrapy's probably better than recursing with wget/curl (which just occurred to me). Here are some scrapy spiders written by someone on github:
https://github.com/dcondrey/scrapy-spiders

Answer 4

Thanks man