My first WebCrawler

Question

Decided to try my hand at a Web crawler today.

First one took all the links from a page and put them in a file. when done with that page it picks a link from that file and continues.

Second one takes all .jpg files and saves them to a folder.

Third one let's you pick a word and counts how many times it appears on a website. because we are lame, when we found words with 1 instance, me and the little women seek and find them.

Since nobody else cares about my programming adventures, figured I'd share

Answer 1

thats very interesting. u might motivate me to make a web crawler

Answer 2

nice job
didn't caught 2nd sent of 3 par

Answer 3

Very Nice. ^.^

Answer 4

only problem is i dont know much about reading from websites and stuff like that, so do u think you can show me what tutorials to start looking into

Answer 5

a webcrawler is a nice project, if you have some experience with web search engines. A lot of exciting aplications, an a lot of complexities too

Answer 6

ok thanks i will start there

Answer 7

Very nice, well done. :D

Answer 8

Do it! I very much enjoyed myself

Answer 9

https://m.youtube.com/watch?v=qfGthiqwaZo

This is the tutorial I watched, but his method didn't work. html2text didn't work for me at all, I think it's a python 2 vs 3 issue. urllib2 had same problem, but I built on the same principles and after some tinkering it is working excellent