Web scraping

Question

Hello friends!
Last few days I was working on developement of some simple scraping codes in python. I was thinking if its possible to make some script challenge where lot of coders can join, post their codes and discuss about given solutions?
Other thing, is there any programer who did some web scripts that is willing to share it with us here? I will be very pleased.
I am asking this since data extraction and web scraping is very interesting branch of programming and I feel like we don't give enough attention to it!

Answer 1

I made a movie web scraper for kodi with python.
It uses one scraper to pull the movie informations and 2 scrapers to scrape the embedded source links.

There’s only two scrapers for the source links for now. But I could make 5. Make each run in a new thread and have one powerfull scraper.

Answer 2

A year ago, I created a profile list for a Sololearn contest, by web scraping the Sololearn profiles.

https://code.sololearn.com/WrHot3955gBr/?ref=app

Answer 3

i watch youtube tuturial videos a lot, there are so many Professional developers making helpful stuffs for free, but my wifi is crazy and I dont want to load the videos every time to waste data.

so its not much but a life saver shell script to download youtube videos for me

step 1. $sudo apt-get install youtube-dl

step 2: save the shell script written below in a folder where u want videos with file name
dwnld. sh

step 3: in same folder compile a list of links of all youtube videos you wanna download in a single go with file name
links. txt

step 4: run the script by typing
. /dwnld.sh

it basically saves me from doing
$youtube-dl - f 22 video _ link (enter)
hundred times and waiting till one file downloads to start for another

I do this everytime when I see a tuturial series, if anyone has better idea please suggest, coz right now the only boring task is copy paste the links.
https://code.sololearn.com/c0u56B1Z2wf5/?ref=app

Answer 4

Not mine but someone else's code.
https://code.sololearn.com/c9YHb9DfYHUn/?ref=app

When i was learning about scraping using nokogiri in ruby i used the below resource among others to learn.
https://thaiwood.io/screen-scraping-with-a-saw-a-nokogiri-tutorial-with-examples/