Python BeautifulSoup help

Question

I am building a web crawler and have a decent one built.
Would love some help to make it better. Ill link it to my github. The code won’t work unless you have all script pieces so this is just a small script to look over. Thanks!
https://github.com/Tasaddar12/crawler_2.0/blob/master/spider.py
I find it to be slower ever since I added the depth parameter check