Why does my web crawler only work for selected websites?

Question

I built a web crawler based on the python tutorials of thenewboston and let the program output all the links on the main page of wikipedia which it did without any problems. I also tried letting the code output all caption of sections and a few other things. It all worked perfectly. But when I simple wanted to output the links on Amazon‘s main page it did nothing.
Is it Amazon not allowing me to do that or the modules I used? (I used the BeautifulSoup method from the pycharm build-in module bs4).

Answer 1

Try adding a header as suggested in this stackoverflow topic:
https://stackoverflow.com/questions/23555283/why-cant-i-scrape-amazon-by-beautifulsoup

Answer 2

thank you Im going to test that