scrapy-redis

What people said

• “I’m biased since I’m an owner of a web scraping agency. I was asking myself the same question in 2019. You can use any programming language, but have settled on this tech-stack Python, Scrapy, Redis, Pos…”

• “If you are a programmer, scrapy will be a good bet. It can handle robots.txt, request throttling by ip, request throttling by domain, proxies and all other common nitty-gritties of crawling. The only drawback is handling pure javascript sites. We have to manually dig into the…”