New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Does this plugin support redirected url? #43

Open

mtabbasi opened this issue Aug 20, 2021 · 1 comment

mtabbasi commented Aug 20, 2021

I ran this plugin for https://quotes.toscrape.com/ but It redirects every requests and this plugin doesn't work for redirects

Collaborator

pawelmhm commented Sep 17, 2021

I get all HTTP 200 responses for quotestoscrape. Are you using quotesbot project?

But when I test for redirects I see it actually doesn't filter out properly. Because it checks for response.request here: https://github.com/scrapy-plugins/scrapy-deltafetch/blob/master/scrapy_deltafetch/middleware.py#L85 instead of response.meta['redirect_urls'] which should be proper value for redirected items.

So it won't work for redirects @mtabbasi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment