Looking at scrapy's documentation for download_delay and Concurrent_requests, what I understand is: if I have 'CONCURRENT_REQUESTS': 25 and DOWNLOAD_DELAY of 1 second, that means that once scrapy requests 25 pages, scrapy will wait for all 25 to download, then wait for 1 seconds, and again request for 25 pages. If I am using download_delay of 0 sec, scrapy does same as above without 1 second delay. But when I used these settings in my scraper, this doesn't work as expected, as when I am 0sec delay, my network usage is 2.1Mbps average when I am using 0 sec delay compared to 1 sec delay when my network usage is just 0.3 Mbps average(0.4Mbps maximum). Here is my code:

class DetailsxxxSpider(scrapy.Spider):
    name = "details"
    allowed_domains = ["www.xxx.com"]
    download_delay = 1
    custom_settings = {'CONCURRENT_REQUESTS': 25}
    def start_requests(self):
        engine = create_engine('sqlite:///temp.db')
        Base.metadata.bind = engine
        DBSession = sessionmaker(bind=engine)
        session = DBSession()
        urls = session.query(Temp.url).filter_by(status = "Insert").all()
        # print urls
        for url in urls:
            yield Request(url[0])

Crawl rate is 1800 pages/minute average when using 0 second delay, and when using 1 second delay is 43 pages/minute though the pages/minute rate ideally should be 900 pages/minute. It seems like download_delay is being set up per request when using 1 second delay.

I am using proxies for every request I make. What am I missing? Is there a way that I can set Delay after 25 requests to be a 1 second. I ideally want the speed of 900 page/minute. Any help will be appreciated.

Thanks