I have a Bull queue setup in Node.js where a large file is split into individual pages and each page is inserted into the database via a queue. The split results in ~736 jobs which are chunked and pushed into an insert queue.

The issue is that 3 jobs are being processed twice, resulting in duplicate records in the database. What makes this unusual is that the **originals were inserted at 1:12 AM and the duplicates appeared at 3:40 AM — a ~2.5 hour gap.

SETUP:**

// Producer — splits jobs into chunks and pushes to queue
const jobChunks = _.chunk(allJobs, parseInt(process.env.CHUNK_SIZE)); // CHUNK_SIZE = 30

for (let chunk of jobChunks) {
    await insertQueue.add({ jobs: chunk }, { 
        removeOnComplete: true 
    });
}

// Queue configuration
const queue = new Bull('insert-queue', REDIS_URL, {
    settings: {
        maxStalledCount: 1,
        stalledInterval: 30000,
    },
    redis: {
        connectTimeout: 10000, // 10 seconds
    },
    defaultJobOptions: {
        timeout: 300000, // 5 minutes
    },
});

// Consumer — processes each chunk
queue.process(1, async (job, done) => {
    try {
        await insertRecordsInDb(job.data.jobs); // inserts 30 records per job
        done();
    } catch (error) {
        done(error);
    }
});

CHUNK_SIZE=30

REDIS_CONNECTION_TIMEOUT=10000

INSERT_CONCURRENT_PROCESS=1

INSERT_MAX_QUEUE=2**

What is Happening**

• Original 736 files are processed and inserted successfully at 1:12 AM

• At 3:40 AM, 90 duplicate records appear in the database

• The duplicates have no metadata or associated data — they appear as hollow records

• The issue only happens in production and not in UAT

• UAT and local both process exactly 736 records with no duplicates

Questions
1. What could be reason for the jobs being re-queued.
If being stalled then what are the reasons for that. And why the 2.5 hours gap ?