Let's say I have a web scraping pipeline. For some reason, I use Celery to write a sanitized payload to the database. The payload is a list for rows.

Due to the nature of the problem, two Celery tasks may partially share rows.

Is there a way to merge two Celery tasks into one, where I can merge the rows and deduplicate?

Even if deduplication is not necessary, It will be nice to do bulk write.

Thank you.