We're evaluating options for managing synthetic data generation jobs in our application and would appreciate feedback from anyone who has implemented something similar.

Current Environment
• Frontend: Vue
• Backend: Python Flask
• Platform: OpenShift
• CI/CD: Jenkins
• Synthetic data generation: SDV

Users can submit synthetic data generation requests through the UI. These jobs can be CPU and memory intensive and may run for several minutes. Our primary concern is controlling concurrency and resource utilization. For example, if 20 users submit requests at the same time, we don't want 20 SDV workloads starting immediately and competing for cluster resources. We'd like requests to be queued and processed according to configurable limits (e.g., only 2-5 jobs running simultaneously). We also need job status tracking for the UI (Queued, Running, Completed, Failed).

The three options I found during research are:

1. Kueue
2. Celery
3. Other scheduling/queueing platforms such as Argo Workflows, Temporal, Airflow, Prefect, Tekton, or Kubeflow.

Any advice on which approach has worked best in production for data-processing workloads, particularly from an operational simplicity and long-term maintainability perspective?