I’m working with a Python monorepo that contains several applications and shared libraries. Each environment (dev, staging, prod) requires different dependency constraints. For example:
devallows latest minor versionsstagingmust use pinned versions validated by QAprodrequires strict, security-scanned versions
The problem is that the CI/CD pipeline (GitHub Actions in this case, but the question is generic) should:
- Use one shared workflow, not three separate pipelines
- Install dependencies depending on the environment
- Avoid duplicating logic like:
- run: pip install -r requirements-dev.txt
- run: pip install -r requirements-staging.txt
- run: pip install -r requirements-prod.txt
- Install dependencies depending on the environment
- Avoid duplicating logic like:
Current attempt
I tried generating the requirements at runtime:
pip-compile requirements.in --extra dev
Or using environment variables:
- name: Install
run: pip install -r requirements-${ENV}.txt
But this creates issues:
- CI matrix becomes difficult to maintain
- Shared libraries inside the monorepo may require different environment scopes
- Security tools (pip-audit, safety) do not always resolve env-based constraints consistently
- In Docker builds, the constraints file used by the CI does not match what the runtime container resolves
Question
Is there a recommended pattern or architecture to manage environment-specific dependency constraints in a Python monorepo using a single CI workflow, while keeping dependency resolution deterministic and avoiding workflow duplication?
I’m especially interested in solutions involving:
- pip-tools
- pyproject.toml + dependency groups
- centralized constraints files
- monorepo build orchestration
- DevOps-friendly approaches to avoid drift between CI and local development
Any best practices or examples from real-world pipelines would be helpful.