I have a question regarding pytest and specifically using coverage for multiple modules. I noticed that when the average test coverage threshold across multiple modules isn't met, pytest doesn't fail with a non-zero exit code, even though it should.

Details

My command:

❯ pytest --cov module1 --cov module2 --cov module3 --cov-report --cov-fail-under=75

I get the following result:

FAIL Required test coverage of 75.0% not reached. Total coverage: 74.79%

But the exit code remains zero:

❯ echo $?
0

As a result, my CI/CD pipeline passes just fine even though my coverage threshold isn't met. This causes main to have too low test coverage.

What I've tried

• Passing --cov=module1,module2,module3 instead of having one pytest call with three different --cov arguments. This causes the coverage report to fail altogether, because it says that CoverageWarning: Module module1,module2,module3 was never imported. I.e., it tries to import module1,module2,module3 as a single module and it fails, since they are separate modules.
• Passing the to-be-included modules in pyproject.toml under [tool.coverage.run] in the include argument. Problem persists: test coverage threshold failure is noted, but exit code is zero.
• Interestingly, when you test on each of the modules individually, you do get a non-zero exit code for the module that has too low coverage.
• On the internet, I can't really seem to find a working solution. Copilot isn't very useful here either.

Thanks for thinking along!