It seems like code coverage is not affected if we remove tests, which actually causes coverage to drop. Since the coverage check only considers new lines of code, we get a false positive when the quality gate passes.
It is important for us to keep our coverage up and not have the ability to drop it by removing tests.