Making test coverage measures mode useful


(Stephen Palmer) #1

As a quality architect working with 30+ teams on several products, I want to see both unit test and integration tests coverage numbers and not a single, merged result so that I have some confidence in the testing being performed in our continuous integration/delivery pipelines.

Modern unit tests that mock out all dependencies and environmental factors and have been proven to run on a developers local machine cannot by definition fail in a continuous integration environment. This makes them very low value in a CI environment. Therefore, 100% unit test coverage but 0% integration test coverage means no meaningful testing is actually happening in the CI build.

In version 5.x of SonarQube we were able to see unit test coverage separate from integration test coverage (we use custom annotations to indicate which is which in our code).

In version 6.x we seem to have lost that ability (https://stackoverflow.com/questions/41785791/configure-sonar-to-see-integration-tests-v6-2).

Can we have this feature back, please?

fwiw: I know that test coverage only looks at path coverage and high-values are not a good indicator of actual coverage of tests or any indicator at all of the quality of tests … but very low values are an indication that someone its cutting corners on testing that will likely come back and bite us later.


(Werner Mueller) #2

I’ve got pretty much the same issue. We told developers to keep an eye on their test-pyramid (having mostly unit tests, fewer integration tests, even fewer system-integration tests and so on). Keeping an eye is only possible if we can measure it. Even if unit and integration-test coverage would not be calculcated together it would be an indicator on test-completeness (90% unit coverage, 98% integration coverage - don’t really care about “is the sum 98% or 99%?” - its enough to verify the test-pyramid and have a feeling on test completeness)

I understand sonarqube wanted to simplify this across different coverage reporters. Can’t say anything against that goal. But if a common number across languages is not possible don’t do one. Show one per language or module? Let users interpret and filter what you cannot state automatically.

This does feel a bit like over-simplification. At least for my test-pyramid monitoring.

Found this different functionality via Jira: https://jira.sonarsource.com/browse/MMF-345

Thanks!