Condition coverage not shown for `coverage.txt` generated by `llvm-cov show`

Hi,

I don’t see that coverage report generated by llvm-cov show contains any information related to condition coverage.

Here is a screenshot:

We generate report as follows:

llvm-cov show tests-coverage.x -instr-profile=tests/temp/tests.profdata -show-expansions -show-regions -show-line-counts -use-color=false > coverage.txt

Here is part of coverage.txt which includes region coverage information:

   34|  8.98k|    if (_isWaitingAuthentication && package->requiresAuthentication()) {
                                                  ^3.24k

Could you help with this?

Thanks!

Hi @manuyavuz-pointr,

I don’t really understand what is your problem and what you would like to achieve.

1 Like

Hi @mpaladin,

We would like to see condition coverage in our dashboard, and currently it’s not shown.

It only shows line coverage. However, I have verified that coverage.txt file includes information about condition coverage (region coverage in llvm-cov terminology).

Here is result for llvm-cov report command for example:

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Filename                                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover
TOTAL                                                                 15695              7027    55.23%        2944               915    68.92%       29242              9523    67.43%

Regions here correspond to Conditions in sonar terminology. We expect to see this to properly presented in sonarcloud dashboard.

Thanks

Hi @manuyavuz-pointr,

I confirm that currently we don’t support import of condition coverage for llvm cov format. Are using llvm cov on a linux project or on macos xcode project?

We use it in our makefile based cpp project which can be run both on linux and macos.

I am trying to use llvm-cov gcov to get gcov based report. But it’s still not clear to me how it performs name matching between generated .gcno and .gdca files and given source files. It looks for wrong path for generated coverage files.

What’s your suggested way to generate report in this case? It would have been great if we could directly use llvm-cov’s default coverage format, not gcov one.

Also, I suggest you to update your documents on test coverage regarding this limitation.

https://docs.sonarqube.org/latest/analysis/coverage/

Above guide does not include anything about condition coverage limitation of llvm-cov.

Hi @manuyavuz-pointr,

if you want condition coverage it would be indeed faster to generate gcov on your side.

I created a ticket to add support for condition coverage but no ETA and you are the first to ask about it: CPP-2340.

1 Like

@mpaladin, when I try gcov report, I get the following result from sonar-scanner:

12:24:08.369 DEBUG: 'gcov' skipped because one of the required properties is missing

This is what I have as sonar-scanner parameter:

-Dsonar.cfamily.gcov.reportPath=.

And I have .gcov files under project base dir in native gcov format generated by llvm-cov gcov.

What could be the problem?

Unfortunately I can’t paste full sonar-scanner output as it’s a private project. How do you proceed to support private projects for these cases?

Best

Hi @manuyavuz-pointr,

The property is sonar.cfamily.gcov.reportsPath (note the typo).

Hi @mpaladin,

Thanks so much, and sorry for taking your time for such a careless mistake.

I could collect branch coverage results form gcov reports now, but there is a problem with that, which is explained here in detail: https://stackoverflow.com/questions/42003783/lcov-gcov-branch-coverage-with-c-producing-branches-all-over-the-place

Basically, gcov treats compiler generated extra blocks (such as try catch blocks) for some cases as branches too, and assumes partial coverage for some weird code blocks such as the following:

## gcov sees below line as having 2 branches, 1 covered
static std::string versionString = "Unknown";

## gcov sees below line as having 4 branches, 3 covered
for (auto ball : balls) {
}

Do you know any workarounds for these type of issues?

Calculating condition coverage result this way is not really helpful for code coverage assessment. For reference, condition coverage shown by sonarcloud for gcov coverage report is 37%, whereas for llvm-cov's default format it’s around 58%.

Due to above limitations, it’s really an urgency for us to be able to use llvm-cov's native coverage format to get correct condition coverage percentages.

Hi @manuyavuz-pointr,

The only thing I can think of is to generate gcov format from the start without passing by llvm-cov format.

I cannot promise anything on this side, we will try to give a look.

The only thing I can think of is to generate gcov format from the start without passing by llvm-cov format.

I don’t think it’s related to llvm-cov because the stackoverflow link that I sent uses gcc’s native gcov tool, not llvm-cov.

Seems like we will have to revert back to native llvm-cov format, and wait for your side to implement the feature.

Is there any way to watch the corresponding JIRA task?

Thanks for your support!

Hi @manuyavuz-pointr,

I believe you should be able to watch the ticket if you register/login to jira.

Cool, I’m in!

Hi @mpaladin,

It’s been 5 month sine this request. Are there any plans to put this in schedule?

To be honest, llvm-cov support without region coverage is very limited, as we only know about line coverage in sonarcloud dashboard.

Hoping to hear some good news!

Hi @manuyavuz-pointr,

No plan yet, I am going to keep it in mind for the upcoming short future.

Hi @manuyavuz-pointr,

I gave a look at llvm-cov format output and I don’t think it can be used to report branch coverage. In order to report branch coverage, for a given line we need to know how many branches exist and how many branches are covered, such information is not available in it.

By looking at the easy example below you can see that branch coverage information is not available in llvm-cov, region coverage is just marking uncovered regions which is not at all equivalent to branch coverage. In the example below not all branches are covered but all regions are covered and llvm-cov report 100% coverage.

+ cat foo.cpp

int main() {
  int a = 0;
  int b = 1;
  if (a == 1 || b == 1) {
  }
  return 0;
}
+ clang -v
Apple clang version 11.0.3 (clang-1103.0.32.29)
Target: x86_64-apple-darwin19.4.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
+ clang++ -fprofile-instr-generate -fcoverage-mapping foo.cpp -o foo
+ LLVM_PROFILE_FILE=foo.profraw
+ ./foo
+ llvm-profdata merge -sparse foo.profraw -o foo.profdata
+ llvm-cov report ./foo -instr-profile=foo.profdata -use-color=0
Filename                       Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/tmp/llvm-cov-test/foo.cpp           5                 0   100.00%           1                 0   100.00%           7                 0   100.00%
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                5                 0   100.00%           1                 0   100.00%           7                 0   100.00%
+ llvm-cov show ./foo -instr-profile=foo.profdata -show-line-counts-or-regions -use-color=0
    1|       |
    2|      1|int main() {
    3|      1|  int a = 0;
    4|      1|  int b = 1;
    5|      1|  if (a == 1 || b == 1) {
    6|      1|  }
    7|      1|  return 0;
    8|      1|}

+ llvm-cov export ./foo -instr-profile=foo.profdata
+ cat foo.json
{"data":[{"files":[{"expansions":[],"filename":"/tmp/llvm-cov-test/foo.cpp","segments":[[2,12,1,true,true],[5,7,1,true,true],[5,13,1,true,false],[5,17,1,true,true],[5,23,1,true,false],[5,25,1,true,true],[6,4,1,true,false],[8,2,0,false,false]],"summary":{"functions":{"count":1,"covered":1,"percent":100},"instantiations":{"count":1,"covered":1,"percent":100},"lines":{"count":7,"covered":7,"percent":100},"regions":{"count":5,"covered":5,"notcovered":0,"percent":100}}}],"functions":[{"count":1,"filenames":["/tmp/llvm-cov-test/foo.cpp"],"name":"main","regions":[[2,12,8,2,1,0,0,0],[5,7,5,13,1,0,0,0],[5,7,5,23,1,0,0,0],[5,17,5,23,1,0,0,0],[5,24,5,25,1,0,0,3],[5,25,6,4,1,0,0,0]]}],"totals":{"functions":{"count":1,"covered":1,"percent":100},"instantiations":{"count":1,"covered":1,"percent":100},"lines":{"count":7,"covered":7,"percent":100},"regions":{"count":5,"covered":5,"notcovered":0,"percent":100}}}],"type":"llvm.coverage.json.export","version":"2.0.0"}

Hi @manuyavuz-pointr,

no need to post comments on the jira ticket, just keep the conversation in this topic, I’ll update ticket with relevant information. I am going to wait for your additional information before making any change to the ticket.

Hi there,

are there any updates on this topic?
There is a branch coverage information inside llvm-cov reports.

llvm-cov report ./foo -instr-profile=foo.profdata -use-color=0

generates:

Filename Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover Branches Missed Branches Cover
foo.cpp 5 0 100.00% 1 0 100.00% 7 0 100.00% 4 2 50.00%

So now it should be possible to add the branch coverage information to SonarQube.

in your example:
llvm-cov show ./foo -instr-profile=foo.profdata -show-line-counts-or-regions --show-branches=count

     1|  int a = 0;
     1|  int b = 1;
     1|  if (a == 1 || b == 1) {
  ------------------
  |  Branch (168:7): [True: 0, False: 1]
  |  Branch (168:17): [True: 1, False: 0]
  ------------------
      1|  }

llvm-cov export ./foo -instr-profile=foo.profdata

...
            "totals": {
                "branches": {
                    "count": 4,
                    "covered": 2,
                    "notcovered": 2,
                    "percent": 50
                },
...

Best regards,
Stefano :slight_smile:

Hello @StefanoRS and welcome to the community.

It seems it is your lucky day.

Branch coverage from llvm-cov has landed in SonarQube 10.0 and on SonarCloud.
You can have a look at an example there.

Cheers

1 Like