Code Coverage Dropped Greatly when switch to xUnit.v3

SonarQube Cloud
, ,
1

We just migrated to xUnit.v3 and setup code coverage according this guide. Code Coverage with Microsoft Testing Platform > xUnit.net. The coverage in my Sonar project as dropped 10% and no longer aligns with what is reported in Azure Devops. I understand the coverage format changed from Open Cover but I was not expecting this drastic a change in the metric and the discrepancy from the reported value in Azure Dev Ops. Any help in understanding what is going on here would be appreciated. Thanks.

2

Hi,

Can you share a debug end log?

Share the Scanner for .NET verbose logs

  • Add /d:"sonar.verbose=true" to the…
    • SonarScanner.MSBuild.exe or dotnet sonarscanner begin command to get more detailed logs
      • For example: SonarScanner.MSBuild.exe begin /k:"MyProject" /d:"sonar.verbose=true"
    • “SonarQubePrepare” or “SonarCloudPrepare” task’s extraProperties argument if you are using Azure DevOps
      • For example:
        - task: SonarCloudPrepare@3
    inputs:
      SonarCloud: 'sonarcloud'
      organization: 'foo'
      scannerMode: 'dotnet'
      projectKey: 'foo_sonar-scanning-someconsoleapp'
      projectName: 'sonar-scanning-someconsoleapp'
      extraProperties: |
        sonar.verbose=true
  • The important logs are in the END step (i.e. SonarQubeAnalyze / SonarCloudAnalyze / “Run Code Analysis”)

Share the msbuild detailed logs

MsBuild.exe /t:Rebuild /v:d

or

dotnet build -v:d

 
Thx,
Ann

3

@ganncamp I’ll work on getting those logs. This looks like it might be related to the format of the coverage file, specifically that the xml format does not support branch coverage calculations. The only way to get this metric with the Microsoft Testing Platform is via cobertura which does not seem to be a supported format for Sonar. Seeing as how the dotnet ecosystem seems like it is moving towards MTP, are there plans for sonar to support this format? It looks like this has been raised a few times on this forum. For example, Use Cobertura Coverage Report for Sonarcloud. Thanks.

4

Thank you for your remark. This is definitely something that is on our list of features. I have recorded your post in support of supporting Cobertura for .NET.

Denis

1 Like
5

@ganncamp Is there a private location I can upload the logs to? Thanks.

6

Hi,

Feel free to redact them as necessary.

 
Ann

7

These are pretty large logs and given I don’t know exactly what you all are looking for it’s going to be difficult to redact them. Sonar’s policy can’t be that I post my companies private information publicly on this forum in order to receive support for an issue.

8

Hi,

Paid support is available if you’re interested.

This is a free and open community of users.

 
Ann

9

I am a Sonar Cloud Subscriber. This is the only link I could find to request support in my Sonar Cloud instance. The help icon at the top right only has links to documentation and this forum. Could you perhaps point me in the direction of where I could submit a private support request for this?

10

Hi,

Your best bet is to ping your sales contact to inquire about commercial support. If you have it, or establish it, they can point you in the right direction.

 
Ann

11

To my knowledge I was never assigned a sales contact. I’m going to be honest this is a pretty terrible way to do business. I pay for a product, there seems to be an issue with a product, and the remedy is to seek out someone in sales so I can pay more to make the product I paid for work. Can you provide an email address or url of where I can submit the request to find out who my sales rep is?

12

Hi,

contact@sonarsource.com.

Or… you could just redact your logs.

 
Ann

13

The Sonar analyze log alone is 36K lines. Redacting a file this large without potentially leaking something sensitive is not a risk I am willing to take. Your lack of regard for Sonar customers privacy and security has been noted. It is my sincere hope that the rest of your company does not share the same cavalier attitude as you in regards to your customers privacy and security. Thank you.

14

Is it not possible to setup a private thread to share the log files? Thank you.

15

@Colin I think we are touching on a larger issue for the community here and I am looking for some clarification. Could you please provide some guidance as I’m not quite sure how to proceed. It seems like I am not going to be helped if I don’t upload logs that have potentially sensitive information in them and are so large that redacting is not really an option. In the past and in other posts this has never been an issue, a private thread has been created to allow uploading the data required to troubleshoot a potential issue with the product and service while preserving privacy. It seems like this is not something that is going to be accommodated in this particular case and I am wondering why this is? Perhaps my previous comments were a bit harsh and may have ruffled some feathers. I apologize to @ganncamp if that is the case, but I still do feel that I as a customer am being put in a bad situation here. I guess what I am looking for here is clarification on what is an acceptable request to post to this forum and what requires paid commercial support in addition to the subscription fees I already pay. From my perspective this is a potential material defect in the software that needs investigation and really should be well within the scope of this community forum. I also feel that asking that my log data remain private is not an unreasonable request and that there are plenty of other cases on this forum where such a request has been accommodated for the good of both the customer and the sonar products and services. I look forward to your clarification and to working further with @ganncamp and @denis.troller to identify root cause of this issue and hopefully resolve it in a timely manner. Thank you.

16

Hey Greg,

Our default approach is transparency.

Sharing code or logs

When possible, we prefer that users share code/logs publicly, redacted as necessary, so that all users can benefit from these interactions in the Community. Our default is transparency.

However, sometimes it’s just not reasonable to post all the necessary material publicly, or even to redact it. In such cases, SonarSourcers may agree to accept your code or logs privately for further troubleshooting. You can identify SonarSourcers by the Sonar “waves” placed over their avatars.

We do this to streamline support for everyone and to benefit the wider community. Since this forum isn’t a paid support channel, handling of private requests is at our discretion and considered case by case.

If you require absolute confidentiality over your code or logs, please consider our commercial support offering, which is designed for such needs.

If your files are large, I suggest narrowing them down to the parts that seem relevant.

All of that said, I have the impression you’ve figured out the problem on your own.

You can confirm this by checking the Activity tab of your project, which will let you view metrics like branch coverage (did it drop to 0?), line coverage, and coverage (combined) over time.

18

@Colin Thanks for the quick response.

If you require absolute confidentiality over your code or logs, please consider our commercial support offering, which is designed for such needs.

We do not require absolute privacy, but not having to post all our test cases / coverage data to the public internet does not seem unreasonable. A private thread to upload logs to a sonar employee is more than adequate it just seems like this option is being avoided here and I cannot figure out why.

I reached out contact@sonarsource.com but have not heard anything back.

I’m still not really sure where the line is regarding commercial support and as I’ve stated earlier asking for more money to investigate defects in a subscription seems a little iffy to me. That being said I would love a response on my private inquiry and a quote for commercial support.

All of that said, I have the impression you’ve figured out the problem on your own.

If your files are large, I suggest narrowing them down to the parts that seem relevant.

Not really. This seems related to the file format but even the non branch coverage is not aligning with what is generated from the rest of the non sonar tooling. In general there seems to be something wrong with sonar’s processing of the coverage output format sonar.cs.vscoveragexml.reportsPaths. The only way I can think for you all to troubleshoot this would be to look at the coverage file and understand why it does no align with the coverage analytics produced by Azure DevOps. This would require uploading the entire coverage file, again exposing alot of private details about our software. I understand this may border on the commercial support but there still appears to be a potential problem in Sonar somewhere that should be looked at as a larger product improvement.

19

Perhaps coverage files are interesting here. But to be fair, Ann asked for build/analysis logs. Not your coverage files. :slight_smile:

Making a small reproducer project that clearly reproduces the issue without exposing any private data is also an option, if you are convinced it’s a bug in how the reports are interpreted.

I will say that the vast majority of the time, when issues like this are reported, SonarQube successfully parses the coverage report. Issues usually stem from the report itself, such as missing files or lines. I recommend comparing the new report to the previous one to identify any discrepancies, if you haven’t already done so.

20

Perhaps coverage files are interesting here. But to be fair, Ann asked for build/analysis logs. Not your coverage files. :slight_smile:

The build logs are ~442K lines long. The analysis logs 36K long. I have no idea what information in here is relevant to @ganncamp in this investigation. Further more almost every line of these logs is going to leak information about the structure of my source code.

I will say that the vast majority of the time, when issues like this are reported, SonarQube successfully parses the coverage report. Issues usually stem from the report itself, such as missing files or lines. I recommend comparing the new report to the previous one to identify any discrepancies, if you haven’t already done so.

The results and coverage files seem to be processed correctly by the Azure Pipeline Tasks and PublishTestResults@2and PublishCodeCoverageResults@2. They show coverage that is very close to the open cover formats I was using previously. It is only in Sonar Cloud that we see the issue. I’m not saying I couldn’t have made a mistake, but the evidence seems to point to a potential issue with Sonar. This seems like it should be enough to at least investigate by the experts. There should be a vested interest in determining if his is actually a defect in the software vs user error.

Thanks.

21

We are interested! But we also have rules in this Community to keep it manageable and in the spirit of community (not paid support).

Personally, I think this is your best bet if you don’t want to share any potentially private details in a public forum.