Custom attributes to exclude from code coverage not recognized when applied at a class level in C#

[rasmuskl (Rasmus Kromann-Larsen)]

• Status: Paying customer 1M lines of code
• ALM used: Azure DevOps
• CI system used: Azure DevOps
• Scanner command used when applicable:

Relevant parts of Azure DevOps pipeline

- task: SonarCloudPrepare@2
  displayName: 'Prepare analysis on SonarCloud'
  inputs:
    scannerMode: 'MSBuild'
    SonarCloud: 'SonarCloud - ...'
    organization: ...
    projectKey: $(codeAnalysisProjectKey)
    projectName: $(codeAnalysisProjectName)
    extraProperties: |
      sonar.projectBaseDir=$(Build.SourcesDirectory)
      sonar.cs.dotcover.reportsPaths=$(Common.TestResultsDirectory)/dotCover.Report.html
      sonar.exclusions=MessageProcessor/Mailing/Templates/**

- script: |
    ./dotnet-dotcover test \
    --no-build \
    --logger trx \
    --no-restore \
    --results-directory './TestResults' \
    --dcXml=./dotCover.Settings.xml \
    --dcOutput="$(Common.TestResultsDirectory)/dotCover.Report.html" \
    --dcReportType="HTML" \
    --dcReturnTargetExitCode \
    -- xUnit.ParallelAlgorithm=aggressive
  displayName: 'Run Backend tests'

- task: SonarCloudAnalyze@2
  displayName: 'Run Code Analysis'
  continueOnError: true

- task: SonarCloudPublish@2
  displayName: 'Publish Quality Gate Result'
  continueOnError: true        

Relevant dotCover configuration:

<?xml version="1.0" encoding="utf-8"?>
<AnalyseParams>
    <AttributeFilters>
      <AttributeFilterEntry>
        <ClassMask>System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute</ClassMask>
      </AttributeFilterEntry>
      <AttributeFilterEntry>
        <ClassMask>(namespace).ExperimentAttribute</ClassMask>        
      </AttributeFilterEntry>
    </AttributeFilters>
    <ReportType>DetailedXML</ReportType>
    <ReturnTargetExitCode>true</ReturnTargetExitCode>
</AnalyseParams>

• Languages of the repository: C#
• Error observed (wrap logs/code around with triple quotes ``` for proper formatting)

We recently introduced a [Experiment] C# attribute that also excludes from coverage. However our code coverage Quality gates fails when using this even though both this attribute and the more standard [ExcludeFromCodeCoverage] applies correctly in produced coverage reports.

We have done some deeper analysis to figure out how it is failing. What we have found so far:

• When coverage exclusion attributes are applied at a method level both [Experiment] and [ExcludeFromCodeCoverage] attributes works correctly.
• When coverage exclusion attributes is applied at a class level, the file does not appear in the coverage report at all, which is expected behaviour from dotCover. However once this is published to SonarCloud, this is correctly recognized for [ExcludeFromCodeCoverage] attributes, however it does not apply at all for [Experiment] attributes.
• Sonar documentation says that the system only relies on coverage report, but it seems like there is a special case to handle [ExcludeFromCodeCoverage] attribute when the entire file is absent from the coverage report due to exclusion

Screenshot of [Experiment]attribute in classes not applying:

Screenshot of [Experiment] attribute working correctly with methods:

Screenshot of [ExcludeFromCodeCoverage] attribute in classes working:

• Potential workaround

The only workaround we have been able to find was to also add [ExcludeFromCodeCoverage] attributes to our experiments when applied at a class level, which is not ideal.

[Colin]

Hey there.

@rasmuskl

Sonar documentation says that the system only relies on coverage report, but it seems like there is a special case to handle [ExcludeFromCodeCoverage] attribute when the entire file is absent from the coverage report due to exclusion

Where are you getting this idea from? I’m almost certain we aren’t doing any special handling of [ExcludeFromCodeCoverage].

If the coverage report isn’t respecting your experimental property (and handling it differently than your Experimental class), SonarCloud isn’t going to either.

Have you confirmed in the raw coverage report whether or not your experimental attribute is having the desired effect at the class level?

If you show me two identical coverage reports, one using [ExcludeFromCodeCoverage] and one using your experimental attribute… then we have a problem.

[denis.troller]

Hi there, @rasmuskl, welcome back to the community!

While we do support [ExcludeFromCodeCoverage] in .NET, because it is a known attribute in the framework, we do not support arbitrary attributes configured through external tools.

The only thing I can recommend at this point is to make your attribute inherit from ExcludeFromCodeCoverageAttribute. I understand it is not exactly ideal, because it does not reflect the reality (you happen to want to exclude code marked with this attribute, it is not part of its function), and is probably more of a workaround. That being said, I do not really see another short-term option here.

Denis

[rasmuskl (Rasmus Kromann-Larsen)]

Thanks for the response and sorry for the late response.

From what I see the issue only happens for entire files that are excluded. I created a local coverage report with dotCover now against all my test classes.

• ExcludeClass excludes entire class with [ExcludeFromCodeCoverage]
• ExperimentClass excludes entire class with [Experiment]
• ExcludeMethodClass excludes a single method in the class with [ExcludeFromCodeCoverage]
• ExperimentMethodClass excludes a single method in the class with [Experiment]

I have a few more classes to test more exotic scenarios, but they don’t show anything interesting.

Running a coverage report on this entire assembly I see this:

Files where the entire class are excluded are not present in the coverage report at all, but the ones where only a method is excluded do appear, which is the expected behavior.

So far so good.

However, when these files are submitted to SonarCloud the ExcludeClass shows as properly covered while ExperimentClass is not.

This is what lead me to the conclusion that there would special treatment of [ExcludeFromCodeCoverage].

[rasmuskl (Rasmus Kromann-Larsen)]

Thanks for the suggestion - I did consider inheriting from [ExcludeFromCodeCoverage], but unfortunately it is sealed.

[Michael_Sendow]

@denis.troller

can recommend at this point is to make your attribute inherit from ExcludeFromCodeCoverageAttribute. I understand it is not exactly ideal, because it does not reflect the reality (you happen to want to exclude code marked with this attribute, it is not part of its function), and is probably more of a workaround. That being said, I do not really see another short-term option here.

I think the problem stems from that Sonar uses what they call:

“executable lines of code” which is used in addition to the coverage info to track lines covered/uncovered. All files excluded does not have any coverage info BUT using the executable lines of code will appear as un-covered on SonarQube.

The same problem you are experiencing is the same as setting:

	<ItemGroup>
		<AssemblyAttribute Include="System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute" />
	</ItemGroup>

in your project file. This is the recommended way by Microsoft to exclude entire assemblies from coverage but is unfortunately ignored by Sonar.

It is in my opinion a bug that should be fixed by Sonar. If a project has this assembly attribute all the underlying files should be excluded from codecoverage analysis.