C++ unit test with FakeIt mock; Line out of range (template issue?)

SonarQube Server / Community Build
1

Template for a good bug report, formatted with Markdown:

  • Versions used (SonarQube, Scanner, Plugin, and any relevant extension)
    SonarQube 8.4.1 Developer Edition, CFamily Analyzer 6.11.0 (build 19130), Java 11 on build server, Cpp

  • Error observed (wrap logs/code around triple quote ``` for proper formatting)
    Added new unit test with a mock. This is header only and heavily based on templates.

##[error]ERROR: Error during SonarScanner execution
ERROR: Error during SonarScanner execution
##[error]java.lang.IllegalStateException: Line 105 is out of range in the file XXX.Test/XXXTest.cpp (lines: 104)
java.lang.IllegalStateException: Line 105 is out of range in the file XXX.Test/XXXTest.cpp (lines: 104)
##[error]at org.sonar.api.utils.Preconditions.checkState(Preconditions.java:61)
at org.sonar.api.utils.Preconditions.checkState(Preconditions.java:61)
##[error]at org.sonar.api.batch.sensor.coverage.internal.DefaultCoverage.validateLine(DefaultCoverage.java:93)
at org.sonar.api.batch.sensor.coverage.internal.DefaultCoverage.validateLine(DefaultCoverage.java:93)
##[error]at org.sonar.api.batch.sensor.coverage.internal.DefaultCoverage.lineHits(DefaultCoverage.java:81)
at org.sonar.api.batch.sensor.coverage.internal.DefaultCoverage.lineHits(DefaultCoverage.java:81)
##[error]at com.sonar.cpp.plugin.visualstudio.VisualStudioCoverageSensor.execute(VisualStudioCoverageSensor.java:76)
at com.sonar.cpp.plugin.visualstudio.VisualStudioCoverageSensor.execute(VisualStudioCoverageSensor.java:76)
##[error]at org.sonar.scanner.sensor.AbstractSensorWrapper.analyse(AbstractSensorWrapper.java:48)
at org.sonar.scanner.sensor.AbstractSensorWrapper.analyse(AbstractSensorWrapper.java:48)
##[error]at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:85)
at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:85)
##[error]at org.sonar.scanner.sensor.ModuleSensorsExecutor.lambda$execute$1(ModuleSensorsExecutor.java:59)
at org.sonar.scanner.sensor.ModuleSensorsExecutor.lambda$execute$1(ModuleSensorsExecutor.java:59)
##[error]at org.sonar.scanner.sensor.ModuleSensorsExecutor.withModuleStrategy(ModuleSensorsExecutor.java:77)
at org.sonar.scanner.sensor.ModuleSensorsExecutor.withModuleStrategy(ModuleSensorsExecutor.java:77)
##[error]at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:59)
at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:59)
##[error]at org.sonar.scanner.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:82)
at org.sonar.scanner.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:82)
##[error]at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:137)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:137)
##[error]at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:123)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:123)
##[error]at org.sonar.scanner.scan.ProjectScanContainer.scan(ProjectScanContainer.java:388)
…
##[error]22:33:29.605 Post-processing failed. Exit code: 1
22:33:29.605 Post-processing failed. Exit code: 1
##[error]The process ‘D:_2_tasks\SonarQubePrepare_15b84ca1-b62f-4a2a-a403-89b77a063157\4.17.0\classic-sonar-scanner-msbuild\SonarScanner.MSBuild.exe’ failed with exit code 1

  • Steps to reproduce
    Unit test with a mock using FakeIt 2.0.5. Test was passing successfully.
    The test file XXXTest.cpp has 104 lines.
    2 methods are replaced by stubs, 12 by Fakes, about 40 are unmocked (defaulted).
    Pure C++ solution (with Java Native Interface JNI methods), no C# or other languages
    Analysis on master branch (before adding unit test with mock) is successful.

  • Potential workaround
    not available yet

  • Scanner command used when applicable (private details masked)

  - task: sonarsource.sonarqube.15B84CA1-B62F-4A2A-A403-89B77A063157.SonarQubePrepare@4

    displayName: 'Prepare analysis on SonarQube'
    inputs:
      SonarQube: Sonarqube
      projectKey: XXX.Cpp
      projectVersion: '$(Build.BuildNumber)'
      # for mixed C++/C# solution the `SonarScanner for MSBuild` must be used!
      extraProperties: |
        sonar.projectDescription=XXX
        sonar.exclusions=**/*.java,**/*Parser.*
        sonar.projectBaseDir=$(SrcDir)
        sonar.cfamily.build-wrapper-output=$(SonarOutDir)
        sonar.cfamily.threads=4
        sonar.cfamily.cache.enabled=true
        sonar.cfamily.cache.path=$(Agent.WorkFolder)/_Cache
        sonar.cfamily.vscoveragexml.reportsPath=$(Common.TestResultsDirectory)/codeCoverage.xml
  - task: petergroenewegen.PeterGroenewegen-Xpirit-Vsts-Build-InlinePowershell.Xpirit-Vsts-Build-InlinePowershell.InlinePowershell@1
    displayName: 'Build solution with build wrapper'
    inputs:
      Script: |
        New-Item -Path $(SonarOutDir) -ItemType directory -Force
        Set-Location $(SrcDir)
        $(build_wrapper) --out-dir $(SonarOutDir) "$(msbuildPath)/MSBuild.exe" $(SrcDir)/XXX.sln /t:Rebuild /nologo /nr:false /t:"Clean" /p:FullRelease=true /p:platform=$(BuildPlatform) /p:configuration=$(BuildConfiguration) /p:VisualStudioVersion="16.0" /m
        if (!$?) {
          Write-Host "##vso[task.complete result=Failed;]DONE"
        }
  • ALM and CI system used:
    • Azure DevOps Server 2019.1.1, Visual Studio 2019.8.3, Language version stdcpp17
    • self hosted build server
2

Hi @milbrandt,

did you check the content of $(Common.TestResultsDirectory)/codeCoverage.xml ? Why does it contain an entry about line 105 for such file?

3

Hi @mpaladin

there are several blocks for the source_id of the file. One block regarding my test method which is fine:

        <function id="429152" name="getMeasureInputCommand_MapToString" namespace="XXX_test" type_name="XXXTest" block_coverage="73.67" line_coverage="64.29" blocks_covered="221" blocks_not_covered="79" lines_covered="9" lines_partially_covered="5" lines_not_covered="0">
      <ranges>
        <range source_id="52" covered="yes" start_line="19" start_column="0" end_line="19" end_column="0" />
        <range source_id="52" covered="yes" start_line="21" start_column="0" end_line="21" end_column="0" />
        <range source_id="52" covered="partial" start_line="22" start_column="0" end_line="22" end_column="0" />
        <range source_id="52" covered="partial" start_line="24" start_column="0" end_line="24" end_column="0" />
        <range source_id="52" covered="partial" start_line="26" start_column="0" end_line="26" end_column="0" />
        <range source_id="52" covered="yes" start_line="81" start_column="0" end_line="81" end_column="0" />
        <range source_id="52" covered="yes" start_line="82" start_column="0" end_line="82" end_column="0" />
        <range source_id="52" covered="partial" start_line="83" start_column="0" end_line="83" end_column="0" />
        <range source_id="52" covered="yes" start_line="86" start_column="0" end_line="86" end_column="0" />
        <range source_id="52" covered="yes" start_line="94" start_column="0" end_line="94" end_column="0" />
        <range source_id="52" covered="yes" start_line="97" start_column="0" end_line="97" end_column="0" />
        <range source_id="52" covered="yes" start_line="100" start_column="0" end_line="100" end_column="0" />
        <range source_id="52" covered="yes" start_line="101" start_column="0" end_line="101" end_column="0" />
        <range source_id="52" covered="partial" start_line="102" start_column="0" end_line="102" end_column="0" />
      </ranges>
    </function>

And several other blocks which are out of range of line number. They come probably from the included FakeIt templates. (All together 83 lines with source_id="52")

        <function id="1809028" name="LongLongSub" type_name="" block_coverage="0.00" line_coverage="0.00" blocks_covered="0" blocks_not_covered="5" lines_covered="0" lines_partially_covered="0" lines_not_covered="8">
      <ranges>
        <range source_id="52" covered="no" start_line="8204" start_column="0" end_line="8204" end_column="0" />
        <range source_id="52" covered="no" start_line="8206" start_column="0" end_line="8206" end_column="0" />
        <range source_id="52" covered="no" start_line="8215" start_column="0" end_line="8215" end_column="0" />
        <range source_id="52" covered="no" start_line="8219" start_column="0" end_line="8219" end_column="0" />
        <range source_id="52" covered="no" start_line="8221" start_column="0" end_line="8221" end_column="0" />
        <range source_id="52" covered="no" start_line="8224" start_column="0" end_line="8224" end_column="0" />
        <range source_id="52" covered="no" start_line="8227" start_column="0" end_line="8227" end_column="0" />
        <range source_id="52" covered="no" start_line="8228" start_column="0" end_line="8228" end_column="0" />
      </ranges>
    </function>

    <function id="308992" name="~MethodMockingContextBase&lt;short,unsigned short,short,short,short,iodbpsd *&gt;()" namespace="fakeit" type_name="MockImpl&lt;AbstractDllFanuc&gt;::MethodMockingContextBase&lt;short,unsigned short,short,short,short,iodbpsd *&gt;" block_coverage="0.00" line_coverage="0.00" blocks_covered="0" blocks_not_covered="1" lines_covered="0" lines_partially_covered="0" lines_not_covered="1">
      <ranges>
        <range source_id="52" covered="no" start_line="105" start_column="0" end_line="105" end_column="0" />
      </ranges>
    </function>
    <function id="309008" name="~MethodMockingContextBase&lt;short,unsigned short,short,short,unsigned short,unsigned short,unsigned short,iodbpmc *&gt;()" namespace="fakeit" type_name="MockImpl&lt;AbstractDllFanuc&gt;::MethodMockingContextBase&lt;short,unsigned short,short,short,unsigned short,unsigned short,unsigned short,iodbpmc *&gt;" block_coverage="0.00" line_coverage="0.00" blocks_covered="0" blocks_not_covered="1" lines_covered="0" lines_partially_covered="0" lines_not_covered="1">
      <ranges>
        <range source_id="52" covered="no" start_line="105" start_column="0" end_line="105" end_column="0" />
      </ranges>
    </function>
4

Hi @milbrandt,

is the file replaced with new content containing more than 104 lines during the build? Or is the report simply not valid?

5

Hi @mpaladin,

There is nothing replaced during build. How are included header lines counted?

#include "CppUnitTest.h"
#include <fakeit.hpp>

#include <XXX/XXX.h>
#include <unordered_map>
#include <YYY/YYY.h>

using namespace Microsoft::VisualStudio::CppUnitTestFramework;
/// docu of FakeIt mocking framework https://github.com/eranpeer/FakeIt/wiki/Quickstart
using namespace fakeit;

namespace XXX_test
{
	TEST_CLASS(XXXTest)
	{
	public:
		TEST_METHOD(getMeasureInputCommand_MapToString)
		{
			// Arrange
			Mock<XXX> mock_xxx;
			When(Method(mock_xxx, cnc_rdsyssoft3))
				.AlwaysDo([](unsigned short, short, short*, short*, ODBSYSS3* p5) ->short { p5->soft_id = 257;	return EW_OK; });
			When(Method(mock_xxx, cnc_getfigure)).AlwaysReturn(EW_REJECT);

			Fake(
				Method(mock_xxx, cnc_allclibhndl),
				Method(mock_xxx, cnc_freelibhndl),

   ... // several lines omitted

   } // line 104, end of namespace XXX_test

If required I can provide Sonarsource confidential the relevant files. But I assume that this issue will happen also with other mocks using FakeIt.

6

Hi @milbrandt,

what are you using to generate the xml code coverage report?

In the mean time, as a workaround you can exclude test directories and files from the analysis using sonar.exclusions property, see: https://docs.sonarqube.org/latest/project-administration/narrowing-the-focus/

7

Hi @mpaladin,

everything is running on self-hosted Azure DevOps pipelines agent.

      - task: VSTest@2
        displayName: 'Test Assemblies C++ Unit Tests'
        inputs:
          testAssemblyVer2: '**/$(BuildConfiguration)_$(BuildPlatform)/**/*test*.dll'
          searchFolder: '$(SrcDir)'
          runOnlyImpactedTests: false
          vsTestVersion: toolsInstaller # using latestStable
          runInParallel: false
          codeCoverageEnabled: false # this crashes the vstest.console (bad image format)
          testRunTitle: 'C++ Unit Tests'
          otherConsoleOptions: '/platform:x64 /Logger:trx /Enablecodecoverage'
          platform: '$(BuildPlatform)'
          configuration: '$(BuildConfiguration)'

      - task: petergroenewegen.PeterGroenewegen-Xpirit-Vsts-Build-InlinePowershell.Xpirit-Vsts-Build-InlinePowershell.InlinePowershell@1
        condition: and(succeeded(), eq(variables['AnalyseSolution'], true))
        displayName: 'Convert code coverage report'
        inputs:
          Script: |
            # download CodeCoverage
            & "$(Build.SourcesDirectory)/nuget.exe" install Microsoft.CodeCoverage -ExcludeVersion -OutputDirectory $(Build.BinariesDirectory)
            $codeCoveragePath = "$(Build.BinariesDirectory)/Microsoft.CodeCoverage/build/netstandard1.0/CodeCoverage/CodeCoverage.exe"

            # convert code coverage
            Set-Location $(Build.SourcesDirectory)
            & "$codeCoveragePath" analyze /output:$(Common.TestResultsDirectory)/codeCoverage.xml (Get-ChildItem TestResults/*/*.coverage)

My “workaround” was to discard MS C++ Unit Testing Framework and FakeIt for this project and switch to Google Test+Mock.
I disliked this, as all other projects and tests in the solution are using MS C++ Testing Framework and now there are two different styles to maintain :frowning: But at least it works.

8

Hi @milbrandt,

we cannot modify our plugin to import invalid coverage reports. Could you try to figure out what is happening? Why is the report containing some invalid line numbers?

9

Hi @milbrandt,

I created a ticket to track the issue: CPP-2879. We are going to ignore entries on exceeding line numbers, it is going to be released in the next version.

10

Thanks @mpaladin.

I n the meanwhile I created a very simple test project (abstract class with only 2 virtual methods) and there everything is ok even with Fake. Either it is something in our solution (403k LOC) or only with more abstract methods. If I can reproduce it with a simple project I’ll report it.

11

Hi @milbrandt,

I tried and for me it was enough:

Mock<SomeInterface> mock;
When(Method(mock,foo)).Return(1);

The templates constructions of Return leads to some implicit destructors being defined with line number the length of the file + 1.