Code coverage issues when using VSTest@2 with multiple agents in parallel

SonarCloud
, , ,
1
  • ALM used: Azure DevOps
  • CI system used: Azure DevOps
  • Languages of the repository: C#

Dear community,

We are having issues reporting code coverage to SonarCloud with a C# .NET 4.7.2 project. To speed up the pipeline testing time, we are using VSTest@2 along with the parallel strategy as described here: Run VSTest tests in parallel - Azure Pipelines | Microsoft Learn

Our abbreviated pipelines look like this:

azure-pipelines.yml

...

pool:
  vmImage: windows-2022

...

stages:
  - stage: ContinuousIntegration
    jobs:
      - job: Tests
        strategy:
          parallel: 6
        steps:
          - template: pipeline-templates/test-steps.yml

      - job: SonarCloud
        dependsOn: Tests
        steps:
          - template: pipeline-templates/sonarcloud-steps.yml
...

pipeline-templates/test-steps.yml

steps:
...

    - task: MSBuild@1
      displayName: 'Building the backend solution'
      inputs:
        msbuildVersion: 'latest'
        solution: 'src/backend/StihlConnect.sln'
        msbuildArchitecture: 'x64'
        platform: 'x64'
        configuration: 'Release'
        clean: true
        msbuildArguments: '/restore /m /nr:false /p:UseSharedCompilation=true'

...

    - task: VSTest@2
      displayName: 'Execute tests and measure code coverage'
      continueOnError: true
      inputs:
          distributionBatchType: 'basedOnTestCases'
          testSelector: 'testAssemblies'
          testAssemblyVer2: |
            **\*.Tests.dll
            **\*.IntegrationTests.dll
            !**\*TestAdapter.dll
            !**\obj\**
          searchFolder: '$(System.DefaultWorkingDirectory)/src/backend'
          vsTestVersion: 'latest'
          runInParallel: false
          codeCoverageEnabled: true
          rerunFailedTests: true
          rerunType: 'basedOnTestFailureCount'
          rerunFailedTestCasesMaxLimit: '10'
          rerunMaxAttempts: '1'

    - task: PublishPipelineArtifact@1
      displayName: 'Publish test result and coverage files'
      inputs:
          artifact: 'TestResults$(Agent.Id)'
          targetPath: '$(Agent.TempDirectory)\TestResults'

pipeline-templates/sonarcloud-steps.yml

steps:
    - task: DownloadPipelineArtifact@2
      displayName: 'Download all pipeline artifacts'
      inputs:
        targetPath: '$(Agent.TempDirectory)'

    - task: SonarCloudPrepare@2
      displayName: 'Prepare SonarCloud Analysis Configuration'
      inputs:
        SonarCloud: 'AzureDevOpsServiceConnectionName'
        organization: 'org'
        scannerMode: 'MSBuild'
        projectKey: 'projectKey'
        projectName: 'project'
        extraProperties: |
          sonar.verbose=true

    - task: SonarCloudAnalyze@2
      displayName: 'Run SonarCloud Code Analysis'

    - task: SonarCloudPublish@2
      displayName: 'Publish SonarCloud Quality Gate result'

We have serious problems getting the .trx and .coverage files on each of the 6 agents. As soon as VSTest@2 finishes, there are no files left in the temp folder. Setting resultsFolder on VSTest@2 to a different one doesn’t help either.

If we remove the parallel strategy and run the test sequentially on one agent, everything works as expected.

If I understand this correctly, our problem is with the VSTest@2 task and not with SonarCloud, but any hints would still be very appreciated. In Issues · microsoft/azure-pipelines-tasks · GitHub there are a few reports of similar problems.

This post might be about the same issue: SonarScanner for Azure DevOps with parallel jobs?

2

Hi,

I think this is the crux of your problem, and there’s not a lot we can do here.

 
Ann

3

I’ve found a solution: VSTest uploads the test results (.trx) and coverage reports (.coverage) from each agent as test run attachments, but doesn’t say so in the logs.

To get all test run attachments, we first need the test run id. This isn’t the build id from the URL, but it can be retrieved using the $(Build.BuildUri) using this endpoint: https://vstmr.dev.azure.com/$org/$project/_apis/testresults/runs

With the test run id, we can query this endpoint to get all attachment filenames and download URLs: https://vstmr.dev.azure.com/$org/$project/_apis/testresults/runs/:runId/attachments

See the docs at

$(System.AccessToken) can be used as a bearer token for authentication for these endpoints.

See Predefined variables - Azure Pipelines | Microsoft Learn for all available variables.

I created a script that downloads them all to $(Agent.TempDirectory)\TestResults and together with sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)\**\*.trx on SonarCloudPrepare everything is working as expected.

1 Like
4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.