Azure DevOps (VSTS) extension no longer import coverage and unit tests automatically

Template for a good bug report, formatted with Markdown:

  • versions used: latest sonarcloud 1.5.1

  • error observed (wrap logs/code around triple quote ``` for proper formatting)

    the plugin looks for trx file under the “wrong” path (a different path than what is actually used).

I would expect the plugin to continue to check for trx file in the same (which seems “correct” according to docs) path but also under “agent temp” directory where the trx and coverage file seems to be saved currently.
it seems the VSTS team recently change test output directory se: https://github.com/Microsoft/azure-pipelines-tasks/issues/7975

2019-01-02T21:51:11.2305659Z 21:51:11.229  Fetching code coverage report information from TFS...
2019-01-02T21:51:11.2321799Z 21:51:11.231  Attempting to locate a test results (.trx) file...
2019-01-02T21:51:11.2919665Z 21:51:11.291  Looking for TRX files in: D:\a\1\TestResults
2019-01-02T21:51:11.2923403Z 21:51:11.291  No test results files found
  • steps to reproduce

setup an azuire pipelines (YAML) build of https://github.com/OpenRIAServices/OpenRiaServices and run it

1 Like

I’ve just run a build on one of my projects, and the test and coverage results seem to be correctly found:

2019-01-09T14:07:33.9623117Z 14:07:33.663  Fetching code coverage report information from TFS...
2019-01-09T14:07:33.9623193Z 14:07:33.665  Attempting to locate a test results (.trx) file...
2019-01-09T14:07:33.9623430Z 14:07:33.671  Looking for TRX files in: D:\a\1\TestResults, D:\a\1\s\TestResults
2019-01-09T14:07:33.9623520Z 14:07:33.671  The following test results files were found: D:\a\1\s\TestResults\VssAdministrator_fv-az24 2019-01-09 14_07_26.trx
2019-01-09T14:07:33.9623609Z 14:07:33.683  The following code coverage attachments were found from the trx files: D:\a\1\s\TestResults\VssAdministrator_fv-az24 2019-01-09 14_07_26\In\fv-az24\VssAdministrator_fv-az24 2019-01-09 14_06_54.coverage

, and therefore correctly reported in SonarCloud (see the results for my Sample .NET Project).

Can this come from missing or incorrect configuration of one of the tasks?

What version of the vstest task is you using, as you can se in the repo I provided I have been using v2 and the hosted (vs 2015) build image.

If you also look at the Microsoft github issue this is a (recent)known change to newer agents and it seems they don’t want users to use the test result directly

I’ve tried to change the vstest task back to v1 and then it works, but v2 does not work (which I currently “need” in order to solve some flaky tests).

If you work at sonarsource I highly recommend you to visit the github issue at microsoft repository (my first link) since at least some developer over there seems to have a goal of not having any build results accessible to other tasks (except by maybe downloading them from artifact)

Daniel, Will you please link that github post you found.

Sonar team, is there a recommendation to get coverage from vstest task?

@Ammo I think this is the same topic as When am running SonarCloud integration with VSTS in Code Coverage section it is showing 0.0%,please help me how to get the report and I think that a ticket should be created to fix this.

The github link is https://github.com/Microsoft/azure-pipelines-tasks/issues/7975 there are some hints to use rest api to download test attachments

Thanks for the hint Daniel, I created https://jira.sonarsource.com/browse/VSTS-179 to track this.

Do we have a solution here, I have pressure mounting to remove sonarcloud and use codecov, as sonar codecoverage is showing 0.0%, tried all options, seems a bug in sonar. Codecov just scans and shows the report in dashboard, i ran corbetura and it generates the xml file in the test, and i can see the code cov report, but cannot find this in sonar, any help , else need to go away from sonar…SAD…it failed

Update: we think we’ve identified the root cause of the issue. It isn’t specifically a problem with v2 of the VSTest task (we use VSTest v2 for our internal builds and we haven’t had any problems with coverage being uploaded). If the output for the VSTest v2 step contains the output Updated Run Settings: then we think you will be affected by this issue.

Instead, it depends on which properties you set for the test step. If you change some of the settings from the defaults (e.g. runInParallel, runOnlyImpactedTests, rerunFailedTests …) then the VSTest task will generate a runsettings file that changes the location of the test output files so the analyze step doesn’t find them.

Many thanks @Daniel-Svensson for the links and the pointers to your repo which greatly helped diagnosing the issue. FYI I forked your repo and tweaked the azure-pipelines.yml. Disabling a couple of the more advanced test settings stopped the VSTest task from generating a new runsettings file which meant the coverage data was uploaded successfully. Unfortunately, these are probably exactly the test settings you need to set to handle your flaky tests.

We’re considering a number of options to fix the bug. The one suggested by Microsoft (use the rest APIs to check for the test artefacts and re-download them to the build agent) is exactly what we do for the legacy XAML builds. In practice, it’s quite an ugly solution - the test artefacts need to be processed on the server before they are available via the Azure REST APIs, so if we try to query the REST APIs too soon then we get a response saying there are no artefacts. This gives us a problem - it might be the case that there really are no results, or it might be that there are test artefacts but they haven’t been processed yet. The solution we use for the legacy XAML builds is to poll the TFS server at intervals then timeout after a few retries. Ugly, not very reliable, and it slows down the build unnecessarily if there are in fact no test results. However, we might have to use a similar solution if we can’t find anything better.

Our aim is to release an new version of the extension at the end of this week. We hope it will contain a fix for this issue, or at least a workaround until we can come up with a complete fix.

Update on this thread: the ticket (VSTS-179) just got fixed and deployed (kudos to @duncanp!). Things should be OK by now. Please feel free to ping us on this thread if you are still experiencing problems.

Update again: a regression was found so we had to revert this change for now (VSTS-193). Hopefully we can provide a new fix soon.

Update: we’ve just release new versions of the SonarQube (v4.6.3) and SonarCloud (v1.6.3) extensions to address the coverage issue and the regression.

FYI the regression was due to this bug in the Scanner for MSBuild which has been fixed in version 4.6.2.

2 Likes

@duncanp I’m using VSO with the latest SonarCloud extensions. I still cannot see the codecoverage in sonarcloud. I use the VsTest task, and see that it writes it output to %drive%\work_temp\TR_%guidlike%. Do i need to alter something to make it work? I tried various things but can’t get the tests & codecoverage in results in sonarcloud