Since the update to Community Build v25.1.0.102122 the count of unit tests has dropped signifcantly in our analysis, from 17K to just 399.
As pointed here: New .NET analyzers released this may be something expected, I think the problem is elsewhere.
Version Community Build v25.1.0.102122 MQR Mode
Running using Docker image
Running analysis in using Github Action
Runner logs:
2025-02-06T04:03:02.6969507Z SonarScanner for MSBuild 9.0.2
2025-02-06T04:03:02.6971314Z Using the .NET Core version of the Scanner for MSBuild
2025-02-06T04:03:05.3638209Z Pre-processing started.
2025-02-06T04:03:05.4861239Z Preparing working directories...
2025-02-06T04:03:05.6670793Z 05:03:05.666 Updating build integration targets...
2025-02-06T04:03:09.4521784Z 05:03:09.452 Using SonarQube v25.1.0.102122.
2025-02-06T04:03:10.9021376Z 05:03:10.901 The JRE provisioning is a time consuming operation.
2025-02-06T04:03:10.9022063Z JRE provisioned: OpenJDK17U-jre_x64_linux_hotspot_17.0.13_11.tar.gz.
Logs where the .trx files are processed:
2025-02-06T05:24:31.2521189Z INFO: Sensor C# Unit Test Results Import [csharp]
2025-02-06T05:24:32.1388234Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.Module1.trx'.
2025-02-06T05:24:32.1693375Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.Module2.trx'.
2025-02-06T05:24:32.1729801Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.Module3.trx'.
--- Removed for brevity (more INFO lines)
2025-02-06T05:24:32.5264041Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.ModuleX.trx'.
2025-02-06T05:24:32.5265992Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.ModuleY.trx'.
2025-02-06T05:24:32.5556485Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.ModuleZ.trx'.
2025-02-06T05:24:32.5929654Z INFO: Sensor C# Unit Test Results Import [csharp] (done) | time=1344ms
Screenshot of the test list (only the last test project is taken into account)
Thanks for the tip, I got a 10Gb log file but I was able to find many lines like this:
2025-02-12T12:16:39.9053770Z 12:15:51.760 DEBUG: Cannot find the file '/home/runner/work/src/tests/Tests.Project/Services/ExternalCustomers/GeneratorTest.cs'. No test results will be imported. Mapped path is '/home/runner/work/src/tests/Tests.Project/Services/ExternalCustomers/GeneratorTest.cs'.
I think this is happening because we’re excluding the code under /test (and the project that was reporting tests is incorrectly outside that folder).
It seem’s that the tests/ folder is completely ignored for some reason (it was not before the latest updates).
The only tests reported are those from a test project that it’s inside the src/ folder by mistake.
As I said, we use Github Actions with the dotnet runner and in the project settings I left everything in the Analysis Scope settings empty, just in case.
Any clues on how to take tests/ folder into account?
Looking into the logs, I also see messages like this: Skipping issue CS8604, input file not found or excluded: for code located in tests/ folder.
It seems so, the .trx files are picked properly, the test counts, duration, etc is show in the logs like this:
2025-02-25T13:19:40.0363807Z 13:17:11.726 INFO: Parsing the Visual Studio Test Results file '/home/runner/work/repo/src/root/./artifacts/tests/Tests.Project1.trx'.
2025-02-25T13:19:40.0364185Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: 1c5af304-9755-98c1-85cd-293ad585e1b1 outcome: Passed, duration: 2
2025-02-25T13:19:40.0364560Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: c2fe5eaf-eeaf-6640-24f9-58a704ce8217 outcome: Passed, duration: 1
2025-02-25T13:19:40.0364924Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: 0de682a1-4d7b-1583-7888-8467f48d8414 outcome: Passed, duration: 0
2025-02-25T13:19:40.0365352Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: 49214e7e-9d21-ee58-5c7d-73053371f776 outcome: Passed, duration: 1
2025-02-25T13:19:40.0365715Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: 349df24a-c599-36e2-4e11-0a7168a275d7 outcome: Passed, duration: 1
2025-02-25T13:19:40.0366084Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: bb663bcb-d515-112b-6605-3327ef903e96 outcome: Passed, duration: 0
...
2025-02-25T13:19:40.0621979Z 13:17:11.735 DEBUG: Added Test Method: Project1.Tests.Services.Service1Tests.Can_get_averaged_prices_for_full_single_month to File: /home/runner/work/project/src/root/tests/Project1.Tests/Services/When_getting_invoice_study_monthly_weighted_unit_prices.cs
...
There are many more like that and in the end I get the error indicating that the test can’t be mapped:
2025-02-25T13:19:40.5094232Z 13:17:11.836 DEBUG: Cannot find the file '/home/runner/work/project/src/root/tests/Project1.Tests/Services/When_getting_invoice_study_monthly_weighted_unit_prices.cs'. No test results will be imported. Mapped path is '/home/runner/work/project/src/root/tests/Project1.Tests/Services/When_getting_invoice_study_monthly_weighted_unit_prices.cs'.
Searching for the file name I algo saw this message:
2025-02-25T12:07:33.0840528Z 13:07:33.041 WARN: File '/home/runner/work/project/src/root/tests/Project1.Tests/Services/When_getting_invoice_study_monthly_weighted_unit_prices.cs' referenced by the protobuf 'TokenTypeInfo' does not exist in the analysis context
It seems to me that everything in the /tests folder is being ignored but I don’t see how to add it.
Analysis piggybacks on the build to undertsand what is / is not in the project. Is there a build of these test projects that happens during the build phase?
I just ran the whole build this morning and got the results with dotnet build logging enabled.
I see the categorization is mostly correct (one MAIN project is detected as TEST because references Moq by mistake). But the TEST projects are properly detected with this reason
2025-02-27T04:06:44.6158719Z Sonar: (Tests.Project1.csproj) Categorizing project as test or product code...
2025-02-27T04:06:44.6164193Z Sonar: (Tests.Project1.csproj) project has the ProjectCapability 'TestContainer' -> test project.
2025-02-27T04:06:44.6165396Z Sonar: (Tests.Project1.csproj) categorized as TEST project (test code).
Just to be clear, if the ~17k tests are generated tests then they are unfortunately no longer able to be counted. This is due to the change you’ve linked above and the scanner not scanning generated code.
If not, I understand that the logs are 10gb, so probably a little large to share.
Are you able to share how you’re invoking the begin step and dotnet build including the properties you are passing?
e.g.
dotnet sonarscanner begin /d:sonar.cs.vstest.reportsPaths="**/VSTest.trx" /d:sonar.verbose="true" sonar.test.exclusions="**/test"
dotnet build --no-restore -v:d
etc.? With everything except any sensitive information.
Regarding this log:
2025-02-25T12:07:33.0840528Z 13:07:33.041 WARN: File '/home/runner/work/project/src/root/tests/Project1.Tests/Services/When_getting_invoice_study_monthly_weighted_unit_prices.cs' referenced by the protobuf 'TokenTypeInfo' does not exist in the analysis context
Assuming the test mapping is accurate and a file exists at that location, the scanner should detect that these projects are test projects, scan them as test projects and index them allowing the test info to properly display.
These warnings normally show up when exclusions are configured. Are there exclusions set on the server?