Most unit tests not taken into account for dotnet in 2025.1

mcliment · February 6, 2025, 8:35am

Since the update to Community Build v25.1.0.102122 the count of unit tests has dropped signifcantly in our analysis, from 17K to just 399.

As pointed here: New .NET analyzers released this may be something expected, I think the problem is elsewhere.

Version Community Build v25.1.0.102122 MQR Mode
Running using Docker image
Running analysis in using Github Action

Runner logs:

2025-02-06T04:03:02.6969507Z SonarScanner for MSBuild 9.0.2
2025-02-06T04:03:02.6971314Z Using the .NET Core version of the Scanner for MSBuild
2025-02-06T04:03:05.3638209Z Pre-processing started.
2025-02-06T04:03:05.4861239Z Preparing working directories...
2025-02-06T04:03:05.6670793Z 05:03:05.666  Updating build integration targets...
2025-02-06T04:03:09.4521784Z 05:03:09.452  Using SonarQube v25.1.0.102122.
2025-02-06T04:03:10.9021376Z 05:03:10.901  The JRE provisioning is a time consuming operation.
2025-02-06T04:03:10.9022063Z JRE provisioned: OpenJDK17U-jre_x64_linux_hotspot_17.0.13_11.tar.gz.

Logs where the .trx files are processed:

2025-02-06T05:24:31.2521189Z INFO: Sensor C# Unit Test Results Import [csharp]
2025-02-06T05:24:32.1388234Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.Module1.trx'.
2025-02-06T05:24:32.1693375Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.Module2.trx'.
2025-02-06T05:24:32.1729801Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.Module3.trx'.
--- Removed for brevity (more INFO lines)
2025-02-06T05:24:32.5264041Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.ModuleX.trx'.
2025-02-06T05:24:32.5265992Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.ModuleY.trx'.
2025-02-06T05:24:32.5556485Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.ModuleZ.trx'.
2025-02-06T05:24:32.5929654Z INFO: Sensor C# Unit Test Results Import [csharp] (done) | time=1344ms

Screenshot of the test list (only the last test project is taken into account)

Interesting fact is that the project with unit tests accounted is not the last nor the first .trx processed.

Any clues on how to make all tests count again?

ganncamp · February 7, 2025, 2:09pm

Hi,

Could we have a full, /d:"sonar.verbose=true, analysis log, please?

Thx,
Ann

mcliment · February 12, 2025, 2:53pm

Hi @ganncamp,

Thanks for the tip, I got a 10Gb log file but I was able to find many lines like this:

2025-02-12T12:16:39.9053770Z 12:15:51.760 DEBUG: Cannot find the file '/home/runner/work/src/tests/Tests.Project/Services/ExternalCustomers/GeneratorTest.cs'. No test results will be imported. Mapped path is '/home/runner/work/src/tests/Tests.Project/Services/ExternalCustomers/GeneratorTest.cs'.

I think this is happening because we’re excluding the code under /test (and the project that was reporting tests is incorrectly outside that folder).

Thanks for your help!

mcliment · February 26, 2025, 9:08am

Hi again @ganncamp

Although I found out what the problem is, I haven’t been able to figure out how to solve it.

Our solution structure is like this:

Solution.sln
src/
   Project1
   Project2
tests/
   Project1.Tests
   Project2.Tests

It seem’s that the tests/ folder is completely ignored for some reason (it was not before the latest updates).
The only tests reported are those from a test project that it’s inside the src/ folder by mistake.

As I said, we use Github Actions with the dotnet runner and in the project settings I left everything in the Analysis Scope settings empty, just in case.

Any clues on how to take tests/ folder into account?

Looking into the logs, I also see messages like this: Skipping issue CS8604, input file not found or excluded: for code located in tests/ folder.

ganncamp · February 26, 2025, 1:15pm

Hi,

Let’s back up: what’s in your test execution report? Is the report being picked up and read without warnings or errors?

Ann

mcliment · February 26, 2025, 3:38pm

It seems so, the .trx files are picked properly, the test counts, duration, etc is show in the logs like this:

2025-02-25T13:19:40.0363807Z 13:17:11.726 INFO: Parsing the Visual Studio Test Results file '/home/runner/work/repo/src/root/./artifacts/tests/Tests.Project1.trx'.
2025-02-25T13:19:40.0364185Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: 1c5af304-9755-98c1-85cd-293ad585e1b1 outcome: Passed, duration: 2
2025-02-25T13:19:40.0364560Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: c2fe5eaf-eeaf-6640-24f9-58a704ce8217 outcome: Passed, duration: 1
2025-02-25T13:19:40.0364924Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: 0de682a1-4d7b-1583-7888-8467f48d8414 outcome: Passed, duration: 0
2025-02-25T13:19:40.0365352Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: 49214e7e-9d21-ee58-5c7d-73053371f776 outcome: Passed, duration: 1
2025-02-25T13:19:40.0365715Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: 349df24a-c599-36e2-4e11-0a7168a275d7 outcome: Passed, duration: 1
2025-02-25T13:19:40.0366084Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: bb663bcb-d515-112b-6605-3327ef903e96 outcome: Passed, duration: 0
...
2025-02-25T13:19:40.0621979Z 13:17:11.735 DEBUG: Added Test Method: Project1.Tests.Services.Service1Tests.Can_get_averaged_prices_for_full_single_month to File: /home/runner/work/project/src/root/tests/Project1.Tests/Services/When_getting_invoice_study_monthly_weighted_unit_prices.cs
...

There are many more like that and in the end I get the error indicating that the test can’t be mapped:

2025-02-25T13:19:40.5094232Z 13:17:11.836 DEBUG: Cannot find the file '/home/runner/work/project/src/root/tests/Project1.Tests/Services/When_getting_invoice_study_monthly_weighted_unit_prices.cs'. No test results will be imported. Mapped path is '/home/runner/work/project/src/root/tests/Project1.Tests/Services/When_getting_invoice_study_monthly_weighted_unit_prices.cs'.

Searching for the file name I algo saw this message:

2025-02-25T12:07:33.0840528Z 13:07:33.041 WARN: File '/home/runner/work/project/src/root/tests/Project1.Tests/Services/When_getting_invoice_study_monthly_weighted_unit_prices.cs' referenced by the protobuf 'TokenTypeInfo' does not exist in the analysis context

It seems to me that everything in the /tests folder is being ignored but I don’t see how to add it.

I hope this helps.

ganncamp · February 26, 2025, 3:54pm

Hi,

Analysis piggybacks on the build to undertsand what is / is not in the project. Is there a build of these test projects that happens during the build phase?

And if so, take a look at your build logs for the project categorization and tell me if the test projects show up at all.

Thx,
Ann

mcliment · February 27, 2025, 9:40am

Hi,

I just ran the whole build this morning and got the results with dotnet build logging enabled.

I see the categorization is mostly correct (one MAIN project is detected as TEST because references Moq by mistake). But the TEST projects are properly detected with this reason

2025-02-27T04:06:44.6158719Z          Sonar: (Tests.Project1.csproj) Categorizing project as test or product code...
2025-02-27T04:06:44.6164193Z          Sonar: (Tests.Project1.csproj) project has the ProjectCapability 'TestContainer' -> test project.
2025-02-27T04:06:44.6165396Z          Sonar: (Tests.Project1.csproj) categorized as TEST project (test code).

So, it does not seem to be the cause.

ganncamp · February 27, 2025, 12:52pm

Hi,

Thanks for checking. I’m going to flag this for the language experts.

Ann

alexander.meseldzija · March 6, 2025, 10:05am

Hi @mcliment,

Just to be clear, if the ~17k tests are generated tests then they are unfortunately no longer able to be counted. This is due to the change you’ve linked above and the scanner not scanning generated code.

If not, I understand that the logs are 10gb, so probably a little large to share.

Are you able to share how you’re invoking the begin step and dotnet build including the properties you are passing?

e.g.

dotnet sonarscanner begin /d:sonar.cs.vstest.reportsPaths="**/VSTest.trx"   /d:sonar.verbose="true" sonar.test.exclusions="**/test" 


dotnet build --no-restore -v:d

etc.? With everything except any sensitive information.

Regarding this log:

2025-02-25T12:07:33.0840528Z 13:07:33.041 WARN: File '/home/runner/work/project/src/root/tests/Project1.Tests/Services/When_getting_invoice_study_monthly_weighted_unit_prices.cs' referenced by the protobuf 'TokenTypeInfo' does not exist in the analysis context

Assuming the test mapping is accurate and a file exists at that location, the scanner should detect that these projects are test projects, scan them as test projects and index them allowing the test info to properly display.

These warnings normally show up when exclusions are configured. Are there exclusions set on the server?

Thanks!

mcliment · March 6, 2025, 11:30am

Hi @alexander.meseldzija ,

I can share some actions code, nothing private here:

    - name: 📡 Initalize SonarScanner
      shell: bash
      working-directory: ./src/main
      run: ./.sonar/scanner/dotnet-sonarscanner begin /k:"main-core" /d:sonar.token="${{ inputs.sonar-token }}" /d:sonar.host.url="${{ inputs.sonar-host-url }}" /d:sonar.ws.timeout=240 /d:sonar.verbose=${{ inputs.sonar-verbose }}

    - name: 🙏 Build
      shell: bash
      working-directory: ./src/main
      run: dotnet build -c Release -v:n

    - name: 🐛 Test with coverage
      shell: bash
      working-directory: ./src/main
      run: dotnet test -c Release --no-build --filter "TestCategory!=Integration&TestCategory!=Performance" --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=opencover
      continue-on-error: true

    - name: 📡 SonarScanner report
      shell: bash
      working-directory: ./src/main
      run: ./.sonar/scanner/dotnet-sonarscanner end /d:sonar.token="${{ inputs.sonar-token }}"
      continue-on-error: true

I’ve checked the logs and the Sonar Scanner, installed using dotnet tool is version 9.2.1.

Also, in the server I cleared all settings in Analysis Scope so no inclusions/exclusions are defined.

Just to be clear, this worked until we upgraded to v25.1.0.102122, with the previous version (old versioning scheme) it worked properly.