Most unit tests not taken into account for dotnet in 2025.1

SonarQube Server / Community Build
, ,
1

Since the update to Community Build v25.1.0.102122 the count of unit tests has dropped signifcantly in our analysis, from 17K to just 399.

As pointed here: New .NET analyzers released this may be something expected, I think the problem is elsewhere.

  • Version Community Build v25.1.0.102122 MQR Mode
  • Running using Docker image
  • Running analysis in using Github Action

Runner logs:

2025-02-06T04:03:02.6969507Z SonarScanner for MSBuild 9.0.2
2025-02-06T04:03:02.6971314Z Using the .NET Core version of the Scanner for MSBuild
2025-02-06T04:03:05.3638209Z Pre-processing started.
2025-02-06T04:03:05.4861239Z Preparing working directories...
2025-02-06T04:03:05.6670793Z 05:03:05.666  Updating build integration targets...
2025-02-06T04:03:09.4521784Z 05:03:09.452  Using SonarQube v25.1.0.102122.
2025-02-06T04:03:10.9021376Z 05:03:10.901  The JRE provisioning is a time consuming operation.
2025-02-06T04:03:10.9022063Z JRE provisioned: OpenJDK17U-jre_x64_linux_hotspot_17.0.13_11.tar.gz.

Logs where the .trx files are processed:

2025-02-06T05:24:31.2521189Z INFO: Sensor C# Unit Test Results Import [csharp]
2025-02-06T05:24:32.1388234Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.Module1.trx'.
2025-02-06T05:24:32.1693375Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.Module2.trx'.
2025-02-06T05:24:32.1729801Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.Module3.trx'.
--- Removed for brevity (more INFO lines)
2025-02-06T05:24:32.5264041Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.ModuleX.trx'.
2025-02-06T05:24:32.5265992Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.ModuleY.trx'.
2025-02-06T05:24:32.5556485Z INFO: Parsing the Visual Studio Test Results file '/home/runner/work/artifacts/tests/Tests.ModuleZ.trx'.
2025-02-06T05:24:32.5929654Z INFO: Sensor C# Unit Test Results Import [csharp] (done) | time=1344ms

Screenshot of the test list (only the last test project is taken into account)

image
image1165×412 12.4 KB

Interesting fact is that the project with unit tests accounted is not the last nor the first .trx processed.

Any clues on how to make all tests count again?

2

Hi,

Could we have a full, /d:"sonar.verbose=true, analysis log, please?

 
Thx,
Ann

3

Hi @ganncamp,

Thanks for the tip, I got a 10Gb log file but I was able to find many lines like this:

2025-02-12T12:16:39.9053770Z 12:15:51.760 DEBUG: Cannot find the file '/home/runner/work/src/tests/Tests.Project/Services/ExternalCustomers/GeneratorTest.cs'. No test results will be imported. Mapped path is '/home/runner/work/src/tests/Tests.Project/Services/ExternalCustomers/GeneratorTest.cs'.

I think this is happening because we’re excluding the code under /test (and the project that was reporting tests is incorrectly outside that folder).

Thanks for your help!

1 Like
4

Hi again @ganncamp

Although I found out what the problem is, I haven’t been able to figure out how to solve it.

Our solution structure is like this:

Solution.sln
src/
   Project1
   Project2
tests/
   Project1.Tests
   Project2.Tests

It seem’s that the tests/ folder is completely ignored for some reason (it was not before the latest updates).
The only tests reported are those from a test project that it’s inside the src/ folder by mistake.

As I said, we use Github Actions with the dotnet runner and in the project settings I left everything in the Analysis Scope settings empty, just in case.

Any clues on how to take tests/ folder into account?

Looking into the logs, I also see messages like this: Skipping issue CS8604, input file not found or excluded: for code located in tests/ folder.

5

Hi,

Let’s back up: what’s in your test execution report? Is the report being picked up and read without warnings or errors?

 
Ann

6

It seems so, the .trx files are picked properly, the test counts, duration, etc is show in the logs like this:

2025-02-25T13:19:40.0363807Z 13:17:11.726 INFO: Parsing the Visual Studio Test Results file '/home/runner/work/repo/src/root/./artifacts/tests/Tests.Project1.trx'.
2025-02-25T13:19:40.0364185Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: 1c5af304-9755-98c1-85cd-293ad585e1b1 outcome: Passed, duration: 2
2025-02-25T13:19:40.0364560Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: c2fe5eaf-eeaf-6640-24f9-58a704ce8217 outcome: Passed, duration: 1
2025-02-25T13:19:40.0364924Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: 0de682a1-4d7b-1583-7888-8467f48d8414 outcome: Passed, duration: 0
2025-02-25T13:19:40.0365352Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: 49214e7e-9d21-ee58-5c7d-73053371f776 outcome: Passed, duration: 1
2025-02-25T13:19:40.0365715Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: 349df24a-c599-36e2-4e11-0a7168a275d7 outcome: Passed, duration: 1
2025-02-25T13:19:40.0366084Z 13:17:11.726 DEBUG: Parsed Visual Studio Unit Test - testId: bb663bcb-d515-112b-6605-3327ef903e96 outcome: Passed, duration: 0
...
2025-02-25T13:19:40.0621979Z 13:17:11.735 DEBUG: Added Test Method: Project1.Tests.Services.Service1Tests.Can_get_averaged_prices_for_full_single_month to File: /home/runner/work/project/src/root/tests/Project1.Tests/Services/When_getting_invoice_study_monthly_weighted_unit_prices.cs
...

There are many more like that and in the end I get the error indicating that the test can’t be mapped:

2025-02-25T13:19:40.5094232Z 13:17:11.836 DEBUG: Cannot find the file '/home/runner/work/project/src/root/tests/Project1.Tests/Services/When_getting_invoice_study_monthly_weighted_unit_prices.cs'. No test results will be imported. Mapped path is '/home/runner/work/project/src/root/tests/Project1.Tests/Services/When_getting_invoice_study_monthly_weighted_unit_prices.cs'.

Searching for the file name I algo saw this message:

2025-02-25T12:07:33.0840528Z 13:07:33.041 WARN: File '/home/runner/work/project/src/root/tests/Project1.Tests/Services/When_getting_invoice_study_monthly_weighted_unit_prices.cs' referenced by the protobuf 'TokenTypeInfo' does not exist in the analysis context

It seems to me that everything in the /tests folder is being ignored but I don’t see how to add it.

I hope this helps.

7

Hi,

Analysis piggybacks on the build to undertsand what is / is not in the project. Is there a build of these test projects that happens during the build phase?

And if so, take a look at your build logs for the project categorization and tell me if the test projects show up at all.

 
Thx,
Ann

1 Like
8

Hi,

I just ran the whole build this morning and got the results with dotnet build logging enabled.

I see the categorization is mostly correct (one MAIN project is detected as TEST because references Moq by mistake). But the TEST projects are properly detected with this reason

2025-02-27T04:06:44.6158719Z          Sonar: (Tests.Project1.csproj) Categorizing project as test or product code...
2025-02-27T04:06:44.6164193Z          Sonar: (Tests.Project1.csproj) project has the ProjectCapability 'TestContainer' -> test project.
2025-02-27T04:06:44.6165396Z          Sonar: (Tests.Project1.csproj) categorized as TEST project (test code).

So, it does not seem to be the cause.

9

Hi,

Thanks for checking. I’m going to flag this for the language experts.

 
Ann

1 Like
12

Hi @mcliment,

Just to be clear, if the ~17k tests are generated tests then they are unfortunately no longer able to be counted. This is due to the change you’ve linked above and the scanner not scanning generated code.

If not, I understand that the logs are 10gb, so probably a little large to share.

Are you able to share how you’re invoking the begin step and dotnet build including the properties you are passing?

e.g.

dotnet sonarscanner begin /d:sonar.cs.vstest.reportsPaths="**/VSTest.trx"   /d:sonar.verbose="true" sonar.test.exclusions="**/test" 


dotnet build --no-restore -v:d

etc.? With everything except any sensitive information.

Regarding this log:

2025-02-25T12:07:33.0840528Z 13:07:33.041 WARN: File '/home/runner/work/project/src/root/tests/Project1.Tests/Services/When_getting_invoice_study_monthly_weighted_unit_prices.cs' referenced by the protobuf 'TokenTypeInfo' does not exist in the analysis context

Assuming the test mapping is accurate and a file exists at that location, the scanner should detect that these projects are test projects, scan them as test projects and index them allowing the test info to properly display.

These warnings normally show up when exclusions are configured. Are there exclusions set on the server?

Thanks!

13

Hi @alexander.meseldzija ,

I can share some actions code, nothing private here:

    - name: 📡 Initalize SonarScanner
      shell: bash
      working-directory: ./src/main
      run: ./.sonar/scanner/dotnet-sonarscanner begin /k:"main-core" /d:sonar.token="${{ inputs.sonar-token }}" /d:sonar.host.url="${{ inputs.sonar-host-url }}" /d:sonar.ws.timeout=240 /d:sonar.verbose=${{ inputs.sonar-verbose }}

    - name: 🙏 Build
      shell: bash
      working-directory: ./src/main
      run: dotnet build -c Release -v:n

    - name: 🐛 Test with coverage
      shell: bash
      working-directory: ./src/main
      run: dotnet test -c Release --no-build --filter "TestCategory!=Integration&TestCategory!=Performance" --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=opencover
      continue-on-error: true

    - name: 📡 SonarScanner report
      shell: bash
      working-directory: ./src/main
      run: ./.sonar/scanner/dotnet-sonarscanner end /d:sonar.token="${{ inputs.sonar-token }}"
      continue-on-error: true

I’ve checked the logs and the Sonar Scanner, installed using dotnet tool is version 9.2.1.

Also, in the server I cleared all settings in Analysis Scope so no inclusions/exclusions are defined.

Just to be clear, this worked until we upgraded to v25.1.0.102122, with the previous version (old versioning scheme) it worked properly.

14

Hi @alexander.meseldzija @ganncamp , any updates or workarounds on this issue? Do you need additional information or logs?

15

Hi @mcliment,
Sorry I let this slip under my radar.

I have tried to recreate in a couple different ways following the project structure above, but have been unable too, except when using generated tests :frowning:

If these tests are generated code, you may be able to activate generated code analysis to load the generated files correctly into the scanner context and allow the tests to be mapped correctly.

It will adversely affect build times, issues will likely be inaccurate and useless for the generated code, but, in theory, this should load the files in the analysis context.

Otherwise, to investigate further, I will need one of (in ideal order):

  • a small reproducer project
  • the begin step verbose logs & the end step verbose logs.
  • the contents of the .sonarqube folder after the begin step, and after the build step
  • the binlog from the build.

Thanks!

16

Hi @alexander.meseldzija

Finally I found some time to reproduce the issue.

I made this repo: GitHub - mcliment/sonar-sample: Sample .NET project to reproduce Sonar issue

The fun part. The original folder for tests is named tests as you can see in the main branch. There’s a PR where the folder is renamed to test. In that case, the tests are reported properly.

Tell me if you need additional info.

2 Likes
20

Hello @mcliment and thanks a lot for the reproducer.

Unfortunately I was not able to reproduce.
I see 4 unit tests on SQ UI (I tested with both “tests” and “test”, no diff).

Could you please send me the verbose logs for this project so I can compare with mine?

Also one question:
You mention in your repo

  • In Project Settings → Languages → C#
  • sonar.cs.opencover.reportsPaths = **/coverage.opencover.xml
  • sonar.cs.vstest.reportsPaths = **/*.trx

In which IDE? Why not set these through the begin step or sq config?
thanks!

21

Hi @Mary_Georgiou , the settings were set in the Sonar UI

image
image787×128 12 KB

image
image770×104 8.24 KB

I hope this helps you reproduce the isses.

Thanks.

1 Like
22

Hello @mcliment!

I still get the coverage and expected amount of tests.

Could you please share the verbose logs on your side from the repro?

image
image419×754 20.3 KB

23

Sure @Mary_Georgiou , I send you a log (user and server redacted)

sonar.log (66.8 KB)

And a log with the tests in /test to see the differences

sonar-2.log (72.9 KB)

I see some differences. In the first report, the tests are not found:

16:13:23.531 INFO: Importing results from 8 proto files in '/Users/user1/Projects/sonar-sample/.sonarqube/out/1/output-cs'
16:13:23.534 WARN: File '/Users/user1/Projects/sonar-sample/tests/Tests.SonarSample.Library/GenericOperatorSubtractionTests.cs' referenced by the protobuf 'TokenTypeInfo' does not exist in the analysis context
16:13:23.536 WARN: File '/Users/user1/Projects/sonar-sample/tests/Tests.SonarSample.Library/GenericOperatorSubtractionTests.cs' referenced by the protobuf 'SymbolReferenceInfo' does not exist in the analysis context
16:13:23.537 INFO: Importing results from 8 proto files in '/Users/user1/Projects/sonar-sample/.sonarqube/out/0/output-cs'

And in the second, it seems to work:

16:29:56.429 INFO: Importing results from 8 proto files in '/Users/user1/Projects/sonar-sample/.sonarqube/out/1/output-cs'
16:29:56.467 INFO: Importing results from 8 proto files in '/Users/user1/Projects/sonar-sample/.sonarqube/out/0/output-cs'

I don’t really understand the reason of this difference…

Tell me if you need something else.

24

Hello again @mcliment and thanks a lot for taking the time to send me the logs!

One very peculiar thing I found is that the trx file is not excluded automatically in your logs.
In mine it’s by default: sonar.exclusions=**/*.trx, **/coverage.opencover.xml
I also see that the auto-discovered test files are not the same at all.

Additionally from your first report:
16:13:22.853 INFO: 9 files ignored because of inclusion/exclusion patterns
In my report there are no ignored files:
14:51:33.602 INFO: 0 files ignored because of inclusion/exclusion patterns

These findings, in combination with the behavior change when renaming the directory from test to tests, makes me fairly positive that there are more analysis properties set somewhere that are automatically included in your analysis and are related to file exclusions.

Please take a look at the documentation and check for possible analysis parameters that add exclusions or are setting the base directory for the analysis.
You should check:

  • SQs global properties (set via the server UI)
  • SQs project properties (I doubt it’s here - as you already analyzed the sample project without setting something)
  • The SonarQube.Analysis.xml that is placed in the install directory of the Scanner for .NET. (I doubt this is the root either).
  • Maybe there’s some env variable?

I’m attaching also the end step on my side in case you wish to take a look.
end-902.txt (108.3 KB)

You can also test, for an extra sanity check, by doing the analysis in a local server on your machine.
Just download the SonarQube server community build, unzip and from the unzipped directory run .\bin\windows-x86-64\StartSonar.bat or .\bin\linux-x86-64\sonar.sh and you’ll have a brand new server to test locally.

Let me know for any question/remark/update.

Thanks again!

1 Like
25

OMG, I forgot that global properties existed… :confounded_face:

image
image762×196 15 KB

This may answer all the questions.

Sorry @Mary_Georgiou for making you spend time with this, but thanks for pointing out the solution.

I did not check that because it all started when we updated the Sonar version, so that’s why I was not fully aware of that. Al least I learned a couple of things about Sonar.

Thanks!

2 Likes