Generic Test: Where can I find message and stacktrace in the SQ UI?

guwirth · July 25, 2022, 3:31pm

Hello there,

I am using the Generic Test Execution Report Format and have created an XML file according to the description: Generic Test Data | SonarQube Docs

There are the XML tags:

message (mandatory): short message describing the cause
stacktrace (optional): long message containing details about failure|error|skipped status

Where can I see the values of the XML report in the SQ UI?

Regards,

ganncamp · July 25, 2022, 4:49pm

Hi,

Ehm… I’m not sure you can anymore. We stripped some of that execution data out several years ago. If it’s still available anywhere, I’d start in the Measures page, with the Cvoerage → Tests → Errors listings.

HTH,
Ann

guwirth · July 26, 2022, 12:15pm

Hi Ann,

I’d start in the Measures page, with the Cvoerage → Tests → Errors listings.

I did a test, the values are not visible in the UI.

Looking into the code you are still reading the value

github.com

SonarSource/sonarqube/blob/dfcd5cf9fe1e261a733ed99a95585b51907ac33c/sonar-scanner-engine/src/main/java/org/sonar/scanner/genericcoverage/GenericTestExecutionReportParser.java#L154


      
            if (child.getNext() != null) {
              String elementName = child.getLocalName();
              if (SKIPPED.equals(elementName)) {
                status = Status.SKIPPED;
              } else if (FAILURE.equals(elementName)) {
                status = Status.FAILURE;
              } else if (ERROR.equals(elementName)) {
                status = Status.ERROR;
              }
              if (Status.OK != status) {
                testCase.setMessage(mandatoryAttribute(child, MESSAGE_ATTR));
              }
            }
            testCase.setStatus(status);
          
          
}
          
          
public int numberOfMatchedFiles() {
            return matchedFileKeys.size();
          }

But seems not to use it?

github.com

SonarSource/sonarqube/blob/6ad373f011432b79d8fff3772146373017b7758b/sonar-scanner-engine/src/main/java/org/sonar/scanner/report/TestExecutionPublisher.java#L66


      
            for (final InputComponent c : componentStore.all()) {
              DefaultInputComponent component = (DefaultInputComponent) c;
              if (component.isFile()) {
                DefaultInputFile file = (DefaultInputFile) component;
                // Recompute test execution measures from MutableTestPlan to take into account the possible merge of several reports
                updateTestExecutionFromTestPlan(file, writer);
              }
            }
          }
          
          
private void updateTestExecutionFromTestPlan(final InputFile inputFile, ScannerReportWriter writer) {
            final DefaultTestPlan testPlan = testPlanBuilder.getTestPlanByFile(inputFile);
            if (testPlan == null || !testPlan.testCases().iterator().hasNext()) {
              return;
            }
            long nonSkippedTests = StreamSupport.stream(testPlan.testCases().spliterator(), false).filter(t -> t.status() != Status.SKIPPED).count();
            appendMeasure(inputFile, writer, new DefaultMeasure<Integer>().forMetric(TESTS).withValue((int) nonSkippedTests));
            long executionTime = StreamSupport.stream(testPlan.testCases().spliterator(), false).map(DefaultTestCase::durationInMs).filter(Objects::nonNull).mapToLong(Long::longValue)
              .sum();
            appendMeasure(inputFile, writer, new DefaultMeasure<Long>().forMetric(TEST_EXECUTION_TIME).withValue(executionTime));
            long errorTests = StreamSupport.stream(testPlan.testCases().spliterator(), false).filter(t -> t.status() == Status.ERROR).count();

The API doing this is marked deprecated:

import org.sonar.scanner.deprecated.test.DefaultTestCase;
import org.sonar.scanner.deprecated.test.DefaultTestCase.Status;
import org.sonar.scanner.deprecated.test.DefaultTestPlan;
import org.sonar.scanner.deprecated.test.TestPlanBuilder;

When do you plan to update the Generic Test sensor and the API?

Regards,

ganncamp · July 26, 2022, 1:24pm

Hi,

I’m not aware of any plans to update the Generic Test format or the API.

Ann

DavidAuger · July 27, 2022, 8:19am

Hello Ann,

Why does this feature has been deprecated ?

My team and I are working with Azure DevOps CICD pipelines for a C++ project and it would have been really helpful to see Unit Tests results in Sonarqube.
We do have errors and success metrics but for more details (which ones, reasons, etc) we are forced to read pipelines logs, as we have a lot of small repositories difficult to have them both in local.

Best regards,

DavidAuger · July 27, 2022, 8:48am

Hello Ann,

We are also wondering why we are not able to see details neither, as it was possible in version 6.7 (at least this one).
We were expecting to find some details like shown on screenshot of documentation Seeing Tests - SonarQube-6.7 (end of page):

But instead, we can see something less detailed like this:

Useful for metrics but less for Unit Test reports

Best regards,

ganncamp · July 27, 2022, 6:11pm

Hi @DavidAuger,

Welcome to the community!

SonarQube 6.7 was a long time ago. A lot has changed since then.

I’ll be honest & say that as an organization we’ve never really believed in those test metrics. Why? Because we feel that if you’ve got failing tests, the pipeline should stop then & there. You shouldn’t just count the failures & move on. That’s why we’ve slowly but surely been moving away from these metrics.

HTH,
Ann

guwirth · July 30, 2022, 5:14pm

Hi Ann,

In case you don’t wanna fix it you should at least update the documentation.

Regards,

ganncamp · August 1, 2022, 11:46am

Hi @guwirth,

What do you think should be fixed / updated in the docs?

Ann

guwirth · August 1, 2022, 12:05pm

Hi Ann,

What do you think should be fixed / updated in the docs?

These two XML tags are no more supported:

message (mandatory): short message describing the cause
stacktrace (optional): long message containing details about failure|error|skipped status

You should give at least a hint that they are not visible in the UI.

Regards,

ganncamp · August 1, 2022, 12:28pm

Hi,

Well, they’re still supported (analysis doesn’t error out), but fair point. I’ll flag this for the docs team.

Thx,
Ann

DavidAuger · August 4, 2022, 7:22am

Hi Ann,

Sorry for the delay (was off some days).

Why? Because we feel that if you’ve got failing tests, the pipeline should stop then & there. You shouldn’t just count the failures & move on. That’s why we’ve slowly but surely been moving away from these metrics.

We could debate about this for a while, but i will try not to.
I just want to say that this statement may not be this straight for everyone (and maybe not us here).
I worked on some project back then with clients asking for features evolution perfectly knowing that it would break algorithm for other cases (letting time to properly fix/change those cases with its users). Keeping those errors help us all to solve this case with debates and design.

And if i may be a bit more incisive following your statement, why keeping errors metrics at all ? If we expect to have only “success” in SQ by stopping pipelines before, there is no need to these metrics
(i mean “numbers” metrics, counting “errors”, “skipped”, percentage of success, etc)

DavidAuger · August 4, 2022, 7:25am

they’re still supported (analysis doesn’t error out), but fair point. I’ll flag this for the docs team.

Yes, please.
Because i understand that there are steal supported, but there is no mention that they are not used or not visible.
And with no indication, we could naturally expect then to show up. Adding that they were before, so that is a level more confusing.

Best regards,
David

ganncamp · August 4, 2022, 11:49am

Hi,

Thanks for sharing this. It’s the first good argument I remember seeing for allowing a build with failed tests to move forward.

Ann

guitcastro · August 29, 2022, 7:35pm

I’ll be honest & say that as an organization we’ve never really believed in those test metrics. Why? Because we feel that if you’ve got failing tests, the pipeline should stop then & there. You shouldn’t just count the failures & move on. That’s why we’ve slowly but surely been moving away from these metrics.

In our case, sometimes the tests only fail in the CI and is very hard to reproduce locally. We want to store the tests result in order to troubleshot them.