Importing Newman test data

SonarQube
, ,
1

SonarQube Enterprise 9.8.0.63668

Trying to import test data output from Postman Lab’s tool: Newman. This is essentially a headless Postman instance that executes all tests in a Postman collection, and generates output in various formats. What is under test is an API either locally or remotely deployed based on the code in the repository. The postman collection itself is a JSON file.

Newman can generate JUnit format output, which I have tried to import using the sonar.junit.reportPaths property. This does not appear to import as I see no count of unit tests on the front page of the project in SonarQube. An example output looks something like this:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite id="0" hostname="XXXXX" package="Status" name="Successful" tests="2" failures="0" errors="0" timestamp="2023-01-28T16:09:01.929" time="0.175">
    <testcase classname="Status / Successful" name="Status - Successful - Response Status Code is 200" time="0.087"/>
    <testcase classname="Status / Successful" name="Status - FAIL - Response Content-Type Header is application/json" time="0.087">
      <failure type="AssertionError" message="expected 'application/json' to include 'application/xml'">
        <![CDATA[AssertionError: expected 'application/json' to include 'application/xml'
   at Object.eval sandbox-script.js:2:1)]]>
      </failure>
    </testcase>
  </testsuite>
...

(ignore the failure, I intentionally modified the test to get a failure to see what that would look like)

When I run SonarScanner with --debug I see no mention of this file at all - so I can only assume it’s not being imported because there is no Java code in the repository. That’s ok I thought, I’ll write an output in Sonar’s generic test data format… Which I did and it gives me this:

<?xml version="1.0" encoding="UTF-8"?>
<testExecutions version="1">
  <file path="tests/XXXXX.postman_collection.json">
    <testCase name="GET XXXXX (Status - Successful - Response Status Code is 200)" duration="0.175"/>
    <testCase name="GET XXXXX (Status - FAIL - Response Content-Type Header is application/json)" duration="0.175">
      <failure message="expected 'application/json' to include 'application/xml'"><![CDATA[AssertionError: expected 'application/json' to include 'application/xml'
   at Object.eval sandbox-script.js:2:1)]]></failure>
    </testCase>
...

So far so good. I then pointed to the output XML file using the sonar.testExecutionReportPaths property. I also have sonar.tests=tests so that the Postman collection JSON is considered a test file. Running SonarScanner with --debug I can see this file referenced now:

17:13:26.513 INFO: Sensor Generic Test Executions Report
17:13:26.514 INFO: Parsing C:\dev\apim-api-XXX\reports\sonarqube\TESTS-XXXXXX.postman_collection.xml
17:13:26.529 INFO: Imported test execution data for 0 files
17:13:26.529 INFO: Test execution data ignored for 1 unknown files, including:
tests/HD_Mobile_API.postman_collection.json
17:13:26.529 INFO: Sensor Generic Test Executions Report (done) | time=16ms

But, the Postman collection in JSON is an ignored file by SonarScanner, so the test results aren’t being tracked.

Is there any way to make this work?

Any and all help appreciated.

2

Ok so I added sonar.test.inclusions=tests/*.json to explicitly include JSON files as tests, and that then indicates that the file is being imported (having fixed durations to be longs not doubles):

09:14:35.046 INFO: Sensor Generic Test Executions Report
09:14:35.047 INFO: Parsing C:\dev\apim-api-XXXXX\reports\sonarqube\TESTS-XXXXX.postman_collection.xml
09:14:35.080 INFO: Imported test execution data for 1 files
09:14:35.080 INFO: Sensor Generic Test Executions Report (done) | time=34ms

But, I still don’t see unit tests on SonarQube project. Realise coverage will be 0% (not started looking into this yet, but coverage of an API is a bit of a different concept to coverage of code) but we’d need the test results to be shown. Any advice?

3

Hi,

You’ve set sonar.tests.includsions, but have you set sonar.tests? A file must be recognized in sonar.tests before it can be included / excluded.

 
Ann

4

Hey, yeah we’ve got sonar.tests=tests

5

Hi,

Could you share your full properties list as well as the entire analysis log?

The analysis / scanner log is what’s output from the analysis command. Hopefully, the log you provide - redacted as necessary - will include that command as well.

This guide will help you find them.

 
Ann

6

Analyser context:

Plugins:
  - (OWN COMPANY PLUGIN - just adds some pages) Plugin for SonarQube 9.x 1.0.0 (XXX)
  - Dependency-Check 3.0.1 (dependencycheck)
  - Azure Active Directory (AAD) Authentication Plug-in for SonarQube 1.3.2 (authaad)
  - Groovy 1.8 (groovy)
Bundled analyzers:
  - IaC Code Quality and Security 1.9.2.2279 (iac)
  - PL/SQL Code Quality and Security 3.8.0.4948 (plsql)
  - Scala Code Quality and Security 1.11.0.3905 (sonarscala)
  - C# Code Quality and Security 8.50.0.58025 (csharp)
  - Vulnerability Analysis 9.8.0.18639 (security)
  - Java Code Quality and Security 7.15.0.30507 (java)
  - HTML Code Quality and Security 3.7.1.3306 (web)
  - Flex Code Quality and Security 2.8.0.3166 (flex)
  - XML Code Quality and Security 2.6.1.3686 (xml)
  - Text file Code Quality and Security 1.2.0.510 (text)
  - VB.NET Code Quality and Security 8.50.0.58025 (vbnet)
  - Swift Code Quality and Security 4.8.0.5759 (swift)
  - CFamily Code Quality and Security 6.40.0.60596 (cpp)
  - Python Code Quality and Security 3.21.0.10628 (python)
  - Dataflow Bug Detection Rules for Python 1.9.0.2901 (dbdpythonfrontend)
  - Dataflow Bug Detection 1.9.0.2901 (dbd)
  - Go Code Quality and Security 1.11.0.3905 (go)
  - JaCoCo 1.1.1.1157 (jacoco)
  - Kotlin Code Quality and Security 2.12.0.1956 (kotlin)
  - RPG Code Quality 3.3.0.3147 (rpg)
  - Dataflow Bug Detection Rules for Java 1.9.0.2901 (dbdjavafrontend)
  - PL/I Code Quality and Security 1.12.0.3443 (pli)
  - T-SQL Code Quality and Security 1.7.0.5449 (tsql)
  - VB6 Code Quality and Security 2.9.0.3341 (vb)
  - Apex Code Quality and Security 1.11.0.3905 (sonarapex)
  - JavaScript/TypeScript/CSS Code Quality and Security 9.12.0.20319 (javascript)
  - Ruby Code Quality and Security 1.11.0.3905 (ruby)
  - Vulnerability Rules for C# 9.8.0.18639 (securitycsharpfrontend)
  - Vulnerability Rules for Java 9.8.0.18639 (securityjavafrontend)
  - Vulnerability Rules for JS 9.8.0.18639 (securityjsfrontend)
  - COBOL Code Quality 5.2.0.5949 (cobol)
  - Vulnerability Rules for Python 9.8.0.18639 (securitypythonfrontend)
  - PHP Code Quality and Security 3.25.0.9077 (php)
  - ABAP Code Quality and Security 3.11.0.4030 (abap)
  - Configuration detection fot Code Quality and Security 1.2.0.267 (config)
  - Vulnerability Rules for PHP 9.8.0.18639 (securityphpfrontend)
Global server settings:
  - email.from=sonarqube@XXXXXXXX
  - sonar.auth.aad.allowUsersToSignUp=true
  - sonar.auth.aad.enableClientCredential=false
  - sonar.auth.aad.enableGroupsSync=true
  - sonar.auth.aad.enabled=true
  - sonar.auth.aad.loginStrategy=Same as Azure AD login
  - sonar.auth.aad.tenantId=XXXXXXXXX
  - sonar.ce.parallelProjectTasks=true
  - sonar.core.id=XXXXXXXX
  - sonar.core.serverBaseURL=XXXXXXXX
  - sonar.core.startTime=2023-01-11T02:52:43+0000
  - sonar.coverage.exclusions=.*.js,rollup.config.js,**/*.stories.*,.storybook/**,**/jest/**,**/*.test.*,**/*.spec.*,**/__test_data__/**,**/__snapshots__/**,**/__mocks__/**
  - sonar.cpd.exclusions=**/model/**,**/__test_data__/**,**/*.stories.*
  - sonar.dbcleaner.branchesToKeepWhenInactive=master,develop,trunk,development,main
  - sonar.dependencyCheck.htmlReportPath=${WORKSPACE}/reports/dependency-check/dependency-check-report.html
  - sonar.dependencyCheck.jsonReportPath=${WORKSPACE}/reports/dependency-check/dependency-check-report.json
  - sonar.dependencyCheck.securityHotspot=true
  - sonar.dependencyCheck.summarize=true
  - sonar.dependencyCheck.xmlReportPath=${WORKSPACE}/reports/dependency-check/dependency-check-report.xml
  - sonar.exclusions=**/node_modules/**,**/vendor/**,**/gw-*/**,build/**,dist/**,docs/**,reports/**,EsLintReports/**,.github/**,azure/**,.vscode/**,**/*License Agreement.html,**/wwwroot/lib/**
  - sonar.forceAuthentication=true
  - sonar.governance.report.view.frequency=Weekly
  - sonar.governance.report.view.recipients=XXXXXXXX
  - sonar.lf.enableGravatar=false
  - sonar.lf.logoUrl=XXXXXXXX
  - sonar.lf.logoWidthPx=188
  - sonar.plsql.file.suffixes=pks,pkb
  - sonar.plugins.risk.consent=ACCEPTED
  - sonar.test.inclusions=**/__tests__/**,**/*.test.*,**/*.spec.*
  - sonar.tsql.file.suffixes=.tsql,.sql
Project server settings:
Project scanner properties:
  - sonar.coverage.exclusions=**/*
  - sonar.exclusions=tests/**/*,newman-reporter-sonarqube/**/*
  - sonar.externalIssuesReportPaths=reports/spectral.json
  - sonar.host.url=XXXXXXXX
  - sonar.login=XXXXXXXX
  - sonar.projectBaseDir=/home/vsts/work/1/s
  - sonar.projectKey=XXXXXXXX
  - sonar.projectName=XXXXXXXX
  - sonar.pullrequest.base=XXXXXXXX
  - sonar.pullrequest.branch=XXXXXXXX
  - sonar.pullrequest.github.repository=XXXXXXXX
  - sonar.pullrequest.key=1
  - sonar.pullrequest.provider=github
  - sonar.python.version=3
  - sonar.scanner.app=ScannerCLI
  - sonar.scanner.appVersion=4.8.0.2856
  - sonar.scanner.metadataFilePath=XXXXXXXX
  - sonar.sourceEncoding=UTF-8
  - sonar.sources=.
  - sonar.test.inclusions=tests/*.json
  - sonar.tests=tests
  - sonar.working.directory=XXXXXXXX/.scannerwork
7

Hi,

Thanks for the context. I’m looking for the command-line output of the analysis command.

 
Ann

8

Hi Ann, I’ve just been trying to get that for you. there’s far too much for me to redact, but the important lines I think are:

09:12:14.016 DEBUG: 'tests\XXXXXX.postman_collection.json' indexed as test with language 'json'
09:12:14.017 DEBUG: File tests/XXXXXX.postman_collection.json excluded for coverage
...
09:12:25.459 DEBUG: 'tests/XXXXXX.postman_collection.json' generated metadata as test  with charset 'windows-1252'
...
09:14:35.046 INFO: Sensor Generic Test Executions Report
09:14:35.047 INFO: Parsing XXXXXX\reports\sonarqube\TESTS-XXXXXX.postman_collection.xml
09:14:35.080 INFO: Imported test execution data for 1 files
09:14:35.080 INFO: Sensor Generic Test Executions Report (done) | time=34ms
9

Hi,

Can we back up to what you’re trying to accomplish? Because from what I can tell, you want to show test execution data (# of tests, # errored / failed / skipped, success %, duration)

Without reporting at all on coverage:

And you want to do it in the context… of a pull request?

I suppose it would work to report test data without reporting coverage, although you should know we’re likely to deprecate test data reporting soon.

But we don’t report test data on PRs. You’ll need to analyze a branch for that.

 
HTH,
Ann