Configure sonarqube in multi maven project running from jenkins

tom_vin · October 9, 2018, 5:49am

assuming you have a maven module with the following structure:

root module
- integration test module
- sources module A
- sources module B
- additional modules…

and you have a pipeline in jenkins which use multiple jobs as part of the build process: preliminary job that compile all modules, and then, additional job that runs the integrations test by executing a maven command on the integration tests module.

now, in sonarqube server, you would like to see all of the sources from all modules, and for each of them, the coverage data.

from what I understand, there are two ways to accomplish this, and the first way is preferable -

Using sonar’s “BaseProjectDir” property.https://docs.sonarqube.org/display/SONAR/Analysis+Parameters#AnalysisParameters-projectBaseDir
run an aggregation job to collect the data after integration test over.

we tried to configure the base directory property ( sonar.projectBaseDir ) as a maven parameter, pom property and in sonar-project.properties file. (also tried an absolute and relative paths)

no matter what we tried, it seems that the base folder is not affecting sonar. as you can see in the picture below (base dir), and by reviewing results in sonar server, the only files that we get are files that located inside integration-tests module (inner module). while we would like that the analysis will be done regarding all the files inside the parent module.

Additional data:

list of sonar parameters (passed by maven)-

sonar.host.url=$SONAR_HOST_URL
sonar.login=$SONAR_AUTH_TOKEN
sonar.sources=src/main
sonar.projectBaseDir=../
sonar.jacoco.reportPaths=**/target/jacoco-it.exec

I am using the following versions:

Sonarqube server: 7.3
sonar-maven-plugin: 3.5.0.1254.

Am I missing something? does sonar support this kind of configuration? I would appreciate any help…

ganncamp · October 9, 2018, 6:32am

Hi,

I don’t understand why you’re trying to override the project base directory. The SonaQube Scanner for Maven is set up to just work - one module or many. Have you tried simply cd-ing into your project root and running mvn sonar:sonar? Assuming your test executions put the reports in the normal places, they should be picked up.

Also, I’m curious why your exclusions include

**/src/test/**/*.java
**/src/it/**/*.java
**/src/it/*.java

Ann

tom_vin · October 9, 2018, 2:04pm

Hey,
I am trying to override the base directory because the execution starts from an inner module (which run’s the tests for all outside modules).
I figured that sonar:sonar goal attached to one of the latest maven phases and therefore it will in turn require execution of some of the maven flow again. (as mentioned, before the sonar goal being executed, mvn clean install command already ran on the root module, and mvn integration-test phase already ran for the inner module).

but I now see, also in this post (https://stackoverflow.com/questions/21013399/what-does-mvn-sonarsonar-do)
that the sonar goal does not require running the preliminary phases again.

are you suggesting that the right way collecting the coverage results and getting source files for all modules is running “mvn sonar:sonar” on the root after the integration tests are done?
(just want to emphasis here that the jacoco ‘exec’ files are only inside the integration test module (where all the tests sits).

if so, what are the “base directory” used for ?

regarding the exclusions, it’s a mistake.

Thanks for the comment.

ganncamp · October 9, 2018, 2:14pm

Hi,

I’m really not sure about this part:

I have the sense that these should be tucked into the target directories of the relevant code instead (altho I’m not sure about this), but in general, yes. The right way to run analysis is to execute mvn sonar:sonar from project root after your test reports have been generated.

The base directory configuration is only for extraordinary circumstances. I’ve had to use it before in analyzing COBOL code when the nightly FTP dump didn’t put it (for obvious reasons) into a Jenkins workspace and I didn’t want to bloat the filesystem by cp-ing it around.

Ann

tom_vin · October 15, 2018, 5:27am

tried to run the analysis from the root directory.
still having unexplained issues…if it’s better to open another thread for that, let me know.

i am getting “file can’t be index twice error”

[ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.5.0.1254:sonar (default-cli) on project yyy-server-root: File Server/query-translation/api/src/main/java/com/…/querytranslation/QueryToDqlCompilerOptions.java can’t be indexed twice. Please check that inclusion/exclusion patterns produce disjoint sets for main and test files -> [Help 1]

now, I have checked my configuration over and over and I don’t see overlapping between test and main files…
here are my sonar properties:

$SONAR_MAVEN_GOAL
–debug
-Dsonar.host.url=$SONAR_HOST_URL
-Dsonar.login=$SONAR_AUTH_TOKEN
-Dsonar.analysis.buildNumber=${BUILD_NUMBER}
-Dsonar.analysis.jobName=${JOB_NAME}
-Dsonar.sources=.
-Dsonar.tests=.
-Dsonar.test.inclusions=**/*Test*/**,**/it/**.java,
-Dsonar.jacoco.reportPaths=**/target/**/jacoco.exec,**/target/**/jacoco-it.exec
-Dsonar.exclusions=**/it/**.java,**/*JSON.java,**/json/**,**/Json.java,**/opb-tests/**,**/NotificationEmail.java,**/*Test*/**,**/*.xml

do you see any issue ?
is it possible to get this kind of error for additional issues ?

thanks,
Tom

ganncamp · October 15, 2018, 11:45am

Hi Tom,

In fact, you do have an overlap between test and main files:

-Dsonar.sources=.
-Dsonar.tests=.

These configs say “everything from here down is sourcetest”. Seen that way, perhaps it’s obvious why analysis is confused.

And once you’ve said “everything is a test”, there’s no need to add test inclusions on top of that

-Dsonar.test.inclusions= `**/*Test*/**,**/it/**.java,`

If I were you, I’d start from the minimal config, and add from there to correct any problems, so:

 $SONAR_MAVEN_GOAL
–debug
-Dsonar.host.url=$SONAR_HOST_URL
-Dsonar.login=$SONAR_AUTH_TOKEN

Ann

tom_vin · October 16, 2018, 10:04am

took the example from stackoverflow.

but anyway,
I tried few variations of inclusion / exclusion patterns, and they all failed.
f or example, as you can see in the following:

Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.5.0.1254:sonar (default-cli) on project yyy-root: File Server/yyy-dal-db-entity-platform/impl/src/main/java/com/…/dal/dbentityplatform/DalServiceBase.java can’t be indexed twice. Please check that inclusion/exclusion patterns produce disjoint sets for main and test files.

this is the excluded & included paths from the logs:

correct me if I am wrong, but the DalServiceBase.java file answer to the definition **/src/main/**/* which is included by sources (you can see in the picture above), and excluded by tests.
so it can’t be the case where this file is being indexed twice by test and main definitions, isn’t it ?

thanks

ganncamp · October 16, 2018, 11:48am

Hi,

Again, I’d start with a minimal configuration: no inclusions, no exclusions. See if you can get that working:

$SONAR_MAVEN_GOAL
–debug
-Dsonar.host.url=$SONAR_HOST_URL
-Dsonar.login=$SONAR_AUTH_TOKEN

Ann