Code coverage on branches does not break quality gate

LucaIpp · April 23, 2019, 8:55am

Hello there

I am currently evaluating SonarQube’s Developer Edition. We are thinking about switching to this license, because we want get analysis data of our branches, wich is not possible with the Community Edition.

I am currently working with a test instance and an evaluation license. I am facing the following problem:
Somehow, code coverage does not break our quality gate on branches.
Our quality gate is configured like this:

Condition Coverage on New Code is less than 80.0%
Coverage on New Code is less than 70.0%
Maintainability Raiting on New Code is worse than B
Reliability Raiting on New Code is worse than B
Security Rating on New Code is worse than A

I’ve added some code to a project that messes up the code coverage and built the project locally

mvn clean install sonar:sonar -Dsonar.branch.name=testBranch -Dsonar.branch.target=master

This resulted in a branch showing up on my SonarQube server for the project. As you can see in the picture below, the quality gate passed. But the coverage metric seems to be broken! Instead of a numeric value I am getting a “-”. But, and this confuses me a little bit, SonarQube is able to calculate an estimated code coverage after the merge of the branch. So coverage data seems to be there, but is somehow not used when checking the quality gate for branches.

To verify that my code actually breaks the quality gate, I built the project with this command

mvn clean install sonar:sonar

which makes sonar assume that the analysis is executed for the master branch / the branch was merged to master. This leads to a broken quality gate on SonarQube’s master branch panel of the project.

Has anyone faced the same issue?
I don’t exactly know where to start with this, does the plugin for code analysis (jacoco) not work properly / is it configured wrong, am I executing the maven build with wrong parameters, is there a configuration on SonarQube that I need to set?

By the way, the quality gate of the branch brakes if Reliability is worse than A, so it seems like the problem is only the code coverage.

Tanks in advance!

Additional information:

SonarQube Version: 7.7.0.23042
Code that gets analyzed: Java / Spring-Boot 2
Plugin for analysis: jacoco

ganncamp · April 24, 2019, 3:18pm

Hi,

If you go to the Code tab of this branch, how many “Lines” do you see listed?

Selection_430

Ann

12588bc5 · April 24, 2019, 3:51pm

I have the same issue, but related to python. My “lines” just appear blank. I believe this was working on previous versions (7.3 was the last version I used before 7.7).

ganncamp · April 24, 2019, 4:07pm

Hi @12588bc5,

Thanks for agreeing to drop your independent thread and join us here.

How many “lines” do you expect to see listed as changed in your SLB? To be clear, deleted lines won’t show up here. So how many lines (approx) did you add or change?

Thx,
Ann

ganncamp · April 24, 2019, 4:35pm

Hi again,

Also, is SCM data available for this SLB? And/or did you commit your changes? (SONAR-11996)

Ann

LucaIpp · April 25, 2019, 6:50am

Hi @ganncamp
Thanks for joining the discussion. I can confirm what @12588bc5 is saying about the line count that is not present

About your other questions:

I added a method with a few hundred lines of code with multiple code bits like if (x<0) then log some text. This method was added than to an existing method where i guaranteed with another if that the method won’t be called. The idea behind this was to create a method with lots of lines that won’t be covered by the unit tests messing up the total coverage as well as the coverage on new code.

My contact @Oguzhan_Ozdemir at SonarSource also mentioned SCM data, but I do not exactly know what this is / how I can check if the SCM data is available. Currently, the branch was only a local branch on my machine, but the changes were commited to the branch. I built the branch locally like this: mvn clean install sonar:sonar -Dsonar.branch.name=testBranch -Dsonar.branch.target=master
Here you can see the output of the sonar-maven-plugin while compiling, there are actually a few lines about “SCM data” in this output:

[INFO] --- sonar-maven-plugin:3.6.0.1398:sonar (default-cli) @ voice-resources ---
[INFO] User cache: /home/taaiplu1/.sonar/cache
[INFO] SonarQube version: 7.7.0
[INFO] Default locale: "en_US", source code encoding: "UTF-8"
[INFO] Load global settings
[INFO] Load global settings (done) | time=181ms
[INFO] Server id: 730A21F2-AWA1TjPjyxejS6Hpl1AD
[INFO] User cache: /home/taaiplu1/.sonar/cache
[INFO] Load/download plugins
[INFO] Load plugins index
[INFO] Load plugins index (done) | time=137ms
[INFO] Load/download plugins (done) | time=155ms
[INFO] Loaded core extensions: developer-scanner
[INFO] Process project properties
[INFO] Execute project builders
[INFO] Execute project builders (done) | time=2ms
[INFO] Project key: com.xxx:voice-resources
[INFO] Base dir: /home/taaiplu1/bootstrap/git-projects/domains/voice/voice-resources
[INFO] Working dir: /home/taaiplu1/bootstrap/git-projects/domains/voice/voice-resources/target/sonar
[INFO] Load project settings for component key: 'com.xxx:voice-resources'
[INFO] Load project settings for component key: 'com.xxx:voice-resources' (done) | time=29ms
[INFO] Load project branches
[INFO] Load project branches (done) | time=33ms
[INFO] Load project pull requests
[INFO] Load project pull requests (done) | time=19ms
[INFO] Load branch configuration
[INFO] Load branch configuration (done) | time=7ms
[INFO] Load project repositories
[INFO] Load project repositories (done) | time=101ms
[INFO] Load quality profiles
[INFO] Load quality profiles (done) | time=51ms
[INFO] Load active rules
[INFO] Load active rules (done) | time=560ms
[INFO] Branch name: testBranch, type: short living
[INFO] SCM collecting changed files in the branch
[INFO] SCM collecting changed files in the branch (done) | time=157ms
[INFO] Indexing files...
[INFO] Project configuration:
[INFO]   Excluded sources for coverage: *Config.java
[INFO] 125 files indexed
[INFO] Quality profile for java: Sonar way
[INFO] Quality profile for xml: Sonar way
[INFO] ------------- Run sensors on module voice-resources
[INFO] Load metrics repository
[INFO] Load metrics repository (done) | time=26ms
[INFO] Sensor JavaSquidSensor [java]
[INFO] Configured Java source version (sonar.java.source): 8
[INFO] JavaClasspath initialization
[INFO] JavaClasspath initialization (done) | time=15ms
[INFO] JavaTestClasspath initialization
[INFO] JavaTestClasspath initialization (done) | time=4ms
[INFO] Java Main Files AST scan
[INFO] 86 source files to be analyzed
[WARNING] Classes not found during the analysis : [org.openid4java.consumer.ConsumerException, org.openid4java.consumer.ConsumerManager, org.springframework.security.openid.AxFetchListFactory, org.springframework.security.openid.OpenIDAuthenticationFilter, org.springframework.security.openid.OpenIDAuthenticationToken, org.springframework.security.openid.OpenIDConsumer]
[INFO] 86/86 source files have been analyzed
[INFO] Java Main Files AST scan (done) | time=4958ms
[INFO] Java Test Files AST scan
[INFO] 38 source files to be analyzed
[INFO] 38/38 source files have been analyzed
[INFO] Java Test Files AST scan (done) | time=2041ms
[INFO] Sensor JavaSquidSensor [java] (done) | time=7504ms
[INFO] Sensor SurefireSensor [java]
[INFO] parsing [/home/taaiplu1/xxx/voice-resources/target/surefire-reports]
[INFO] Sensor SurefireSensor [java] (done) | time=112ms
[INFO] Sensor JaCoCoSensor [java]
[INFO] JaCoCo report not found: 'target/jacoco-it.exec'
[INFO] Analysing /home/taaiplu1/xxx/voice-resources/target/jacoco.exec
[WARNING] The following class(es) did not match with execution data:
[WARNING] > 'com/xxx/util/URIFactory'
[WARNING] In order to have accurate coverage measures, the same class files must be used as at runtime for report generation.
[INFO] No information about coverage per test.
[INFO] Sensor JaCoCoSensor [java] (done) | time=192ms
[INFO] Sensor SonarJavaXmlFileSensor [java]
[INFO] 1 source files to be analyzed
[INFO] Sensor SonarJavaXmlFileSensor [java] (done) | time=433ms
[INFO] 1/1 source files have been analyzed
[INFO] Sensor XML Sensor [xml]
[WARNING] Metric 'comment_lines_data' is deprecated. Provided value is ignored.
[INFO] Sensor XML Sensor [xml] (done) | time=138ms
[INFO] ------------- Run sensors on project
[INFO] Sensor Zero Coverage Sensor
[INFO] Sensor Zero Coverage Sensor (done) | time=6ms
[INFO] Sensor Java CPD Block Indexer
[INFO] Sensor Java CPD Block Indexer (done) | time=158ms
[INFO] SCM provider for this project is: git
[INFO] 1 files to be analyzed
[INFO] 1/1 files analyzed
[INFO] 40 files had no CPD blocks
[INFO] Calculating CPD for 46 files
[INFO] CPD calculation finished
[INFO] SCM writing changed lines
[INFO] SCM writing changed lines (done) | time=1ms
[INFO] Analysis report generated in 100ms, dir size=168 KB
[INFO] Analysis report compressed in 87ms, zip size=105 KB
[INFO] Analysis report uploaded in 65ms
[INFO] ANALYSIS SUCCESSFUL, you can browse https://powerpuff-sonarqube.dos.corproot.net/dashboard?id=com.voice-resources&branch=testBranch&resolved=false
[INFO] Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
[INFO] More about the report processing at https://powerpuff-sonarqube.dos.corproot.net/api/ce/task?id=AWpTPGt3X07XoykTyyyM
[INFO] Analysis total time: 11.603 s

LucaIpp · April 30, 2019, 6:36am

Hi @ganncamp, any updates on how i should proceed?

ganncamp · April 30, 2019, 11:46am

Hi,

Sorry for the delay. I’m out of my depth. I’ve summoned help, but community support is a sideline for all of us. Please continue to be patient.

Ann

LucaIpp · May 2, 2019, 12:31pm

Hi @ganncamp

Thanks for your response. Since I’m currently evaluating the Developers Edition of SonarQube, I am kinda blocked by this. The evaluation license is valid until Friday next week.
Just for my clear understanding, is this a bug that is in your backlog for getting fixed (https://jira.sonarsource.com/browse/SONAR-11996) or does it seem to you like a problem on my configurations?

The quality gate for branches breaks when infringing against rules regarding reliability, security or maintainability, I know that we are facing this issue only in context with code coverage. Since checking code coverage on branches is our most requested features, it would be great to know how this issue is handled by you exactly.

Are there alternatives like using a slightly older version of SonarQube where this issue was not present?

Thanks in advance and best regards
Luca

ganncamp · May 2, 2019, 12:52pm

Hi Luca,

I really can’t answer. But I’m sure there won’t be a problem getting you an extension on the evaluation.

Ann

Christophe · May 3, 2019, 7:54am

Hi,

I think we have the same issue (No coverage on short-lived branches), I will drop my thread and paste its content here :

I am using Teamcity to build and analyse a C# project on Sonarcloud. As our project needs multiple repositories to build, we have disable SCM to avoid the Not inside a Git work tree error related to Teamcity root workspace. The leak period is set on a fix date (beginning of the quarter).

Everything works fine on the master branch but for an unknown reason I have empty values in short-lived branches. It seems that the new code is not considered as new : when looking in a class, the code is not on yellow background as it is on the main branch.

Here are some screenshots of a projet created to demonstrate the issue:

Overview :

Code view :

Class detail (the ReturnNull method is newly added code):

dmeneses · May 6, 2019, 2:42pm

Hi there,
As replied in the topic that you created, https://jira.sonarsource.com/browse/SONAR-11996 is most likely the cause of these problems.
We will work on a fix soon. For the time being, you need to use git (or other supported SCM) to work with short lived branches and P/Rs.

Christophe · May 7, 2019, 8:26am

Hi,

Thanks for your answer, unfortunatly SCM does not work in case of multiple repositories used along with Jenkins or Teamcity as the root workspace does not include a .git directory (those are located in sub directories created for each repository). Do you know if there is a way to specify the directory where to look for SCM information ?

janos · May 8, 2019, 10:06am

No, I’m afraid this is currently not possible, and not even planned. I’ve seen other users copy the .git directory to work around this issue.

Falak_Shah · May 10, 2019, 11:17am

I have same problem. When I execute sonar-sccaner from my local system it works well and I can see coverage report on UI
But when same is executed form Jenkins it does not show coverage report:

Following is log line from local

16:08:14.019 DEBUG: Using pattern ‘build_results/coverage.xml’ to find reports
16:08:15.314 INFO: Python test coverage
16:08:15.316 INFO: Parsing report ‘/Users/falak.shah/xxxx/xxxx/xxxxx/build_results/coverage.xml’

Following is log line from Jenkins server

16:32:37.850 INFO: Sensor Python Squid Sensor [python] (done) | time=10477ms
16:32:37.851 INFO: Sensor Cobertura Sensor for Python coverage [python]
16:32:37.852 DEBUG: Using pattern ‘build_results/coverage.xml’ to find reports
16:32:38.234 WARN: No report was found for sonar.python.coverage.reportPaths using pattern build_results/coverage.xml
16:32:38.234 INFO: Sensor Cobertura Sensor for Python coverage [python] (done) | time=383ms
16:32:38.234 INFO: Sensor PythonXUnitSensor [python]
16:32:38.235 DEBUG: Using pattern ‘build_results/testresults.xml’ to find reports
16:32:38.455 INFO: Processing report ‘/Users/Shared/Jenkins/Home/workspace/test/build_results/testresults.xml’

You can see that it is able to parse unit test result file successfully but giving error for coverage.xml

Note: Using SonarQube - Community Edition Version 7.7 (build 23042)

LucaIpp · May 10, 2019, 11:34am

@Falak_Shah As far as I know SonarQube Community Edition doesn’t support coverage analysis on branches, so this doesn’t sound like the same problem for me.

dmeneses · May 15, 2019, 7:38pm

Hi Falak,
If I understood correctly, your problem is not related to branches, it’s about a coverage report that wasn’t found by the scanner.
The warning indicates that there was no file with the pattern ‘build_results/coverage.xml’. Are you sure a coverage report is being generated to this location during the build in Jenkins?

Falak_Shah · May 16, 2019, 4:06am

Yes I checked in jenkins workspace. coverage.xml is generated under build_results folder. I tried putting coverage.xml in root folder as well. but getting same error. Infect tried with default setting of coverage report path in sonarqube, still getting same error.
But if you see scanner is able to find testresults.xml in same build_results folder and process that report.

dmeneses · May 16, 2019, 12:54pm

Ok. Do you have logs with debug enabled?

LucaIpp · May 16, 2019, 1:10pm

Would it be possible to open up a seperate thread for this issue so discussions around here will be in the problem related context