SonarJS appears to be rescanning entire file if there is no new code/only deleted code

SonarQube Server / Community Build
1

Template for a good bug report, formatted with Markdown:
SonarQube server, developer edition, 7.9.1.27448
SonarJS 6.1 (build 11503) installed

Deleting an unused function from a .js file (with no new code being added/changed) appears to trigger an entire rescan of the file. Thus failing quality gates and similar for old issues that haven’t been addressed yet.

The logs show:

WARN: File '/var/lib/jenkins/workspace/_ITSVC-4330-this-ticket-is-a-lie/includes/form_functions.php' was detected as changed but without having changed lines WARN: File '/var/lib/jenkins/workspace/_ITSVC-4330-this-ticket-is-a-lie/portal/js/global.js' was detected as changed but without having changed lines

The php file scans as expected (no new errors found, etc) however the .js file finds 1 new bug, 119 new code smells, etc.

to reproduce:

  • Do an initial scan of a legacy project that includes .js code. Preferably, legacy code that would fail current/modern quality gates… But it will “pass” because it’s the initial scan. (in our case, it was php and js)
  • After that scan, create a new branch. delete some functions or other code from the .js, do not make any other changes (don’t fix typos, don’t add comments, nothing else new/changed)
  • Run a new “branch aware” scan.

You’ll find the above warnings in the log. the PHP files will show as epxected (no new errors/vulnerabilities/smells) but the .js file will show all of the issues as “new”

3

Hi,

Are all the lines in that JS file highlighted as “new” (yellow background)?
Is it a short living branch?

Any chance we could get the scanner logs with debug enabled?

It looks to me that git is not finding the merge base commit for whatever reason. Strange thing is that in v7.9.1 that doesn’t explain by itself why issues would appear as new in the short lived branch.

4

Duarte,

Thanks for the response.

My team has been trying to debug this issue, and determine workarounds so, I’ll need to create a test project, and test branch to duplicate the issues again in a pristine environment. As such, I can’t produce the debug logs until tomorrow, and I can’t answer if the file is being highlighted as new.

That said, you mentioned that it seems like git can;t find the merge base. I can tell you that this team did specifically create the branch via the jira/bitbucket UI, not natively in git. so now I’m wondering if maybe that factors in (though, I have tried to duplicate the issue via that method, but still only see the issue when I delete lines from the .js file.)

I’ll get the requested info for you tomorrow or possibly monday. Thanks again.

1 Like
5

Sorry this took so long. I needed to re-create the environment, get approval to post this etc.

The below output is from Jenkins. I’ve included the relevant sonar-scanner logs (set in debug mode)
The very last line shows the commit failing a quality gate, where the qualitygate shows a ton of new bugs, and code smells (as previously mentioned, this is legacy code we’re attempting to clean up, so a lot of these old issues have been inherited) However, the code changes to the .php and .js files, are simply line deletions of entire functions. The .php files seem to be ok, the .js files seem to be deemed new code.

Injecting SonarQube environment variables using the configuration: sonarqube
[Pipeline] {
[Pipeline] sh
+ /var/lib/jenkins/tools/hudson.plugins.sonar.SonarRunnerInstallation/sonarqube/bin/sonar-scanner -Dsonar.branch.name=ITSVC-4-fake-issue
INFO: Scanner configuration file: /var/lib/jenkins/tools/hudson.plugins.sonar.SonarRunnerInstallation/sonarqube/conf/sonar-scanner.properties
INFO: Project root configuration file: /var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/sonar-project.properties
INFO: SonarQube Scanner 4.2.0.1873
INFO: Java 1.8.0_171 Oracle Corporation (64-bit)
INFO: Linux 3.16.0-4-amd64 amd64
INFO: User cache: /home/jenkins/.sonar/cache
INFO: SonarQube server 7.9.1
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
WARN: SonarScanner will require Java 11+ to run starting in SonarQube 8.x
INFO: Load global settings
INFO: Load global settings (done) | time=457ms
INFO: Server id: D881460E-AWqX06zVVnUYfXnRslQl
INFO: User cache: /home/jenkins/.sonar/cache
INFO: Load/download plugins
INFO: Load plugins index
INFO: Load plugins index (done) | time=156ms
INFO: Load/download plugins (done) | time=221ms
INFO: Loaded core extensions: developer-scanner
INFO: Process project properties
INFO: Project key: exampleproject
INFO: Base dir: /var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue
INFO: Working dir: /var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/.scannerwork
INFO: Load project settings for component key: 'exampleproject'
INFO: Load project settings for component key: 'exampleproject' (done) | time=72ms
INFO: Load project branches
INFO: Load project branches (done) | time=92ms
INFO: Load project pull requests
INFO: Load project pull requests (done) | time=60ms
INFO: Load branch configuration
INFO: Load branch configuration (done) | time=4ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=158ms
INFO: Detected Jenkins
INFO: Load active rules
INFO: Load active rules (done) | time=1690ms
INFO: Branch name: ITSVC-4-fake-issue, type: short living
INFO: SCM collecting changed files in the branch
INFO: SCM collecting changed files in the branch (done) | time=257ms
INFO: Indexing files...
INFO: Project configuration:
INFO:   Excluded sources: includes/AWSSDKforPHP/**, includes/S3/**, includes/PHPExcel/**, includes/dompdf/**, includes/fpdf153/**, includes/phpmailer/**
WARN: Invalid character encountered in file /var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/includes/phpreports/PHPReportsUtil.php at line 329 for encoding UTF-8. Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.
WARN: Invalid character encountered in file /var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/exampleproject/js/tablekit.js at line 606 for encoding UTF-8. Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.
INFO: 1477 files indexed
INFO: 0 files ignored because of inclusion/exclusion patterns
INFO: 0 files ignored because of scm ignore settings
INFO: Quality profile for css: Sonar way
INFO: Quality profile for js: Sonar way
INFO: Quality profile for php: exampleproject_test1
INFO: Quality profile for py: Sonar way
INFO: Quality profile for shell: ShellCheck
INFO: Quality profile for web: Sonar way
INFO: Quality profile for xml: Sonar way
INFO: ------------- Run sensors on module exampleproject
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=807ms
INFO: Sensor Python Sensor [python]
INFO: Sensor Python Sensor [python] (done) | time=1834ms
INFO: Sensor Cobertura Sensor for Python coverage [python]
INFO: Sensor Cobertura Sensor for Python coverage [python] (done) | time=88ms
INFO: Sensor PythonXUnitSensor [python]
INFO: Sensor PythonXUnitSensor [python] (done) | time=62ms
INFO: Sensor SonarCSS Metrics [cssfamily]
INFO: Sensor SonarCSS Metrics [cssfamily] (done) | time=3659ms
INFO: Sensor SonarCSS Rules [cssfamily]
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/includes/admin_body.php, line 2572, Unknown word
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/includes/auth.php, line 231, Unknown word
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/includes/header.php, line 247, Unknown word
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/includes/reports_functions.php, line 686, Unknown word
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/exampleproject/700_dial_plan_configuration.php, line 280, Unknown word
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/exampleproject/dialplan.php, line 1153, Unknown word
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/exampleproject/extensions.php, line 2345, Unclosed string
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/exampleproject/extspeeddials.php, line 275, Unknown word
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/exampleproject/fd_manager.php, line 198, Unknown word
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/exampleproject/greatPlainsIntegration.php, line 130, Unknown word
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/exampleproject/sbh_manager.php, line 199, Unknown word
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/exampleproject/scheduling_popup.html, line 749, Unknown word
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/exampleproject/starcenter3_manager.html, line 642, Unknown word
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/exampleproject/starwatch_manager.html, line 133, Unknown word
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/exampleproject/tax_exemption/index.php, line 294, Unknown word
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/exampleproject/tax_exemption/download/index.php, line 294, Unknown word
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/exampleproject/tax_exemption/includes/index.php, line 294, Unknown word
ERROR: Failed to parse file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/exampleproject/mass_config/includes/functions/draw_tree_view.php, line 65, Unknown word
INFO: Sensor SonarCSS Rules [cssfamily] (done) | time=16062ms
INFO: Sensor JaCoCo XML Report Importer [jacoco]
INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=17ms
INFO: Sensor ShellCheck Sensor [shellcheck]
INFO: Sensor ShellCheck Sensor [shellcheck] (done) | time=2061ms
INFO: Sensor JavaScript analysis [javascript]
INFO: 77 source files to be analyzed
INFO: 33/77 files analyzed, current file: exampleproject/js/global.js
INFO: 46/77 files analyzed, current file: exampleproject/js/maint_window.js
INFO: 59/77 files analyzed, current file: exampleproject/prototype.js
INFO: 77/77 source files have been analyzed
INFO: Sensor SonarJS [javascript]
INFO: 77 source files to be analyzed
INFO: 6/77 files analyzed, current file: exampleproject/dashboard/dashboard.js
INFO: 11/77 files analyzed, current file: exampleproject/js/EasyEdits.js
INFO: 23/77 files analyzed, current file: exampleproject/js/circuit_scw.js
INFO: 33/77 files analyzed, current file: exampleproject/js/global.js
INFO: 41/77 files analyzed, current file: exampleproject/js/jquery.weekcalendar.js
INFO: 59/77 files analyzed, current file: exampleproject/prototype.js
INFO: 68/77 files analyzed, current file: exampleproject/scriptaculous/slider.js
INFO: Sensor SonarJS [javascript] (done) | time=75065ms
INFO: 77/77 source files have been analyzed
INFO: Sensor JavaScript analysis [javascript] (done) | time=120751ms
INFO: Sensor JavaXmlSensor [java]
INFO: 15 source files to be analyzed
INFO: Sensor JavaXmlSensor [java] (done) | time=3540ms
INFO: Sensor HTML [web]
INFO: 15/15 source files have been analyzed
INFO: Sensor HTML [web] (done) | time=11885ms
INFO: Sensor XML Sensor [xml]
INFO: 48 source files to be analyzed
WARN: Unable to analyse file file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/includes/phpreports/PHPReport.xsd;
WARN: Unable to analyse file file:///var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/exampleproject/crossdomain.xml;
INFO: Sensor XML Sensor [xml] (done) | time=3010ms
INFO: Sensor PHP sensor [php]
INFO: 48/48 source files have been analyzed
INFO: 642 source files to be analyzed
WARN: Failed to build control flow graph for file [includes/BillingAccess.php] at line 326 (activate debug logs for more details)
INFO: 15/642 files analyzed, current file: includes/Domain/Model/Customer/Customer.php
INFO: 43/642 files analyzed, current file: includes/WestAccess.php
INFO: 47/642 files analyzed, current file: includes/auth.php
INFO: 57/642 files analyzed, current file: includes/business_continuity_report.php
INFO: 83/642 files analyzed, current file: includes/classes/EEWorksheet.php
INFO: 116/642 files analyzed, current file: includes/configuration_summary.php
INFO: 150/642 files analyzed, current file: includes/dealerinfo_header.php
INFO: 170/642 files analyzed, current file: includes/libXML.php
INFO: 195/642 files analyzed, current file: includes/orderTracking_header.php
INFO: 203/642 files analyzed, current file: includes/phone_numbers/form.php
INFO: 260/642 files analyzed, current file: includes/portingAdminAddNewNumbers.php
INFO: 260/642 files analyzed, current file: includes/portingAdminAddNewNumbers.php
INFO: 290/642 files analyzed, current file: includes/starrecovery_header.php
INFO: 301/642 files analyzed, current file: includes/tracking_definitions.php
INFO: 304/642 files analyzed, current file: includes/update_icr.php
INFO: 354/642 files analyzed, current file: exampleproject/admin/login-as.php
WARN: Failed to build control flow graph for file [exampleproject/cold_equipment/customer_inventory.php] at line 1 (activate debug logs for more details)
INFO: 425/642 files analyzed, current file: exampleproject/controllers/sample.php
INFO: 445/642 files analyzed, current file: exampleproject/dealer_manager.php
INFO: 453/642 files analyzed, current file: exampleproject/didlookup.php
INFO: 470/642 files analyzed, current file: exampleproject/fs/rmt_dialplan.php
INFO: 520/642 files analyzed, current file: exampleproject/mass_config/import_area_map.php
INFO: 580/642 files analyzed, current file: exampleproject/service/exampleproject2sd.php
INFO: 603/642 files analyzed, current file: exampleproject/tax_exemption/download/historical_records.php
WARN: Failed to build control flow graph for file [exampleproject/tech_support/endpoint_config_patch.php] at line 1 (activate debug logs for more details)
INFO: 642/642 source files have been analyzed
INFO: No PHPUnit test report provided (see 'sonar.php.tests.reportPath' property)
INFO: No PHPUnit coverage reports provided (see 'sonar.php.coverage.reportPaths' property)
INFO: Sensor PHP sensor [php] (done) | time=245374ms
INFO: Sensor Analyzer for "php.ini" files [php]
INFO: Sensor Analyzer for "php.ini" files [php] (done) | time=13ms
INFO: ------------- Run sensors on project
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=1056ms
INFO: SCM provider for this project is: git
INFO: 1 files to be analyzed
INFO:
INFO: 1/1 files analyzed
INFO: 92 files had no CPD blocks
INFO: Calculating CPD for 758 files
INFO: CPD calculation finished
INFO: SCM writing changed lines
WARN: File '/var/lib/jenkins/workspace/tal_sonarqube_ITSVC-4-fake-issue/exampleproject/js/global.js' was detected as changed but without having changed lines
INFO: SCM writing changed lines (done) | time=924ms
INFO: Analysis report generated in 2541ms, dir size=3 MB
INFO: Analysis report compressed in 3406ms, zip size=1 MB
INFO: Analysis report uploaded in 1307ms
INFO: ANALYSIS SUCCESSFUL, you can browse https://sonar.sv.xxxxxxxx.tld/dashboard?id=exampleproject&branch=ITSVC-4-fake-issue&resolved=false
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at https://sonar.sv.xxxxxxxx.tld/api/ce/task?id=AW_tfpNZlugsTqUCSCW8
INFO: Analysis total time: 7:38.582 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 7:41.711s
INFO: Final Memory: 22M/445M
INFO: ------------------------------------------------------------------------
[Pipeline] }
[Pipeline] // withSonarQubeEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (SonarQube Quality Gate)
[Pipeline] tool
[Pipeline] envVarsForTool
[Pipeline] withEnv
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 10 min
[Pipeline] {
[Pipeline] waitForQualityGate
Checking status of SonarQube task 'AW_tfpNZlugsTqUCSCW8' on server 'sonarqube'
SonarQube task 'AW_tfpNZlugsTqUCSCW8' status is 'SUCCESS'
SonarQube task 'AW_tfpNZlugsTqUCSCW8' completed. Quality gate is 'ERROR'
6

@dmeneses any thoughts on the scanner log I had included? We’re still at a loss for explaining this behavior.

7

I don’t see anything abnormal in the logs.
The log level is not set to debug. In debug you’d get among other things the git merge sha1, which you can use to confirm that the diff detected by the scanner is correct.

About the problem itself.
All files are scanned, whether they were changed or not. The warning about not having changed lines is a bug and it could affect how lines are classified as “changed” or “not changed” in the branch.
However, in v7.9.1 the detection of “new issues” is not related to the lines being classified as changed. A comparison is made between the issues in both branches instead.

I tried to reproduce the problem and failed. Could you indicate what version of the SCM Git plugin you have in SonarQube? That way I can try again with a matching plugin.

8

@dmeneses Sorry about that, I thought I had enabled debugging, but I did it for the test project, not the real one demonstrating the issue.

Do you have a way I can send the debug log privately?

9

I sent you a private message, you can reply to it with the logs.