SonarScanner - Java analyzer cache incorrect?

Dennis_DECA · October 10, 2023, 1:37pm

Must-share information (formatted with Markdown):

SonarQube server 9.9.1.69595 LTS
SonarScanner 5.0.1.3006

Setup:
Github.com pull request scan. Self hosted Github runner.
Workflow snippet:

      - name: SonarQube Scan
        if: env.RUN_SonarQube == 'true'
        uses: sonarsource/sonarqube-scan-action@master
        env:
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
          SONAR_HOST_URL: ${{ vars.SONAR_HOST }}
          SONAR_SCANNER_OPTS: -Xmx6500m
        with:
          projectBaseDir: core
          args: >
            -Dsonar.projectBaseDir=/github/workspace/core
            -Dsonar.working.directory=/github/workspace/core/.scannerwork
            -Dsonar.java.skipUnchanged=true

Branch setup:

develop (main/default)
master
release_1
release_2

Problem:
Server-side caching is enabled. The Java analyzer was able to leverage cached data from previous analyses for 13459 out of 17735 files. These files will not be parsed.

=> The PR has only a few changed files. This case looks incorrect to me. When there is a low load, it seems to be fine. Under high load, it looks like this, leading to higher scanning times. Is there a way to clear the cache or make sure the correct one is used?

Log:

2023-10-10T09:34:33.0652413Z ##[group]Run sonarsource/sonarqube-scan-action@master
2023-10-10T09:34:33.0653021Z with:
2023-10-10T09:34:33.0653439Z   projectBaseDir: core
2023-10-10T09:34:33.0654147Z   args: -Dsonar.projectBaseDir=/github/workspace/core -Dsonar.working.directory=/github/workspace/core/.scannerwork -Dsonar.java.skipUnchanged=true -Dsonar.verbose=true
2023-10-10T09:34:33.0654858Z env:
2023-10-10T09:34:33.0655243Z   RUN_SonarQube: true
2023-10-10T09:34:33.0655657Z   SKIP_SonarQube: false
2023-10-10T09:34:33.0656247Z   SONAR_TOKEN: ***
2023-10-10T09:34:33.0656736Z   SONAR_HOST_URL: http://sonar.AAAAA.be:9000
2023-10-10T09:34:33.0657211Z   SONAR_SCANNER_OPTS: -Xmx6500m
2023-10-10T09:34:33.0657611Z ##[endgroup]
2023-10-10T09:34:33.0894375Z ##[command]/usr/bin/docker run --name e99f520a583f4ac3684989b59ad749c682a3ee_0eaa62 --label e99f52 --workdir /github/workspace --rm -e "RUN_SonarQube" -e "SKIP_SonarQube" -e "SONAR_TOKEN" -e "SONAR_HOST_URL" -e "SONAR_SCANNER_OPTS" -e "INPUT_PROJECTBASEDIR" -e "INPUT_ARGS" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e GITHUB_ACTIONS=true -e CI=true --entrypoint "/entrypoint.sh" -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/opt/github-runner/action-runner/_work/_temp/_github_home":"/github/home" -v "/opt/github-runner/action-runner/_work/_temp/_github_workflow":"/github/workflow" -v "/opt/github-runner/action-runner/_work/_temp/_runner_file_commands":"/github/file_commands" -v "/opt/github-runner/action-runner/_work/BBBBB-core/BBBBB-core":"/github/workspace" e99f52:0a583f4ac3684989b59ad749c682a3ee -Dsonar.projectBaseDir=/github/workspace/core -Dsonar.working.directory=/github/workspace/core/.scannerwork -Dsonar.java.skipUnchanged=true -Dsonar.verbose=true
2023-10-10T09:34:33.8744489Z WARN: Property 'sonar.projectBaseDir' with value 'core' is overridden with value '/github/workspace/core'
2023-10-10T09:34:33.8781836Z INFO: Scanner configuration file: /opt/sonar-scanner/conf/sonar-scanner.properties
2023-10-10T09:34:33.8804620Z INFO: Project root configuration file: /github/workspace/core/sonar-project.properties
2023-10-10T09:34:33.9185379Z 09:34:33.913 INFO: SonarScanner 5.0.1.3006
2023-10-10T09:34:33.9187499Z 09:34:33.918 INFO: Java 17.0.8 Alpine (64-bit)
2023-10-10T09:34:33.9189338Z 09:34:33.918 INFO: Linux 5.15.0-69-generic amd64
2023-10-10T09:34:33.9208799Z 09:34:33.920 INFO: SONAR_SCANNER_OPTS=-Xmx6500m
2023-10-10T09:34:34.0542323Z 09:34:34.053 DEBUG: keyStore is : 
2023-10-10T09:34:34.0543543Z 09:34:34.053 DEBUG: keyStore type is : pkcs12
2023-10-10T09:34:34.0544600Z 09:34:34.053 DEBUG: keyStore provider is : 
2023-10-10T09:34:34.0545584Z 09:34:34.053 DEBUG: init keystore
2023-10-10T09:34:34.0546905Z 09:34:34.054 DEBUG: init keymanager of type SunX509
2023-10-10T09:34:34.1468775Z 09:34:34.146 DEBUG: Create: /opt/sonar-scanner/.sonar/cache
2023-10-10T09:34:34.1471163Z 09:34:34.146 INFO: User cache: /opt/sonar-scanner/.sonar/cache
2023-10-10T09:34:34.1472706Z 09:34:34.146 DEBUG: Create: /opt/sonar-scanner/.sonar/cache/_tmp
2023-10-10T09:34:34.1495874Z 09:34:34.149 DEBUG: Extract sonar-scanner-api-batch in temp...
2023-10-10T09:34:34.1521892Z 09:34:34.151 DEBUG: Get bootstrap index...
2023-10-10T09:34:34.1523184Z 09:34:34.151 DEBUG: Download: http://sonar.AAAAA.be:9000/batch/index
2023-10-10T09:34:34.2150402Z 09:34:34.214 DEBUG: Get bootstrap completed
2023-10-10T09:34:34.2174052Z 09:34:34.216 DEBUG: Download http://sonar.AAAAA.be:9000/batch/file?name=scanner-developer-9.9.1.69595-all.jar to /opt/sonar-scanner/.sonar/cache/_tmp/fileCache9617005575399295100.tmp
2023-10-10T09:34:34.7209512Z 09:34:34.720 DEBUG: Create isolated classloader...
2023-10-10T09:34:34.7371832Z 09:34:34.736 DEBUG: Start temp cleaning...
2023-10-10T09:34:34.7396226Z 09:34:34.739 DEBUG: Temp cleaning done
2023-10-10T09:34:34.7397301Z 09:34:34.739 DEBUG: Execution getVersion
2023-10-10T09:34:34.7490570Z 09:34:34.748 INFO: Analyzing on SonarQube server 9.9.1.69595
2023-10-10T09:34:34.7493513Z 09:34:34.749 INFO: Default locale: "en_US", source code encoding: "UTF-8"
2023-10-10T09:34:34.7498243Z 09:34:34.749 DEBUG: Work directory: /github/workspace/core/.scannerwork
2023-10-10T09:34:34.7505796Z 09:34:34.750 DEBUG: Execution execute
2023-10-10T09:34:35.0067252Z 09:34:35.006 DEBUG: Developer 9.9.1.69595
2023-10-10T09:34:35.1976815Z 09:34:35.197 INFO: Load global settings
2023-10-10T09:34:35.2347074Z 09:34:35.234 DEBUG: GET 200 http://sonar.AAAAA.be:9000/api/settings/values.protobuf | time=35ms
2023-10-10T09:34:35.3036264Z 09:34:35.303 INFO: Load global settings (done) | time=106ms
2023-10-10T09:34:35.3118351Z 09:34:35.311 INFO: Server id: CCCCC
2023-10-10T09:34:35.3158793Z 09:34:35.314 DEBUG: Create : /opt/sonar-scanner/.sonar/_tmp
2023-10-10T09:34:35.3160183Z 09:34:35.315 INFO: User cache: /opt/sonar-scanner/.sonar/cache
2023-10-10T09:34:35.3191390Z 09:34:35.318 INFO: Load/download plugins
2023-10-10T09:34:35.3192619Z 09:34:35.318 INFO: Load plugins index
2023-10-10T09:34:44.4364449Z 09:34:44.435 INFO: Load/download plugins (done) | time=9117ms
2023-10-10T09:34:44.7722905Z 09:34:44.771 DEBUG: Plugins:
2023-10-10T09:34:44.7723933Z 09:34:44.771 DEBUG:   * IaC Code Quality and Security 1.11.0.2847 (iac)
2023-10-10T09:34:44.7731647Z 09:34:44.771 DEBUG:   * Java Custom Rules for BBBBB Core 1.0-SNAPSHOT (javacustomBBBBBcore)
2023-10-10T09:34:44.7734856Z 09:34:44.772 DEBUG:   * PL/SQL Code Quality and Security 3.8.0.4948 (plsql)
2023-10-10T09:34:44.7735561Z 09:34:44.772 DEBUG:   * Scala Code Quality and Security 1.11.0.3905 (sonarscala)
2023-10-10T09:34:44.7736216Z 09:34:44.772 DEBUG:   * C# Code Quality and Security 8.51.0.59060 (csharp)
2023-10-10T09:34:44.7736798Z 09:34:44.772 DEBUG:   * Vulnerability Analysis 9.9.0.19083 (security)
2023-10-10T09:34:44.7738538Z 09:34:44.772 DEBUG:   * Java Code Quality and Security 7.16.0.30901 (java)
2023-10-10T09:34:44.7743336Z 09:34:44.772 DEBUG:   * HTML Code Quality and Security 3.7.1.3306 (web)
2023-10-10T09:34:44.7744007Z 09:34:44.772 DEBUG:   * Flex Code Quality and Security 2.8.0.3166 (flex)
2023-10-10T09:34:44.7744631Z 09:34:44.773 DEBUG:   * XML Code Quality and Security 2.7.0.3820 (xml)
2023-10-10T09:34:44.7745220Z 09:34:44.773 DEBUG:   * Text Code Quality and Security 2.0.2.1090 (text)
2023-10-10T09:34:44.7745824Z 09:34:44.773 DEBUG:   * VB.NET Code Quality and Security 8.51.0.59060 (vbnet)
2023-10-10T09:34:44.7746425Z 09:34:44.773 DEBUG:   * Swift Code Quality and Security 4.8.0.5759 (swift)
2023-10-10T09:34:44.7747206Z 09:34:44.773 DEBUG:   * Dependency-Check 3.1.0 (dependencycheck)
2023-10-10T09:34:44.7747808Z 09:34:44.773 DEBUG:   * CFamily Code Quality and Security 6.41.0.60884 (cpp)
2023-10-10T09:34:44.7748470Z 09:34:44.773 DEBUG:   * Python Code Quality and Security 3.24.0.10784 (python)
2023-10-10T09:34:44.7754293Z 09:34:44.774 DEBUG:   * Dataflow Bug Detection Rules for Python 1.10.0.3046 (dbdpythonfrontend)
2023-10-10T09:34:44.7760311Z 09:34:44.774 DEBUG:   * Dataflow Bug Detection 1.10.0.3046 (dbd)
2023-10-10T09:34:44.7761019Z 09:34:44.774 DEBUG:   * Go Code Quality and Security 1.11.0.3905 (go)
2023-10-10T09:34:44.7761562Z 09:34:44.774 DEBUG:   * JaCoCo 1.3.0.1538 (jacoco)
2023-10-10T09:34:44.7762104Z 09:34:44.774 DEBUG:   * Kotlin Code Quality and Security 2.12.0.1956 (kotlin)
2023-10-10T09:34:44.7762757Z 09:34:44.774 DEBUG:   * Dataflow Bug Detection Rules for Java 1.10.0.3046 (dbdjavafrontend)
2023-10-10T09:34:44.7763519Z 09:34:44.774 DEBUG:   * T-SQL Code Quality and Security 1.7.0.5449 (tsql)
2023-10-10T09:34:44.7764198Z 09:34:44.774 DEBUG:   * JavaScript/TypeScript/CSS Code Quality and Security 9.13.0.20537 (javascript)
2023-10-10T09:34:44.7764838Z 09:34:44.774 DEBUG:   * Ruby Code Quality and Security 1.11.0.3905 (ruby)
2023-10-10T09:34:44.7765486Z 09:34:44.774 DEBUG:   * Vulnerability Rules for C# 9.9.0.19083 (securitycsharpfrontend)
2023-10-10T09:34:44.7766153Z 09:34:44.775 DEBUG:   * Vulnerability Rules for Java 9.9.0.19083 (securityjavafrontend)
2023-10-10T09:34:44.7766816Z 09:34:44.775 DEBUG:   * Vulnerability Rules for JS 9.9.0.19083 (securityjsfrontend)
2023-10-10T09:34:44.7767578Z 09:34:44.775 DEBUG:   * Vulnerability Rules for Python 9.9.0.19083 (securitypythonfrontend)
2023-10-10T09:34:44.7768257Z 09:34:44.775 DEBUG:   * PHP Code Quality and Security 3.27.1.9352 (php)
2023-10-10T09:34:44.7768894Z 09:34:44.775 DEBUG:   * ABAP Code Quality and Security 3.11.0.4030 (abap)
2023-10-10T09:34:44.7769558Z 09:34:44.775 DEBUG:   * Configuration detection fot Code Quality and Security 1.2.0.267 (config)
2023-10-10T09:34:44.7770253Z 09:34:44.775 DEBUG:   * Vulnerability Rules for PHP 9.9.0.19083 (securityphpfrontend)
2023-10-10T09:34:44.8247142Z 09:34:44.820 INFO: Loaded core extensions: developer-scanner
2023-10-10T09:34:44.8344996Z 09:34:44.833 DEBUG: Installed core extension: developer-scanner
2023-10-10T09:34:45.2618205Z 09:34:45.260 INFO: Process project properties
2023-10-10T09:34:45.2693841Z 09:34:45.268 INFO: Process project properties (done) | time=8ms
2023-10-10T09:34:45.2717355Z 09:34:45.271 INFO: Execute project builders
2023-10-10T09:34:45.2724007Z 09:34:45.271 DEBUG: Execute project builder: org.sonar.plugins.csharp.CSharpGlobalProtobufFileProcessor
2023-10-10T09:34:45.2731395Z 09:34:45.272 DEBUG: Execute project builder: org.sonar.plugins.vbnet.VbNetGlobalProtobufFileProcessor
2023-10-10T09:34:45.2737274Z 09:34:45.273 INFO: Execute project builders (done) | time=2ms
2023-10-10T09:34:45.2759303Z 09:34:45.275 INFO: Project key: BBBBB:core:bamboo:develop
2023-10-10T09:34:45.2761059Z 09:34:45.275 INFO: Base dir: /github/workspace/core
2023-10-10T09:34:45.2767573Z 09:34:45.275 INFO: Working dir: /github/workspace/core/.scannerwork
2023-10-10T09:34:45.2768653Z 09:34:45.276 DEBUG: Project global encoding: UTF-8, default locale: en_US
2023-10-10T09:34:45.2846518Z 09:34:45.284 INFO: Load project settings for component key: 'BBBBB:core:bamboo:develop'
2023-10-10T09:34:45.3030210Z 09:34:45.302 DEBUG: GET 200 http://sonar.AAAAA.be:9000/api/settings/values.protobuf?component=BBBBB%3Acore%3Abamboo%3Adevelop | time=18ms
2023-10-10T09:34:45.3222259Z 09:34:45.321 INFO: Load project settings for component key: 'BBBBB:core:bamboo:develop' (done) | time=37ms
2023-10-10T09:34:45.4194737Z 09:34:45.418 INFO: Load project branches
2023-10-10T09:34:45.4458479Z 09:34:45.444 DEBUG: GET 200 http://sonar.AAAAA.be:9000/api/project_branches/list?project=BBBBB%3Acore%3Abamboo%3Adevelop | time=25ms
2023-10-10T09:34:45.4655140Z 09:34:45.464 INFO: Load project branches (done) | time=46ms
2023-10-10T09:34:45.4655842Z 09:34:45.465 INFO: Load branch configuration
2023-10-10T09:34:45.4676999Z 09:34:45.467 INFO: Detected branch/PR in 'GitHub Action'
2023-10-10T09:34:45.4679370Z 09:34:45.467 INFO: Auto-configuring pull request '18732'
2023-10-10T09:34:45.4701294Z 09:34:45.469 INFO: Load branch configuration (done) | time=4ms
2023-10-10T09:34:45.4778149Z 09:34:45.477 DEBUG: Creating module hierarchy
2023-10-10T09:34:45.4778937Z 09:34:45.477 DEBUG:   Init module 'BBBBB Core Bamboo develop'
2023-10-10T09:34:45.4784285Z 09:34:45.478 DEBUG:     Base dir: /github/workspace/core
2023-10-10T09:34:45.4785779Z 09:34:45.478 DEBUG:     Working dir: /github/workspace/core/.scannerwork
2023-10-10T09:34:45.4787373Z 09:34:45.478 DEBUG:     Module global encoding: UTF-8, default locale: en_US
2023-10-10T09:34:45.4946974Z 09:34:45.494 DEBUG: Available languages:
2023-10-10T09:34:45.4949777Z 09:34:45.494 DEBUG:   * Terraform => "terraform"
2023-10-10T09:34:45.4950541Z 09:34:45.494 DEBUG:   * CloudFormation => "cloudformation"
2023-10-10T09:34:45.4951201Z 09:34:45.494 DEBUG:   * Kubernetes => "kubernetes"
2023-10-10T09:34:45.4951679Z 09:34:45.494 DEBUG:   * Docker => "docker"
2023-10-10T09:34:45.4953369Z 09:34:45.495 DEBUG:   * PL/SQL => "plsql"
2023-10-10T09:34:45.4955213Z 09:34:45.495 DEBUG:   * Scala => "scala"
2023-10-10T09:34:45.4957815Z 09:34:45.495 DEBUG:   * C# => "cs"
2023-10-10T09:34:45.4958341Z 09:34:45.495 DEBUG:   * Java => "java"
2023-10-10T09:34:45.4958814Z 09:34:45.495 DEBUG:   * HTML => "web"
2023-10-10T09:34:45.4960036Z 09:34:45.495 DEBUG:   * JSP => "jsp"
2023-10-10T09:34:45.4960523Z 09:34:45.495 DEBUG:   * Flex => "flex"
2023-10-10T09:34:45.4961050Z 09:34:45.495 DEBUG:   * XML => "xml"
2023-10-10T09:34:45.4962876Z 09:34:45.496 DEBUG:   * Text => "text"
2023-10-10T09:34:45.4963627Z 09:34:45.496 DEBUG:   * Secrets => "secrets"
2023-10-10T09:34:45.4965991Z 09:34:45.496 DEBUG:   * VB.NET => "vbnet"
2023-10-10T09:34:45.4966448Z 09:34:45.496 DEBUG:   * Swift => "swift"
2023-10-10T09:34:45.4969086Z 09:34:45.496 DEBUG:   * Neutral => "neutral"
2023-10-10T09:34:45.4970081Z 09:34:45.496 DEBUG:   * C => "c"
2023-10-10T09:34:45.4972990Z 09:34:45.496 DEBUG:   * C++ => "cpp"
2023-10-10T09:34:45.4973738Z 09:34:45.496 DEBUG:   * Objective-C => "objc"
2023-10-10T09:34:45.4974194Z 09:34:45.496 DEBUG:   * Python => "py"
2023-10-10T09:34:45.4978107Z 09:34:45.496 DEBUG:   * Go => "go"
2023-10-10T09:34:45.4979189Z 09:34:45.497 DEBUG:   * Kotlin => "kotlin"
2023-10-10T09:34:45.4984608Z 09:34:45.497 DEBUG:   * T-SQL => "tsql"
2023-10-10T09:34:45.4985241Z 09:34:45.497 DEBUG:   * JavaScript => "js"
2023-10-10T09:34:45.4987078Z 09:34:45.497 DEBUG:   * TypeScript => "ts"
2023-10-10T09:34:45.4987529Z 09:34:45.498 DEBUG:   * CSS => "css"
2023-10-10T09:34:45.4988073Z 09:34:45.498 DEBUG:   * Ruby => "ruby"
2023-10-10T09:34:45.4988591Z 09:34:45.498 DEBUG:   * PHP => "php"
2023-10-10T09:34:45.4989732Z 09:34:45.498 DEBUG:   * ABAP => "abap"
2023-10-10T09:34:45.4990183Z 09:34:45.498 DEBUG:   * YAML => "yaml"
2023-10-10T09:34:45.4990631Z 09:34:45.498 DEBUG:   * JSON => "json"
2023-10-10T09:34:45.5379966Z 09:34:45.537 INFO: Auto-configuring with CI 'Github Actions'
2023-10-10T09:34:45.5401802Z 09:34:45.539 INFO: Load quality profiles
2023-10-10T09:34:45.5816208Z 09:34:45.581 DEBUG: GET 200 http://sonar.AAAAA.be:9000/api/qualityprofiles/search.protobuf?project=BBBBB%3Acore%3Abamboo%3Adevelop | time=41ms
2023-10-10T09:34:45.6105426Z 09:34:45.610 INFO: Load quality profiles (done) | time=71ms
2023-10-10T09:34:45.6157372Z 09:34:45.615 INFO: Load active rules
2023-10-10T09:34:47.6953309Z 09:34:47.694 INFO: Load active rules (done) | time=2079ms
2023-10-10T09:34:47.6994380Z 09:34:47.699 INFO: Load analysis cache
2023-10-10T09:34:47.7083438Z 09:34:47.707 DEBUG: GET 200 http://sonar.AAAAA.be:9000/api/analysis_cache/get?project=BBBBB:core:bamboo:develop&branch=develop | time=8ms
2023-10-10T09:34:49.9565401Z 09:34:49.955 INFO: Load analysis cache | time=2256ms
2023-10-10T09:34:49.9832255Z 09:34:49.982 INFO: Pull request 18732 for merge into develop from AAA/ABC-123/BBBBBCLI-ends-in-error
2023-10-10T09:34:50.0021650Z 09:34:50.001 INFO: Load project repositories
2023-10-10T09:34:50.5024673Z 09:34:50.501 DEBUG: GET 200 http://sonar.AAAAA.be:9000/batch/project.protobuf?key=BBBBB%3Acore%3Abamboo%3Adevelop&branch=develop | time=500ms
2023-10-10T09:34:50.5817915Z 09:34:50.581 INFO: Load project repositories (done) | time=579ms
2023-10-10T09:34:50.5833573Z 09:34:50.582 INFO: SCM collecting changed files in the branch
2023-10-10T09:34:50.5970449Z 09:34:50.596 DEBUG: loading config FileBasedConfig[/tmp/jgit/config]
2023-10-10T09:34:50.5976411Z 09:34:50.597 DEBUG: readpipe [/usr/bin/git, --version],/usr/bin
2023-10-10T09:34:50.6556652Z 09:34:50.654 DEBUG: readpipe may return 'git version 2.38.5'
2023-10-10T09:34:50.6557855Z 09:34:50.655 DEBUG: remaining output:
2023-10-10T09:34:50.6558458Z 
2023-10-10T09:34:50.6563938Z 09:34:50.655 DEBUG: readpipe [/usr/bin/git, config, --system, --show-origin, --list, -z],/usr/bin
2023-10-10T09:34:50.6587640Z 09:34:50.658 DEBUG: readpipe may return 'null'
2023-10-10T09:34:50.6588700Z 09:34:50.658 DEBUG: remaining output:
2023-10-10T09:34:50.6589264Z 
2023-10-10T09:34:50.6782435Z 09:34:50.677 DEBUG: readpipe rc=128
2023-10-10T09:34:50.6783675Z 09:34:50.677 DEBUG: Exception caught during execution of command '[/usr/bin/git, config, --system, --show-origin, --list, -z]' in '/usr/bin', return code '128', error message 'fatal: unable to read config file '/etc/gitconfig': No such file or directory
2023-10-10T09:34:50.6784433Z '
2023-10-10T09:34:50.6800040Z 09:34:50.679 DEBUG: loading config FileBasedConfig[/root/.gitconfig]
2023-10-10T09:34:50.7012142Z 09:34:50.700 DEBUG: Thread[JGit-FileStoreAttributeReader-1,5,main]: start measure timestamp resolution /github/workspace (/dev/mapper/ubuntu-opt) in /github/workspace/core/.git
2023-10-10T09:34:50.7027879Z 09:34:50.702 DEBUG: Thread[JGit-FileStoreAttributeReader-1,5,main]: end measure timestamp resolution /github/workspace (/dev/mapper/ubuntu-opt) in /github/workspace/core/.git; got PT0.000007432S
2023-10-10T09:34:50.7033435Z 09:34:50.702 DEBUG: Thread[JGit-FileStoreAttributeReader-1,5,main]: start measure minimal racy interval in /github/workspace/core/.git
2023-10-10T09:34:50.8007064Z 09:34:50.800 DEBUG: Thread[main,5,main]: use fallback timestamp resolution for directory /github/workspace/core/.git
2023-10-10T09:34:50.9274356Z 09:34:50.926 INFO: Merge base sha1: 24509f4a89d2ef24715521bbf8f04468ec038b17
2023-10-10T09:34:50.9599382Z 09:34:50.959 INFO: SCM collecting changed files in the branch (done) | time=377ms
2023-10-10T09:34:50.9600024Z 09:34:50.959 DEBUG: SCM reported 1 file changed in the branch
2023-10-10T09:34:50.9847229Z 09:34:50.984 DEBUG: Will ignore generated code
2023-10-10T09:34:50.9858399Z 09:34:50.985 DEBUG: Will ignore generated code
2023-10-10T09:34:50.9943639Z 09:34:50.993 INFO: Indexing files...
2023-10-10T09:34:50.9947229Z 09:34:50.993 INFO: Project configuration:
2023-10-10T09:34:50.9950963Z 09:34:50.994 INFO:   Excluded sources: **/*Stub.java, BBBBBRoot*/query/**/*.*
2023-10-10T09:34:52.1728944Z 09:34:52.172 DEBUG: 38601 non excluded files in this Git repository
2023-10-10T09:36:16.9696012Z 09:36:16.969 INFO: Server-side caching is enabled. The Java analyzer was able to leverage cached data from previous analyses for 13459 out of 17735 files. These files will not be parsed.
2023-10-10T09:36:16.9745864Z 09:36:16.974 INFO: Using ECJ batch to parse 4276 Main java source files with batch size 340 KB.
2023-10-10T09:36:17.0822154Z 09:36:17.081 INFO: Starting batch processing.

ganncamp · October 11, 2023, 5:14pm

Hi,

I believe this is about the web of other classes that use/are used by the changed file. Do you have a high degree of coupling?

Ann

Dennis_DECA · October 13, 2023, 11:59am

Hi Ann,

Im not a developer, but I get what you are hinting at. In the case you are referring to, I can understand and agree.
But I also see plenty of cases like this were a single XML file was changed with the same result. In this case there are no classes or coupling.

ganncamp · October 13, 2023, 1:25pm

Hi,

Are we having the same discussion in two separate threads?

Ann

Dennis_DECA · October 13, 2023, 3:22pm

Hi Ann,
I created 2 topics to avoid this, but it seems there is some overlap to the issues im investigating.
For the topic you mentioned I want to know how ignored files are handled by the scanner. I have an answer for that question in that topic.

For this topic I would like to keep the focus on the diffs detected.

1st question:
Could you explain how this mechanism works in the sonar scanner?

Server-side caching is enabled. The Java analyzer was able to leverage cached data from previous analyses for 13459 out of 17735 files. These files will not be parsed.

The way I understand it:

Cache is downloaded
Changed files are checked and referenced files/classes are included in the scope
The delta between files in cache and files changed + referenced files/classes are excluded from the analyzers

Do I understand this correctly?

2nd question:
How can a single .xml file change in a PR trigger a scope 4.000 files to be scanned by the analyzers?

Based on my understanding, something is wrong. My suspicion is an incorrect cache.
3th question:
How can I recreate the cache? How can I delete the cache? Where is the cache coming from? Downloaded from sonarQube server? Is there a cache per branch?

ganncamp · October 13, 2023, 4:10pm

Hi,

What is the file? If it’s a POM, I’m not surprised the analyzer saw a lot changed.

For the rest, I’m going to flag this for more expert eyes.

Ann

Dorian_Burihabwa · October 18, 2023, 12:50pm

Hey @Dennis_DECA,
Thanks for your post. That number of files that need to be re-analyzed from scratch looks pretty high.

I think Ann hinted at the right point to start the investigation: if a file that is depended upon by a lot of other files changes, it might have a cascading effect.

1st question:
Could you explain how this mechanism works in the sonar scanner?

Sure, as we analyze the base branch of the project, we build a cache that can then be leveraged by PRs that branch off from it.
The number of files for which we manage to leverage the cache depends on the detection of changes:

sources (will change in your versioning and potential CI build actions)
class files (will change with your build system and compiler)
dependencies between files (will change based on your code’s logic)

We tend to stay on the side of safety meaning that whenever there is a doubt about whether a file has been changed, we analyze it from scratch and we do not leverage the entry in the cache.

2nd question:
How can a single .xml file change in a PR trigger a scope 4.000 files to be scanned by the analyzers?

Consider the impact that changing a plugin within your pom.xml or even explicitly setting a property might have on the sources (eg: enforced formatting) and binaries (eg: obfuscation) at the end of the build.
Again, we try to stay on the safe side to serve reports that are as fresh as possible

3th question:
How can I recreate the cache? How can I delete the cache? Where is the cache coming from? Downloaded from sonarQube server? Is there a cache per branch?

The cache is coming from your SonarQube instance. You should be able to recreate it by re-analyzing your base branch in build conditions that are similar enough (ie: a similar pom).
As for dropping the cache from your SonarQube instance entirely, I am not sure that this is possible without playing around with the database (and that is where we are getting out of my comfort zone ).

Give the re-analyzing the base branch a try and let us know if that works for you

Cheers,

Dorian

Dennis_DECA · October 20, 2023, 8:39am

Hi @Dorian_Burihabwa ,
Thank you for the extra insights. I do agree messing directly with the DB is a no no.

We do a full scan of the main branch every night, so it should be up to date. Would it be better to perform a scan on every push to the main branch?

Our branching model:

develop (main/default)
master
release_1
…

As for the XML. It was not a pom.xml. It was a datasource xml. Each file contains a standalone SQL, not connected or referencing anything else in the code.
Would it be possible to list all impacted files (not used from cache) when running in debug mode? This would make it easier to investigate.

Lastly, the new code page has no impact on this issue?

Dorian_Burihabwa · November 29, 2023, 4:25pm

Hi @Dennis_DECA,

Sorry for the late reply.

Would it be better to perform a scan on every push to the main branch?

Running a regular analysis of your main branch would be good in general. Ideally, it would be re-analyzed every time something is merged, but a periodic cron job would already be a good start.

Lastly, the new code page has no impact on this issue?

It should not. You should be good to go if you have analyzed your reference branch at least once with this SonarQube version.

Would it be possible to list all impacted files (not used from cache) when running in debug mode?

There is a way you can do this, but you will have to log at a lower level than DEBUG; you will have to log at TRACE level. Beware, this is going to get really verbose, especially if your project is as large as described above.

For a Maven project, you will need to append both these options

-Dsonar.log.level= TRACE
-Dorg.slf4j.simpleLogger.defaultLogLevel=TRACE

For java files, we should be able to see cache misses by looking up the following lines

Could not find key <file key> in the cache

To clarify your project type, does it contain one module with over 17,000 files? That seems like a lot, and I am not sure we explicitly tested the cache against such a large module.

Let us know what you find in your exploration!

Dorian

Dennis_DECA · December 7, 2023, 8:12am

Hi @Dorian_Burihabwa ,
Thanks for your reply. I checked some things, but I couldn’t get it to work. I set the properties in the sonar scanner that we run via a github action. However I could never find any trace information in the log file. The Log file from the scanner.

      - name: SonarQube Scan
        if: env.RUN_SonarQube == 'true'
        uses: sonarsource/sonarqube-scan-action@master
        env:
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN_XXX }}
          SONAR_HOST_URL: ${{ vars.SONAR_HOST_URL_XXX }}
          SONAR_SCANNER_OPTS: -Xmx6500m
        with:
          projectBaseDir: core
          args: >
            -Dsonar.projectBaseDir=/github/workspace/core
            -Dsonar.working.directory=/github/workspace/core/.scannerwork
            -Dsonar.java.skipUnchanged=true
            -Dsonar.log.level=TRACE
            -Dorg.slf4j.simpleLogger.defaultLogLevel=TRACE`
            -Dsonar.verbose=true

I also checked with our developers: 17.000 files in a module seems normal. Currently there are no plans to split this in smaller modules.

Topic		Replies	Views
SonarScanner: Java Analyzer and caching. Do I do it right? SonarQube Server / Community Build	4	218	January 31, 2024
Sonar-scanner not finding cached files with gitlab SonarQube Server / Community Build sonarqube , scanner	5	1980	May 5, 2023
The Java analyzer cannot skip unchanged files in this context SonarQube Server / Community Build java , sonarqube , speedofanalysis	2	2322	June 21, 2023
C++ analysis cache SonarQube Cloud coverage	2	130	February 9, 2024
How to improve scan performance SonarQube Server / Community Build	10	1414	February 22, 2024

SonarScanner - Java analyzer cache incorrect?

Related topics