Could not fix elasticsearch indices error

SonarQube Server / Community Build
1

Must-share information (formatted with Markdown):

  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
    Enterprise Edition - Version 9.9.1

  • how is SonarQube deployed: zip, Docker, Helm
    zip

  • what are you trying to achieve
    Sonar scan is returning the error: Elasticsearch indices have become inconsistent. Consider re-indexing. Check documentation for more information [Troubleshooting](https://docs.sonarsource.com/sonarqube/latest/setup-and-upgrade/troubleshooting/)

  • what have you tried so far to achieve this
    Tried to clear all the indexes, still not working.

Do not share screenshots of logs – share the text itself (bonus points for being well-formatted)!
logs of sonar scan:

2023-09-27T04:07:06.3020395Z ##[group]Run sonar-scanner-4.8.0.2856-linux/bin/sonar-scanner -Dsonar.host.url=*** -Dsonar.login=*** \
2023-09-27T04:07:06.3021443Z e[36;1msonar-scanner-4.8.0.2856-linux/bin/sonar-scanner -Dsonar.host.url=*** -Dsonar.login=*** \e[0m
2023-09-27T04:07:06.3022271Z e[36;1m-Dsonar.projectKey=${SONARQUBE_PROJECT_KEY} -Dsonar.projectName=${SONARQUBE_PROJECT_KEY} \e[0m
2023-09-27T04:07:06.3023071Z e[36;1m-Dsonar.projectBaseDir=${DEFAULT_WORKSPACE} -Dsonar.sources=src \e[0m
2023-09-27T04:07:06.3023736Z e[36;1m-Dsonar.testExecutionReportPaths=coverage/sonar-report.xml -Dsonar.tests=src \e[0m
2023-09-27T04:07:06.3025243Z e[36;1m-Dsonar.exclusions='src/index.tsx, src/globals.ts, src/pact/*, src/utils/services/assetsHost.mock.js, src/setupTests.ts, src/**/*.pact.ts, src/utils/**/*.ts, src/utils/**/*.tsx, src/**/*.d.ts, src/utils/fileTransformer.js' \e[0m
2023-09-27T04:07:06.3026173Z e[36;1m-Dsonar.inclusions='**' -Dsonar.test.inclusions='src/**/*.spec.*, src/**/*.e2e.ts' \e[0m
2023-09-27T04:07:06.3026888Z e[36;1m-Dsonar.typescript.lcov.reportPaths='coverage/lcov.info' \e[0m
2023-09-27T04:07:06.3027598Z e[36;1m-Dsonar.qualitygate.wait='true'e[0m
2023-09-27T04:07:06.3079213Z shell: /usr/bin/bash -e {0}
2023-09-27T04:07:06.3079645Z env:
2023-09-27T04:07:06.3080028Z   NODE_TLS_REJECT_UNAUTHORIZED: 0
2023-09-27T04:07:06.3080568Z   SONARQUBE_PROJECT_KEY: sop-app-switcher
2023-09-27T04:07:06.3081031Z   DEFAULT_WORKSPACE: sop-app-switcher
2023-09-27T04:07:06.3081422Z ##[endgroup]
2023-09-27T04:07:06.5223067Z INFO: Scanner configuration file: /home/runner/actions-runner/_work/sop/sop/sonar-scanner-4.8.0.2856-linux/conf/sonar-scanner.properties
2023-09-27T04:07:06.5247930Z INFO: Project root configuration file: NONE
2023-09-27T04:07:06.5987945Z INFO: SonarScanner 4.8.0.2856
2023-09-27T04:07:06.5989178Z INFO: Java 11.0.17 Eclipse Adoptium (64-bit)
2023-09-27T04:07:06.5989922Z INFO: Linux 5.19.0-41-generic amd64
2023-09-27T04:07:07.1002027Z INFO: User cache: /home/runner/.sonar/cache
2023-09-27T04:07:08.8962198Z INFO: Analyzing on SonarQube server 9.9.1.69595
2023-09-27T04:07:08.8966038Z INFO: Default locale: "en", source code encoding: "UTF-8" (analysis is platform dependent)
2023-09-27T04:07:09.7134788Z INFO: Load global settings
2023-09-27T04:07:09.9112911Z INFO: Load global settings (done) | time=199ms
2023-09-27T04:07:09.9190001Z INFO: Server id: 1B7B86E8-AXXBNOB9aq8i2SLeGmQ-
2023-09-27T04:07:09.9275185Z INFO: User cache: /home/runner/.sonar/cache
2023-09-27T04:07:09.9309879Z INFO: Load/download plugins
2023-09-27T04:07:09.9310429Z INFO: Load plugins index
2023-09-27T04:07:10.0479278Z INFO: Load plugins index (done) | time=117ms
2023-09-27T04:07:15.3490764Z INFO: Load/download plugins (done) | time=5418ms
2023-09-27T04:07:16.0799654Z INFO: Loaded core extensions: developer-scanner
2023-09-27T04:07:17.0558797Z INFO: Process project properties
2023-09-27T04:07:17.0677169Z INFO: Process project properties (done) | time=12ms
2023-09-27T04:07:17.0719122Z INFO: Execute project builders
2023-09-27T04:07:17.0747838Z INFO: Execute project builders (done) | time=2ms
2023-09-27T04:07:17.0793907Z INFO: Project key: sop-app-switcher
2023-09-27T04:07:17.0795323Z INFO: Base dir: /home/runner/actions-runner/_work/sop/sop/sop-app-switcher
2023-09-27T04:07:17.0798821Z INFO: Working dir: /home/runner/actions-runner/_work/sop/sop/sop-app-switcher/.scannerwork
2023-09-27T04:07:17.0943504Z INFO: Load project settings for component key: 'sop-app-switcher'
2023-09-27T04:07:17.1465774Z INFO: Load project settings for component key: 'sop-app-switcher' (done) | time=52ms
2023-09-27T04:07:17.2589292Z INFO: Load project branches
2023-09-27T04:07:17.3269538Z INFO: Load project branches (done) | time=68ms
2023-09-27T04:07:17.3279531Z INFO: Load branch configuration
2023-09-27T04:07:17.3319415Z INFO: Detected branch/PR in 'GitHub Action'
2023-09-27T04:07:17.3321703Z INFO: Auto-configuring branch 'version_0.69.0'
2023-09-27T04:07:17.3336585Z INFO: Load branch configuration (done) | time=6ms
2023-09-27T04:07:17.4665373Z INFO: Auto-configuring with CI 'Github Actions'
2023-09-27T04:07:17.4746888Z INFO: Load quality profiles
2023-09-27T04:07:17.5809349Z INFO: Load quality profiles (done) | time=107ms
2023-09-27T04:07:17.5927001Z INFO: Load active rules
2023-09-27T04:07:20.4103823Z INFO: ------------------------------------------------------------------------
2023-09-27T04:07:20.4105552Z INFO: EXECUTION FAILURE
2023-09-27T04:07:20.4107823Z INFO: ------------------------------------------------------------------------
2023-09-27T04:07:20.4111220Z INFO: Total time: 13.891s
2023-09-27T04:07:20.4935213Z ERROR: Error during SonarScanner execution
2023-09-27T04:07:20.4937005Z INFO: Final Memory: 21M/77M
2023-09-27T04:07:20.4940144Z INFO: ------------------------------------------------------------------------
2023-09-27T04:07:20.4944079Z ERROR: Elasticsearch indices have become inconsistent. Consider re-indexing. Check documentation for more information https://docs.sonarqube.org/latest/setup/troubleshooting
2023-09-27T04:07:20.4946503Z ERROR: 
2023-09-27T04:07:20.4948094Z ERROR: Re-run SonarScanner using the -X switch to enable full debug logging.
2023-09-27T04:07:20.8392506Z ##[error]Process completed with exit code 2.
2

Hi,

Welcome to the community!

I’ll be frank, and say I have never seen that error in analysis before.

Your scanner is a version behind (5.0.1 is current), but that shouldn’t be a factor here.

When you say you tried to clear all the indexes, what did you do? Did you

  • shut SonarQube down
  • delete `$SONARQUBE-HOME/data/es7
  • restart SonarQube

?

Is there anything interesting in your server logs?

 
Ann

3

Hi Ann,

We have tried running these commands several times, it’s not working unfortunately :frowning:

I have attached the es log for your reference:
sonarqube_es.log (8.5 KB)

4

Hi,

What percentage of free disc space is available on the SonarQube host?

Those logs look normal to me except for the index.refresh_interval lines.

Anything interesting in your other server logs?

 
Ann

5

Hi,

The free disc space is: Disk Available 43.8 GB

6

Hi,

I don’t care about the total amount free, and neither does Elasticsearch. Elasticsearch looks at the percentage of free disc. That’s why I keep asking you

 
Ann

7

Hi,

It’s 97% free, only 3% is being used.

root@l8ng-prod1-xxx-sonar:/home/ubuntu# df -h
Filesystem                                 Size  Used Avail Use% Mounted on

/dev/mapper/dot--vg--sonar-dot--lv--sonar   44G  962M   41G   3% /var/lib/sonarqube
8

Hi,

Thanks for the percentage. Sorry I was short. I got you confused with another user whom I’ve asked 3 times for the percentage. (Still no excuse. I apologize. :frowning:)

I really have no idea where to go from here, so I’m going to refer this one to the experts.

 
Ann

11

Hello @kanglee_mbg,

Thanks for reaching out to us.

This error message is related to the ElasticSearch call we do when we try to get the active rules of a Quality Profile.

One way to figure out the Quality Profile that is the source of this issue is to disable some rules in the Quality Profile that your project uses. You can copy the existing Profile and deactivate randomly some/most rules. More information here.
After deactivating some rules, try to run the analysis again.