7.9.1 Database upgrade fails

SonarQube Server / Community Build
1

Hello Community,
this is a fresh installation of sonar on macOS with homebrew.
Sonarqube version 7.9.1
PostgreSQL: 11.4

Yes, I have saearched forums and stack overflow and even this forum and tried very very many suggestins.

On the setup page I am asked to upgrade the database, but it fails with the error “Database connection cannot be established. Please check database status and JDBC settings.”
The user owns the database and I can use other programs to connect to it with this user. No issues there.
Could you please let me know where it is that I am missing it?

Many thanls,
Arthur

My sonar.properties has the following settings
sonar.jdbc.username=sonar
sonar.jdbc.password=sonar
sonar.jdbc.url=jdbc:postgresql://localhost:5432/sonar
sonar.web.javaOpts=-Xmx1024m -Xms1024m -XX:+HeapDumpOnOutOfMemoryError
sonar.ce.javaOpts=-Xmx1024m -Xms128m -XX:+HeapDumpOnOutOfMemoryError
sonar.search.javaOpts=-Xms1024m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError

2

Do you see anything relevant in the Postgres logs? Are there any other errors in Sonar logs?

On my linux non-production instances (with Postgres 9.x and 10.x) I’m skipping lots of issues by adding the below settings:

  • postgresql.conf
    • listen_addresses='*'
  • pg_hba.conf
    • host all all all md5

Of course, this may not be useful at all in your particular case.

3

Hi Arthur,

The setup page isn’t smart enough to not ask you to upgrade on a new instance; it’s static content, not dynamic.

I think the answer is going to be in your server logs, where you’ll probably find in the web.log that SonarQube isn’t connecting to your database.

 
Ann

4

Hi,
thanks for looking at this.
Yes I tried those settings as well as using trust instead of md5.
The connection still fails.
What I do have are ‘connection refused’ from elastic search. Am not sure what I have to do in that case.

5

Hi Ann,
thanks for looking into this.
Please look at these files. I could really appreciate some help in interpreting them.

cheers,
Arthuraccess.log.txt (1.5 KB) es.log.txt (545.7 KB) sonar.log.txt (20.9 KB) web.log.txt (30.8 KB)sonar.properties.txt (19.7 KB)

6

In your es.log I’ve seen this warning:

2019.08.23 00:48:05 WARN  es[][o.e.c.r.a.DiskThresholdMonitor] flood stage disk watermark [95%] exceeded on [I1tMMakFQYSWwmrZ3iaweQ][sonarqube][/usr/local/Cellar/sonarqube/7.9.1/libexec/data/es6/nodes/0] free: 86.7gb[4.6%], all indices on this node will be marked read-only

Could you try freeing some disk space? It might be easier than digging for the ES config file where this limit can be adjusted.

Eduard

1 Like
7

Hi,

Just to be specific, @edu has correctly identified the problem. Elasticsearch locks up if there’s not enough free disk space, and SonarQube won’t run with a locked ES. You need to provide free disk space (by whatever means) and start SonarQube back up. At startup, SQ unlocks any locked ES indices, so if there’s enough free space you should be good to go.

 
Ann

8

Have freed up space on the hard drive.
There is just below 1TB free.
In the es.log there are connection rfused mesages while in the server log there are connection established messages both to elastic search.
Here are the latest logs. Thanks for your help.access.log.txt (9.1 KB) disk_space.txt (155 Bytes) es.log.txt (555.0 KB) sonar.log.txt (22.1 KB) web.log.txt (32.8 KB)

9

Hi,
I was able to free up space on the disk. (Have posted the logs in the reply to @edu). Unfortunately, there is still a problem which I am missing.
The es.log reports an established connection while the sonar.log reports connection refused.
Thanks for your help.

10

Hi,

If ES truly has enough disk space, then you may have to simply delete the existing indices ($SONARQUBE_HOME/data/es6) and let ES rebuild them. If you still have the same problem after that, then you really are back to disk space.

 
Ann

11

@arthur_kahwa at this point I would simply fire up a VM with the latest ubuntu/debian/centos/whatever you prefer, install a fresh instance of sonar 7.9.1 to validate the environment, and only then I would try the upgrade.

12

@edu and @ganncamp thank you both for your help.
After installing on a centos vm it worked.
The problem was not the database but Elasticsearch and the filesystem settings I could not set on the mac.

Thanks once again.