SonarQube Enterprise instance fails to start following 8.4 upgrade

Sam_Anthonisz · July 6, 2020, 5:56am

Hi there,

I have just attempted upgrading our SonarQube Enterprise Edition test instance from 7.9.1 to the latest 8.4.0 release. Unfortunately it now wont start and the only error I can find in the log files is the following obscure message in web.log:

2020.07.06 15:00:36 ERROR web[][o.s.s.p.Platform] Background initialization failed. Stopping SonarQube
java.lang.IllegalArgumentException: Comparison method violates its general contract!
        at java.base/java.util.TimSort.mergeLo(TimSort.java:781)
        at java.base/java.util.TimSort.mergeAt(TimSort.java:518)
        at java.base/java.util.TimSort.mergeCollapse(TimSort.java:448)
        at java.base/java.util.TimSort.sort(TimSort.java:245)
        at java.base/java.util.Arrays.sort(Arrays.java:1515)
        at java.base/java.util.ArrayList.sort(ArrayList.java:1749)
        at org.sonar.server.issue.index.AsyncIssueIndexingImpl.sortProjectUuids(AsyncIssueIndexingImpl.java:104)
        at org.sonar.server.issue.index.AsyncIssueIndexingImpl.triggerOnIndexCreation(AsyncIssueIndexingImpl.java:81)
        at org.sonar.server.issue.index.IssueIndexer.triggerAsyncIndexOnStartup(IssueIndexer.java:102)
        at org.sonar.server.es.IndexerStartupTask.asynchronousIndexing(IndexerStartupTask.java:94)
        at org.sonar.server.es.IndexerStartupTask.indexUninitializedTypes(IndexerStartupTask.java:73)
        at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
        at org.sonar.server.es.IndexerStartupTask.execute(IndexerStartupTask.java:55)
        at java.base/java.util.Optional.ifPresent(Optional.java:183)
        at org.sonar.server.platform.platformlevel.PlatformLevelStartup$1.doPrivileged(PlatformLevelStartup.java:86)
        at org.sonar.server.user.DoPrivileged.execute(DoPrivileged.java:46)
        at org.sonar.server.platform.platformlevel.PlatformLevelStartup.start(PlatformLevelStartup.java:82)
        at org.sonar.server.platform.PlatformImpl.executeStartupTasks(PlatformImpl.java:198)
        at org.sonar.server.platform.PlatformImpl.access$400(PlatformImpl.java:46)
        at org.sonar.server.platform.PlatformImpl$1.lambda$doRun$1(PlatformImpl.java:122)
        at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.runIfNotAborted(PlatformImpl.java:370)
        at org.sonar.server.platform.PlatformImpl$1.doRun(PlatformImpl.java:122)
        at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.run(PlatformImpl.java:354)
        at java.base/java.lang.Thread.run(Thread.java:834)
2020.07.06 15:00:36 INFO  web[][o.s.p.ProcessEntryPoint] Hard stopping process
2020.07.06 15:38:52 INFO  web[][o.s.s.n.NotificationDaemon] Notification service stopped
2020.07.06 15:38:52 INFO  web[][o.s.s.e.CoreExtensionStopper] Stopping Governance
2020.07.06 15:38:52 INFO  web[][o.s.s.e.CoreExtensionStopper] Stopping Governance (done) | time=0ms
2020.07.06 15:38:52 WARN  web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [Thread-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:\n java.base@11.0.4/java.net.SocketInputStream.socketRead0(Native Method)\n java.base@11.0.4/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)\n java.base@11.0.4/java.net.SocketInputStream.read(SocketInputStream.java:168)\n java.base@11.0.4/java.net.SocketInputStream.read(SocketInputStream.java:140)\n java.base@11.0.4/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)\n java.base@11.0.4/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)\n java.base@11.0.4/java.io.BufferedInputStream.read(BufferedInputStream.java:351)\n java.naming@11.0.4/com.sun.jndi.ldap.Connection.run(Connection.java:832)\n java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Are you please able to provide any guidance on how to investigate/resolve this issue?

Thanks,

Sam

pierreguillot · July 6, 2020, 7:06am

Hi Sam, I’m investigating this. To be sure about your specific issue, could you please post here the result of this sql query please? You may have a lot of rows, so please upload the result as an attached file.

select component_uuid, created_at from snapshots

Sam_Anthonisz · July 6, 2020, 7:10am

Hi Pierre, thanks for the prompt reply. I’ve attached the requested output from the query. Not sure if it’s relevant but we’re using SQL Server.sq_diag.txt (297.3 KB)

pierreguillot · July 6, 2020, 9:29am

Thanks for your quick answer. I’m narrowing down the issue, it looks like this issue may come from projects or branches that does not have recent snapshot. Could you please run this query and tell us what you get? (this is counting projects+branches with not attached snapshot)

select count(pb.*) from project_branches pb
left join snapshots s on pb.uuid = s.component_uuid 
where s.component_uuid is null

Sam_Anthonisz · July 7, 2020, 2:03am

Sorry for the delayed response - we’re on very different time zones (I’m in ). On running that query I can see that there are 10 results. Out of interest I ran the same query on our production SonarQube instance (running 7.9.1) and there were 33 results. As that instance is accessible I decided to do some investigating and added the projects table to the query so I could see which projects were in the results. It then quickly became apparent that they fit one of two scenarios:

They were manually-created projects that had not yet been analysed at all
They were manually-created projects where the master branch had not yet been analysed

Hopefully this will provide useful!

pierreguillot · July 7, 2020, 3:09pm

Thanks for the details I reproduced and prepared a fix. This bug is tracked down here. This fix will be applied to master today and is likely to be backported to an 8.4.1 version later this week.

Sam_Anthonisz · July 7, 2020, 11:36pm

That’s good news Pierre, thanks again for the rapid response!

thewire247 · July 8, 2020, 8:48am

Hi. Thanks for fixing this so quickly!

We are experiencing very similar issues. I have attempted to build from master but am still getting the same error as above. Do you know if the fix made it into master yet?

pierreguillot · July 8, 2020, 12:40pm

Hi, the change should be on master by tomorrow. We internally use another git repo that is sync with the public one every night. So you’ll have to wait for the sync

thewire247 · July 9, 2020, 11:25am

Thanks! Built from master today and the issue has gone away. Will revert to the 8.4.1 docker image when released.

S0obi · July 13, 2020, 1:57pm

Hello Pierre. Any idea when 8.4.1 will be released?
Thanks in advance

pierreguillot · July 13, 2020, 3:20pm

Should happen this week

S0obi · July 14, 2020, 8:21pm

Thanks everyone for this new 8.4.1.35646 release. It’s working as excepted.

For people in the same (delicate) situation as I was : upgrading to 8.4.0.35506, facing https://jira.sonarsource.com/browse/SONAR-13590 then upgrade to 8.4.1.35646.
I got strange behavior on my projects (total showing 40 projects but only 10 displayed), I had to stop SonarQube and clean data/es6 folder in order to recreate the ES indexes. After this operation, SonarQube seems to behave as before.

PS : of course, we should do all a DB backup before any upgrade