Sonarqube 9.9 loading always

Hi team,
We have upgraded to sonarqube 9.9 EE (from 9.4). After installing I see that my sonarqube service (OR) the $SQ_HOME/logs/web.log tells me to navigate to /setup But I see that the webpage keeps on displaying Loading…. I’m not sure why this is happening…
Can anyone tell me how to fix this?

Regards,
Shafeeq

This is what I see in logs

2023.05.09 17:24:44 DEBUG web[][c.z.h.p.HikariPool] HikariPool-1 - keepalive: connection org.postgresql.jdbc.PgConnection@2973c6f3 is alive
2023.05.09 17:24:53 DEBUG web[][c.z.h.p.HikariPool] HikariPool-1 - Before cleanup stats (total=10, active=0, idle=10, waiting=0)
2023.05.09 17:24:53 DEBUG web[][c.z.h.p.HikariPool] HikariPool-1 - After cleanup  stats (total=10, active=0, idle=10, waiting=0)
2023.05.09 17:24:53 DEBUG web[][c.z.h.p.HikariPool] HikariPool-1 - Fill pool skipped, pool has sufficient level or currently being filled (queueDepth=0).
2023.05.09 17:25:06 ERROR web[][o.s.s.p.w.RootFilter] Processing of request /favicon.ico failed
javax.servlet.ServletException: javax/security/auth/Subject$SecureSet$1
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:334)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:221)

This is my service file:-

$ cat /etc/systemd/system/sonarqube.service
[Unit]
Description=SonarQube service
Requires=network.target
After=network.target network-online.target

[Service]
Type=simple
User=sonarqube
Group=sonarqube
PermissionsStartOnly=true
ExecStart=/usr/bin/nohup /app/jdk/current/bin/java -Xms4G -Xmx4G -Djava.net.preferIPv4Stack=true -jar /app/sonarsource/install/sonarqube/lib/sonar-application-9.9.0.65466.jar
StandardOutput=syslog
#TimeoutStartSec=5
RestartSec=10
Restart=on-failure
SuccessExitStatus=143
LimitNOFILE=131072
LimitNPROC=8192

[Install]
WantedBy=multi-user.target

My service output

**$ systemctl status sonarqube.service**
● sonarqube.service - SonarQube service
   Loaded: loaded (/etc/systemd/system/sonarqube.service; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2023-05-09 17:09:14 UTC; 22min ago
 Main PID: 7554 (java)
    Tasks: 136 (limit: 4915)
   CGroup: /system.slice/sonarqube.service
           ├─7554 /app/jdk/current/bin/java -Xms4G -Xmx4G -Djava.net.preferIPv4Stack=true -jar /app/sonarsource/install/sonarqube/lib/sonar-application-9.9.0.65466.jar
           ├─7714 /app/jdk/jdk-17.0.7+7/bin/java -XX:+UseG1GC -Djava.io.tmpdir=/app/sonarsource/home/sonarqube/temp -XX:ErrorFile=/app/sonarsource/home/sonarqube/logs/es_hs_err_pid%p.log -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djna.tmpdir=/app/sonarsource/home/sonarqube/temp -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeyS
           └─9533 /app/jdk/jdk-17.0.7+7/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/app/sonarsource/home/sonarqube/temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.b

 nohup[7554]: OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
 nohup[7554]: OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
 nohup[7554]: OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
 nohup[7554]: WARNING: A terminally deprecated method in java.lang.System has been called
 nohup[7554]: WARNING: System::setSecurityManager has been called by org.sonar.process.PluginSecurityManager (file:/app/sonarsource/install/sonarqube-9.9.0.65466/lib/sonar-application-9.9.0.65466.jar)
 nohup[7554]: WARNING: Please consider reporting this to the maintainers of org.sonar.process.PluginSecurityManager
 nohup[7554]: WARNING: System::setSecurityManager will be removed in a future release
 nohup[7554]: 2023.05.09 17:09:55 WARN  app[][startup] ################################################################################
 nohup[7554]: 2023.05.09 17:09:55 WARN  app[][startup] The database must be manually upgraded. Please backup the database and browse /setup. For more information: https://docs.sonarqube.org/latest/setup/upgrading
 nohup[7554]: 2023.05.09 17:09:55 WARN  app[][startup] ################################################################################

Hi,

Did you read this last log? What did you make of the warning?

 
Ann

Hi Ann,

Can you elaborate your comment more? We usually upgrade db at ‘/setup’ page. But every page of sonarqube keeps showing LOADING…

Hi,

Even the /setup page?

 
Ann

Yes Ann.

Hi,

Anything unusual in the browser console?

Is there anything between you and SonarQube on the network? I.e. a proxy?

 
Ann

Just ran into this exact problem while trying to upgrade 9.6.1 to 9.9.1.69595. We are running SQ as a Docker container.

Tried access via browser and via curl from command line, same result.

javax.servlet.ServletException: javax/security/auth/Subject$SecureSet$1 logged which would indicate the problem is server-side?

Hi @edlevin6612,

Thanks for your additional report. Since multiple people are experiencing this, it can’t be just one user’s environment. I’ve flagged this for more expert eyes.

 
Ann

Thank you, just wanted to add I tried upgrading to one of the prior versions as a test, here are the results:

  • 9.8.0 → same issue
  • 9.7.1 → no issue

So it would appear the breaking change took place between 9.7.1 and 9.8, (maybe related to switch to Java 17?).

Also note this issue does not occur on a clean install, it manifests only when upgrading and it is needed to run /setup to upgrade the database

Hi @ganncamp and Hi @edlevin6612 ,

We do have an AWS ALB behind the EC2 instance (that hosts SonarQube server) which forwards traffic to target group on port 9000 of the same EC2 instance (As sonarqube runs on port 9000 obviously).

For me the issue got fixed by removing the Newrelic APM arguments which were configured as additional jvm params in our custom sonar.properties file. After removing them and restarting SonarQube, the UI came up.

However, at /setup page, where we upgrade the database, got failed due to the following error:-

Caused by: java.lang.IllegalArgumentException: Index name length can't be more than 30

To fix this issue, it was later suggested to proceed with two workarounds:-

  1. Remove the index directly from the database (we use AWS Postgresql DB for SonarQube)
    (OR)
  2. Upgrade SonarQube from 9.9 to 9.9.1 where the above mentioned error was fixed

And so I tried to upgrade to SonarQube 9.9.1 and the /setup page with successful DB migration happened this time and SonarQube came completely up and running.

This was still with my NewRelic arguments removed from our custom sonar.properties file BUT we really need this as we use APM systems like NewRelic to monitor the application. So, I again added those NewRelic args back and restarted SonarQube and it was successful this time. I don’t know how, still I’m wondering…

@shafemoh thanks for narrowing this down to Newrelic APM

We use Newrelic agent as well via -javaagent setting and removing it resolved the issue and allowed us to migrate the DB and SQ is functional.

For us, however, reintroducing Newrelic continues to break SQ (we are on 9.9.1 now). Here is an error example:

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'jdk.internal.loader.ClassLoaders$AppClassLoader@7382f612-org.sonar.server.platform.web.requestid.RequestIdGeneratorImpl': Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jdk.internal.loader.ClassLoaders$AppClassLoader@7382f612-org.sonar.server.platform.web.requestid.RequestIdGeneratorBaseImpl': Instantiation of bean failed; nested exception is java.lang.ClassCircularityError: javax/security/auth/Subject$SecureSet$1

We are using the latest New Relic agent 8.2.0 which supports Java 17.

@ganncamp not sure if this is a Sonarqube or Newrelic issue, I am guessing the former, we use the same NR agent for many other apps in a similar way with no issues. Definitely something that needs to be addressed since having functional APM is vital in production setting.

Thank you

2 Likes

Hi @edlevin6612 We are currently using newrelic java agent of version v7.7.0 downloaded from here https://download.newrelic.com/newrelic/java-agent/newrelic-agent/7.7.0/newrelic-java-7.7.0.zip

Just a side check, did you try to upgrade your Java version on the SonarQube server? We have upgraded to 17.0.7+7. see Prerequisites and overview

$ java -version
openjdk version "17.0.7" 2023-04-18
OpenJDK Runtime Environment Temurin-17.0.7+7 (build 17.0.7+7)
OpenJDK 64-Bit Server VM Temurin-17.0.7+7 (build 17.0.7+7, mixed mode, sharing)

Hi,

We’ve seen other people have problems with NewRelic. Here’s what we got 3rd-hand from NewRelic in another instance:

I think you may be running into a known issue. I’m still consulting with our Java development team to get the specifics, but there are strong parallels between your case and another case we’ve seen recently (specifically Java 17, SonarQube’s Java process, and the error you’re seeing in your stack trace).

Unfortunately, SonarQube is not something we currently test against, so we can’t explicitly guarantee that the agent will work in this environment. At present, we’re treating this as an unsupported environment, meaning my best path forward is to file a feature request on your behalf.

 
HTH,
Ann

We use the official SonarQube Docker image so we get Java 17 along with that, but thanks for pointing out.

2 Likes

Hi @edlevin6612

Even now I get the exact and same error as you got after adding back the newrelic settings:-

2023.05.15 18:35:42 ERROR web[][o.s.s.p.w.RootFilter] Processing of request / failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'jdk.internal.loader.ClassLoaders$AppClassLoader@251a69d7-org.sonar.server.platform.web.requestid.RequestIdGeneratorImpl': Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jdk.internal.loader.ClassLoaders$AppClassLoader@251a69d7-org.sonar.server.platform.web.requestid.RequestIdGeneratorBaseImpl': Instantiation of bean failed; nested exception is java.lang.ClassCircularityError: javax/security/auth/Subject$SecureSet$1

Yeah, I tried downgrading NR to 7.7.0 just as a test but no luck unfortunately.