SonarQube UI occasionally crashes after update to 7.9.5

cantoress · March 2, 2021, 8:43am

Hello!
The issue started after upgrade SonarQube 6.7.7 → 7.9.5.

We use SonarQube dockerized server on our corporate AWS account on a separate EC2 instance.
We had the server 6.7.7 version and it works fine. After we upgraded the server to the version 7.9.5 the UI started to occasionally fail fetching requests. If you reload the page several times you can find a moment when all requests fetch successfully. But most of the time at least one request fail to fetch.
The same happens when we start the brand new instance with the 7.9.5 version.

I understand that it might be some corporate firewall issue, but the support can’t find the breaking point.
Were there any changes made with the UI correspondence to the backend with the 6.7.7 → 7.9.5 transition?
Thanks for any hints!

We use docker-compose file to start up the server. No changes were made during upgrade except the docker image tag.

version: "2"

services:
  sonarqube:
    image: sonarqube:7.9.5-community
    ports:
      - "80:9000"
    networks:
      - sonarnet
    environment:
      SONAR_JDBC_URL: jdbc:postgresql://db:5432/sonar
      SONAR_JDBC_USERNAME: sonar
      SONAR_JDBC_PASSWORD: sonar
    volumes:
      - sonarqube_conf:/opt/sonarqube/conf
      - sonarqube_data:/opt/sonarqube/data
      - sonarqube_extensions:/opt/sonarqube/extensions
      - sonarqube_bundled-plugins:/opt/sonarqube/lib/bundled-plugins
    ulimits:
      nofile:
        soft: 65536
        hard: 65536

  db:
    image: postgres:10.4
    networks:
      - sonarnet
    environment:
      - POSTGRES_USER=sonar
      - POSTGRES_PASSWORD=sonar
    volumes:
      - postgresql:/var/lib/postgresql
      # This needs explicit mapping due to https://github.com/docker-library/postgres/blob/4e48e3228a30763913ece952c611e5e9b95c8759/Dockerfile.template#L52
      - postgresql_data:/var/lib/postgresql/data

networks:
  sonarnet:
    driver: bridge
    driver_opts:
      com.docker.network.enable_ipv6: "false"
    ipam:
      driver: default
      config:
        -
          subnet: 172.21.219.0/24

volumes:
  sonarqube_conf:
  sonarqube_data:
  sonarqube_extensions:
  sonarqube_bundled-plugins:
  postgresql:
  postgresql_data:

Here is the screenshot of the UI page with failed requests:

The following error messages can be found in the Console:

This is the only error message that can be found in the SonarQube logs:

sonarqube_1  | 2021.03.02 07:35:46 DEBUG web[AXfx3UV13EnAm04wAAAD][o.s.s.w.WebServiceEngine] Request http://10.15.2.229/api/l10n/index?locale=en&ts=2021-03-02T06%3A57%3A01%2B0000 has been aborted by client
sonarqube_1  | org.sonar.api.utils.text.WriterException: Fail to write JSON
sonarqube_1  |  at org.sonar.api.utils.text.JsonWriter.rethrow(JsonWriter.java:396)
sonarqube_1  |  at org.sonar.api.utils.text.JsonWriter.close(JsonWriter.java:391)
sonarqube_1  |  at org.sonar.server.platform.ws.IndexAction.handle(IndexAction.java:84)
sonarqube_1  |  at org.sonar.server.ws.WebServiceEngine.execute(WebServiceEngine.java:110)
sonarqube_1  |  at org.sonar.server.ws.WebServiceFilter.doFilter(WebServiceFilter.java:88)
sonarqube_1  |  at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:126)
sonarqube_1  |  at org.sonar.server.platform.web.MasterServletFilter.doFilter(MasterServletFilter.java:95)
sonarqube_1  |  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
sonarqube_1  |  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
sonarqube_1  |  at org.sonar.server.user.UserSessionFilter.doFilter(UserSessionFilter.java:87)
sonarqube_1  |  at org.sonar.server.user.UserSessionFilter.doFilter(UserSessionFilter.java:71)
sonarqube_1  |  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
sonarqube_1  |  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
sonarqube_1  |  at org.sonar.server.platform.web.CacheControlFilter.doFilter(CacheControlFilter.java:76)
sonarqube_1  |  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
sonarqube_1  |  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
sonarqube_1  |  at org.sonar.server.platform.web.SecurityServletFilter.doHttpFilter(SecurityServletFilter.java:76)
sonarqube_1  |  at org.sonar.server.platform.web.SecurityServletFilter.doFilter(SecurityServletFilter.java:48)
sonarqube_1  |  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
sonarqube_1  |  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
sonarqube_1  |  at org.sonar.server.platform.web.RedirectFilter.doFilter(RedirectFilter.java:58)
sonarqube_1  |  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
sonarqube_1  |  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
sonarqube_1  |  at org.sonar.server.platform.web.requestid.RequestIdFilter.doFilter(RequestIdFilter.java:63)
sonarqube_1  |  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
sonarqube_1  |  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
sonarqube_1  |  at org.sonar.server.platform.web.RootFilter.doFilter(RootFilter.java:62)
sonarqube_1  |  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
sonarqube_1  |  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
sonarqube_1  |  at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
sonarqube_1  |  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
sonarqube_1  |  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
sonarqube_1  |  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
sonarqube_1  |  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
sonarqube_1  |  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
sonarqube_1  |  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
sonarqube_1  |  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
sonarqube_1  |  at ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:256)
sonarqube_1  |  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
sonarqube_1  |  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
sonarqube_1  |  at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
sonarqube_1  |  at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
sonarqube_1  |  at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
sonarqube_1  |  at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
sonarqube_1  |  at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
sonarqube_1  |  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
sonarqube_1  |  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
sonarqube_1  |  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
sonarqube_1  |  at java.base/java.lang.Thread.run(Unknown Source)
sonarqube_1  | Caused by: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Connection reset by peer
sonarqube_1  |  at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:364)
sonarqube_1  |  at org.apache.catalina.connector.OutputBuffer.appendByteArray(OutputBuffer.java:803)
sonarqube_1  |  at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:732)
sonarqube_1  |  at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:399)
sonarqube_1  |  at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:377)
sonarqube_1  |  at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)
sonarqube_1  |  at java.base/sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
sonarqube_1  |  at java.base/sun.nio.cs.StreamEncoder.implWrite(Unknown Source)
sonarqube_1  |  at java.base/sun.nio.cs.StreamEncoder.implWrite(Unknown Source)
sonarqube_1  |  at java.base/sun.nio.cs.StreamEncoder.write(Unknown Source)
sonarqube_1  |  at java.base/sun.nio.cs.StreamEncoder.write(Unknown Source)
sonarqube_1  |  at java.base/java.io.OutputStreamWriter.write(Unknown Source)
sonarqube_1  |  at java.base/java.io.Writer.write(Unknown Source)
sonarqube_1  |  at org.apache.commons.io.IOUtils.write(IOUtils.java:1986)
sonarqube_1  |  at org.sonar.server.ws.CacheWriter.close(CacheWriter.java:57)
sonarqube_1  |  at com.google.gson.stream.JsonWriter.close(JsonWriter.java:555)
sonarqube_1  |  at org.sonar.api.utils.text.JsonWriter.close(JsonWriter.java:389)
sonarqube_1  |  ... 47 common frames omitted
sonarqube_1  | Caused by: java.io.IOException: Connection reset by peer
sonarqube_1  |  at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
sonarqube_1  |  at java.base/sun.nio.ch.SocketDispatcher.write(Unknown Source)
sonarqube_1  |  at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
sonarqube_1  |  at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
sonarqube_1  |  at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
sonarqube_1  |  at java.base/sun.nio.ch.SocketChannelImpl.write(Unknown Source)
sonarqube_1  |  at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:134)
sonarqube_1  |  at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
sonarqube_1  |  at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:157)
sonarqube_1  |  at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1306)
sonarqube_1  |  at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:726)
sonarqube_1  |  at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:496)
sonarqube_1  |  at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:434)
sonarqube_1  |  at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:623)
sonarqube_1  |  at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:123)
sonarqube_1  |  at org.apache.coyote.http11.filters.GzipOutputFilter$FakeOutputStream.write(GzipOutputFilter.java:175)
sonarqube_1  |  at java.base/java.util.zip.DeflaterOutputStream.deflate(Unknown Source)
sonarqube_1  |  at java.base/java.util.zip.DeflaterOutputStream.write(Unknown Source)
sonarqube_1  |  at java.base/java.util.zip.GZIPOutputStream.write(Unknown Source)
sonarqube_1  |  at org.apache.coyote.http11.filters.GzipOutputFilter.doWrite(GzipOutputFilter.java:86)
sonarqube_1  |  at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:225)
sonarqube_1  |  at org.apache.coyote.Response.doWrite(Response.java:602)
sonarqube_1  |  at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:352)
sonarqube_1  |  ... 63 common frames omitted

ganncamp · March 2, 2021, 3:56pm

Hi,

It’s difficult to answer this since 7.9(.0) was released 1 July 2018, i.e. a very long time ago. And most of the work that went into the 6.7 → 7.9 transition is even older than that.

What I can tell you is that your guesses about corporate firewall look like the right direction to me, based on this:

And on the fact that it’s the only error you see in your logs. BTW, you might also check the server access logs & see if you can match up the errors you see client side with… anything in the access log. If there’s no log item for a client-side failure I think that’s a pretty good indicator something in the middle is intercepting & killing those requests.

HTH,
Ann

cantoress · March 3, 2021, 7:21am

Hello, Ann!

Thanks for your answer!
Your advice to check the access logs is really helpful! I can see all the requests that fail from the browser perspective to have no issues in the access logs, all requests has HTTP 200 and 304 statuses so it indeed seems like server is responding fine and then something is blocking server responses from reaching back the browser.

I will provide updated information to our support team, hope they will be able to find the solution soon.

cantoress · March 3, 2021, 7:27pm

Just an update for the future me or any other developers seeking for the solution (as it was driving me crazy):

After providing the support team the access logs file, exception from the server logs they finally found the blocker in the corporate firewall and, as they said, “opened a firewall change to allow sonarqube traffic at the application layer”, whatever it means.

Now SonarQube Ui opens fine and it wasn’t the SonarQube issue.

Thanks, @ganncamp , for the hint as it helped for the faster resolution!

system · March 10, 2021, 7:28pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Sonarqube Crashes after upgrade from 9.4 to 9.9 SonarQube Server / Community Build helm	11	1524	March 24, 2023
Sonarqube Fails after upgrade from 8.5 to 9.0 SonarQube Server / Community Build sonarqube	1	297	August 11, 2021
Upgrade Failure from SonarQube v10.5.1 to v10.7.1 SonarQube Server / Community Build sonarqube	5	27	December 12, 2024
Upgrade Sonarqube from 6.7.7 to 7.9.1 fails (Windows) SonarQube Server / Community Build	5	1631	October 3, 2019
SQ crashes after upgrade to v9 SonarQube Server / Community Build	12	3073	February 28, 2024

SonarQube UI occasionally crashes after update to 7.9.5

Related topics