Our Sonarqube application is deployed using docker and is connected to our Gitlab postgresql database. We’re guessing the massive report getting sent up to the server is getting bounced for being too big for the default message size allowance on the server. The report is about 2.7GB.
The changes made on database postgresql.conf file were:
shared_buffer parameter from 128MB to 256MB
uncommented the parameter work_mem 4MB
Changes made on sonar.properties (application config file):
Update the min and max for web and ce from their defaults to 4096
sonar.web.javaOpts=-Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError
sonar.ce.javaOpts=-Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError
We’ve pared this down pretty far actually, though the repo we’re drawing from is for a large project. The analysis file I most recently tried to send up was about 250MB, though compression probably brought that down even more. We still got the 500 error from the server.
It’s certainly still possible at 250 megabytes we’d still be tripping some cap on the server or db level for the file size, though it’s hard to say for sure that’s what’s happening. Like Lauren said, we made those above changes to the postgress db settings, and the sonar server’s config file, but that didn’t shake anything loose. I don’t have access to that ce.log file, but Lauren I believe you said earlier that it didn’t show much of anything at the time of the upload? If there’s anything else we could look at, Colin, in terms of logging, please let us know.
2024.07.09 18:50:07 INFO ce[][o.s.c.c.ComputeEngineContainerImpl] Running Developer edition
2024.07.09 18:50:07 INFO ce[][o.s.ce.app.CeServer] Compute Engine is started
2024.07.09 22:29:18 WARN ce[][c.z.h.pool.PoolBase] HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@176e4aae (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024.07.09 22:29:18 WARN ce[][c.z.h.pool.PoolBase] HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@254f90da (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024.07.09 22:29:18 WARN ce[][c.z.h.pool.PoolBase] HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@55e12e1e (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024.07.09 22:29:18 WARN ce[][c.z.h.pool.PoolBase] HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@303f2536 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024.07.09 22:29:18 WARN ce[][c.z.h.pool.PoolBase] HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@15ea62e7 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024.07.09 22:29:18 WARN ce[][c.z.h.pool.PoolBase] HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@4c7ae9c0 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2024.07.09 22:29:26 ERROR ce[][o.s.c.t.CeWorkerImpl] Failed to pop the queue of analysis reports
org.apache.ibatis.exceptions.PersistenceException:
### Error querying database. Cause: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 8001ms.
### The error may exist in org.sonar.db.property.InternalPropertiesMapper
### The error may involve org.sonar.db.property.InternalPropertiesMapper.selectAsText
### The error occurred while executing a query
### Cause: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 8001ms.
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:156)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:142)
at org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:147)
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:80)
at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:142)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86)
at jdk.proxy2/jdk.proxy2.$Proxy53.selectAsText(Unknown Source)
at org.sonar.db.property.InternalPropertiesDao.selectByKey(InternalPropertiesDao.java:179)
at org.sonar.ce.queue.CeQueueImpl.getWorkersPauseStatus(CeQueueImpl.java:350)
at org.sonar.ce.queue.InternalCeQueueImpl.peek(InternalCeQueueImpl.java:78)
at org.sonar.ce.taskprocessor.CeWorkerImpl.tryAndFindTaskToExecute(CeWorkerImpl.java:170)
at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:153)
at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
The upgrade from 10.3.x to the latest version 10.6.x was completed and we still get the ‘fail to insert data of CE task’. I don’t see the error in the ce.log but I see it in the web.log. We’re having a database administrator look at our database. Are there any configurations specifically that you know of we should be looking at in our postgresql.conf file or database in general. Thank you.
2024.07.18 19:23:13 ERROR web[35893586-83d7-49d1-80aa-afe5b5b6681d][o.s.s.w.WebServiceEngine] Fail to process request http://a2i2-sq.761link.net:443/api/ce/submit?projectKey=eot_mav_platform_r2_AY021p1jTVMTeanOUF7B
org.sonar.server.exceptions.ServerException: Fail to insert data of CE task e317dcaa-d711-45fe-88a9-7fd83beec000
at org.sonar.server.ce.ws.SubmitAction.handle(SubmitAction.java:114)
at org.sonar.server.ws.WebServiceEngine.execute(WebServiceEngine.java:114)
at org.sonar.server.platform.web.WebServiceFilter.doFilter(WebServiceFilter.java:84)
at org.sonar.server.platform.web.MasterServletFilter$JavaxFilterAdapter.doFilter(MasterServletFilter.java:227)
at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:198)
at org.sonar.server.platform.web.MasterServletFilter$HttpFilterChainAdapter.doFilter(MasterServletFilter.java:241)
at org.sonar.server.platform.web.SonarLintConnectionFilter.doFilter(SonarLintConnectionFilter.java:66)
at org.sonar.server.platform.web.MasterServletFilter$JavaxFilterAdapter.doFilter(MasterServletFilter.java:227)
at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:198)
at org.sonar.server.platform.web.MasterServletFilter.doFilter(MasterServletFilter.java:146)
at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:222)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:250)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:202)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:177)
at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:138)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:137)
at org.sonar.server.platform.web.UserSessionFilter.doFilter(UserSessionFilter.java:83)
at org.sonar.server.platform.web.UserSessionFilter.doFilter(UserSessionFilter.java:70)
at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:222)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:250)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:202)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:177)
at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:138)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:137)
at org.sonar.server.platform.web.UserSessionFilter.doFilter(UserSessionFilter.java:83)
at org.sonar.server.platform.web.UserSessionFilter.doFilter(UserSessionFilter.java:70)
at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:222)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:250)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:202)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:177)
at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:138)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:137)
at org.sonar.server.platform.web.CrossOriginFilter.doFilter(CrossOriginFilter.java:51)
at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:222)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:250)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:202)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:177)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:137)
at org.sonar.server.platform.web.SecurityServletFilter.doHttpFilter(SecurityServletFilter.java:60)
at org.sonar.server.platform.web.SecurityServletFilter.doFilter(SecurityServletFilter.java:47)
at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:222)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:250)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:202)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:177)
at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:138)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:137)
at org.sonar.server.platform.web.RedirectFilter.doFilter(RedirectFilter.java:56)
at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:222)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:250)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:202)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:177)
at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:138)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:137)
at org.sonar.server.platform.web.EndpointPathFilter.doFilter(EndpointPathFilter.java:47)
at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:222)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:250)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:202)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:177)
at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:138)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:137
at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:222)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:250)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:202)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:177)
at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:138)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:137)
at org.sonar.server.platform.web.RequestIdFilter.doFilter(RequestIdFilter.java:66)
at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:222)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:250)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:202)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:177)
at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:138)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:137)
at org.sonar.server.platform.web.RootFilter.doFilter(RootFilter.java:65)
at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:222)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:250)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:202)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:177)
at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:138)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:137)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
at ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:267)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1786)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
at java.base/java.lang.Thread.run(Unknown Source)