Sonarqube Failing to generate report - ### Cause: java.sql.SQLException: Data source is closed

upen_raj · August 12, 2019, 7:56pm

Sonarqube version - 7.9.1
Plugins
checkstyle-sonar-plugin-4.20.jar
sonar-html-plugin-3.1.0.1615.jar
sonar-auth-github-plugin-1.5.0.870.jar
sonar-pmd-plugin-3.2.1.jar
sonar-scala-plugin-1.5.0.315.jar
sonar-typescript-plugin-1.9.0.3766.jar
sonar-xml-plugin-2.0.1.2020.jar
sonar-yaml-plugin-1.4.2.jar
sonar-json-plugin-2.3.jar
sonar-findbugs-plugin-3.10.0.jar

When I run an analysis, the report generation fails for projects with lot of code like 220k.

The last analysis has failed.

Sonarqube is running on Kubernetes with postgresDB.

I tried deleting the es directory under data by exec on the pod. Still no luck.

Error trace below

2019.08.12 19:43:31 INFO web[][o.a.c.h.Http11Processor] Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the HTTP protocol
    at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:541)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:684)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:834)
2019.08.12 19:43:32 WARN app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 137
2019.08.12 19:43:32 INFO app[][o.s.a.SchedulerImpl] Process[es] is stopped
2019.08.12 19:43:33 INFO ce[][o.s.p.ProcessEntryPoint] Hard stopping process
2019.08.12 19:43:33 INFO ce[][o.s.ce.app.CeServer] Compute Engine is stopping...
2019.08.12 19:43:33 INFO ce[][o.s.c.t.CeProcessingSchedulerImpl] Hard stopping workers...
2019.08.12 19:43:33 INFO ce[][o.s.c.t.CeProcessingSchedulerImpl] Some in-progress tasks are getting killed.
2019.08.12 19:43:34 WARN ce[][o.s.p.ProcessEntryPoint$HardStopperThread] Can not stop in 1000ms
2019.08.12 19:43:34 INFO ce[AWyHWq95f1aeYonL9sbw][o.s.c.t.s.ComputationStepExecutor] Persist sources | status=SUCCESS | time=11323ms
2019.08.12 19:43:43 WARN ce[][o.s.c.t.CeProcessingSchedulerExecutorServiceImpl] Pool CeProcessingSchedulerExecutorServiceImpl did not terminate
2019.08.12 19:43:43 INFO ce[][o.s.ce.app.CeServer] Compute Engine is stopped
2019.08.12 19:43:43 INFO ce[AWyHWq95f1aeYonL9sbw][o.s.c.t.s.ComputationStepExecutor] Persist cross project duplications | status=FAILED | time=1ms
2019.08.12 19:43:43 ERROR ce[AWyHWq95f1aeYonL9sbw][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Execution of task class org.sonar.ce.task.projectanalysis.webhook.WebhookPostTask failed
org.apache.ibatis.exceptions.PersistenceException:
### Error querying database. Cause: java.sql.SQLException: Data source is closed
### The error may exist in org.sonar.db.component.ComponentMapper
### The error may involve org.sonar.db.component.ComponentMapper.selectByUuid
### The error occurred while executing a query
### Cause: java.sql.SQLException: Data source is closed
    at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:149)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76)
    at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:87)
    at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:58)
    at com.sun.proxy.$Proxy26.selectByUuid(Unknown Source)
    at org.sonar.db.component.ComponentDao.selectByUuid(ComponentDao.java:93)
    at org.sonar.server.webhook.WebHooksImpl.readWebHooksFrom(WebHooksImpl.java:68)
    at org.sonar.server.webhook.WebHooksImpl.sendProjectAnalysisUpdate(WebHooksImpl.java:93)
    at org.sonar.ce.task.projectanalysis.webhook.WebhookPostTask.finished(WebhookPostTask.java:45)
    at org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor.executeTask(PostProjectAnalysisTasksExecutor.java:113)
    at org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor.finished(PostProjectAnalysisTasksExecutor.java:107)
    at org.sonar.ce.task.step.ComputationStepExecutor.executeListener(ComputationStepExecutor.java:91)
    at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:63)
    at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
    at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:209)
    at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:191)
    at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:158)
    at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:133)
    at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:85)
    at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.sql.SQLException: Data source is closed
    at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:2087)
    at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
    at org.sonar.db.profiling.NullConnectionInterceptor.getConnection(NullConnectionInterceptor.java:31)
    at org.sonar.db.profiling.ProfiledDataSource.getConnection(ProfiledDataSource.java:317)
    at org.apache.ibatis.transaction.jdbc.JdbcTransaction.openConnection(JdbcTransaction.java:138)
    at org.apache.ibatis.transaction.jdbc.JdbcTransaction.getConnection(JdbcTransaction.java:60)
    at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:336)
    at org.apache.ibatis.executor.ReuseExecutor.prepareStatement(ReuseExecutor.java:88)
    at org.apache.ibatis.executor.ReuseExecutor.doQuery(ReuseExecutor.java:59)
    at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324)
    at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
    at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
    at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:83)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
    ... 27 common frames omitted
2019.08.12 19:43:44 ERROR ce[AWyHWq95f1aeYonL9sbw][o.s.c.t.CeWorkerImpl] Failed to finalize task with uuid 'AWyHWq95f1aeYonL9sbw' and persist its state to db
org.apache.ibatis.exceptions.PersistenceException:
### Error querying database. Cause: java.sql.SQLException: Data source is closed
### The error may exist in org.sonar.db.ce.CeQueueMapper
### The error may involve org.sonar.db.ce.CeQueueMapper.selectByUuid
### The error occurred while executing a query
### Cause: java.sql.SQLException: Data source is closed
    at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:149)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76)
    at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:87)
    at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:58)
    at com.sun.proxy.$Proxy24.selectByUuid(Unknown Source)
    at org.sonar.db.ce.CeQueueDao.selectByUuid(CeQueueDao.java:85)
    at org.sonar.ce.queue.InternalCeQueueImpl.remove(InternalCeQueueImpl.java:117)
    at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.finalizeTask(CeWorkerImpl.java:242)
    at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.afterExecute(CeWorkerImpl.java:236)
    at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.close(CeWorkerImpl.java:196)
    at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:159)
    at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:133)
    at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:85)
    at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.sql.SQLException: Data source is closed
    at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:2087)
    at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
    at org.sonar.db.profiling.NullConnectionInterceptor.getConnection(NullConnectionInterceptor.java:31)
    at org.sonar.db.profiling.ProfiledDataSource.getConnection(ProfiledDataSource.java:317)
    at org.apache.ibatis.transaction.jdbc.JdbcTransaction.openConnection(JdbcTransaction.java:138)
    at org.apache.ibatis.transaction.jdbc.JdbcTransaction.getConnection(JdbcTransaction.java:60)
    at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:336)
    at org.apache.ibatis.executor.ReuseExecutor.prepareStatement(ReuseExecutor.java:88)
    at org.apache.ibatis.executor.ReuseExecutor.doQuery(ReuseExecutor.java:59)
    at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324)
    at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
    at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
    at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:83)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
    ... 21 common frames omitted
2019.08.12 19:43:44 INFO ce[AWyHWq95f1aeYonL9sbw][o.s.c.t.CeWorkerImpl] Executed task | project=CRM-UI:CRM-UI | type=REPORT | id=AWyHWq95f1aeYonL9sbw | status=CANCELED | time=59037ms
2019.08.12 19:43:44 ERROR ce[][o.s.c.t.CeWorkerImpl] Call to listener ReportAnalysisFailureNotificationExecutionListener failed.
org.apache.ibatis.exceptions.PersistenceException:
### Error getting a new connection. Cause: java.sql.SQLException: Data source is closed
### Cause: java.sql.SQLException: Data source is closed
    at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.getConnection(DefaultSqlSession.java:299)
    at org.sonar.db.DbSessionImpl.getConnection(DbSessionImpl.java:192)
    at org.sonar.db.property.PropertiesDao.hasProjectNotificationSubscribersForDispatchers(PropertiesDao.java:101)
    at org.sonar.server.notification.NotificationService.hasProjectSubscribersForTypes(NotificationService.java:166)
    at org.sonar.ce.notification.ReportAnalysisFailureNotificationExecutionListener.onEnd(ReportAnalysisFailureNotificationExecutionListener.java:74)
    at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.lambda$finalizeTask$1(CeWorkerImpl.java:252)
    at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.lambda$callListeners$2(CeWorkerImpl.java:259)
    at java.base/java.util.Arrays$ArrayList.forEach(Arrays.java:4390)
    at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.callListeners(CeWorkerImpl.java:257)
    at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.finalizeTask(CeWorkerImpl.java:252)
    at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.afterExecute(CeWorkerImpl.java:236)
    at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.close(CeWorkerImpl.java:196)
    at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:159)
    at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:133)
    at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:85)
    at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.sql.SQLException: Data source is closed
    at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:2087)
    at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
    at org.sonar.db.profiling.NullConnectionInterceptor.getConnection(NullConnectionInterceptor.java:31)
    at org.sonar.db.profiling.ProfiledDataSource.getConnection(ProfiledDataSource.java:317)
    at org.apache.ibatis.transaction.jdbc.JdbcTransaction.openConnection(JdbcTransaction.java:138)
    at org.apache.ibatis.transaction.jdbc.JdbcTransaction.getConnection(JdbcTransaction.java:60)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.getConnection(DefaultSqlSession.java:297)
    ... 22 common frames omitted
2019.08.12 19:43:44 INFO app[][o.s.a.SchedulerImpl] Process[ce] is stopped
2019.08.12 19:43:44 INFO web[][o.s.p.ProcessEntryPoint] Hard stopping process
2019.08.12 19:43:44 INFO web[][o.a.c.c.StandardWrapper] Waiting for [1] instance(s) to be deallocated for Servlet [default]
2019.08.12 19:43:44 INFO web[][o.s.s.n.NotificationDaemon] Notification service stopped
2019.08.12 19:43:44 INFO app[][o.s.a.SchedulerImpl] Process[web] is stopped
2019.08.12 19:43:44 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped

Caused by: java.lang.IllegalStateException: Fail to execute ES refresh request on indices 'components'
    at org.sonar.server.es.request.ProxyRefreshRequestBuilder.get(ProxyRefreshRequestBuilder.java:44)
    at org.sonar.server.es.request.ProxyRefreshRequestBuilder.get(ProxyRefreshRequestBuilder.java:32)
    at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:120)
    at org.sonar.server.component.index.ComponentIndexer.delete(ComponentIndexer.java:165)
    at org.sonar.ce.task.projectanalysis.purge.IndexPurgeListener.onComponentsDisabling(IndexPurgeListener.java:41)
    at org.sonar.db.purge.PurgeDao.purgeDisabledComponents(PurgeDao.java:107)
    at org.sonar.db.purge.PurgeDao.purge(PurgeDao.java:71)
    at org.sonar.ce.task.projectanalysis.purge.ProjectCleaner.purge(ProjectCleaner.java:63)
    at org.sonar.ce.task.projectanalysis.purge.PurgeDatastoresStep.execute(PurgeDatastoresStep.java:76)
    at org.sonar.ce.task.projectanalysis.purge.PurgeDatastoresStep.access$000(PurgeDatastoresStep.java:38)
    at org.sonar.ce.task.projectanalysis.purge.PurgeDatastoresStep$1.visitProject(PurgeDatastoresStep.java:63)
    at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitNode(DepthTraversalTypeAwareCrawler.java:70)
    at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:51)
    at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39)
... 18 common frames omitted
Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes were available: [{sonarqube}{4GDUAUFBRcKiHD8xbvE41A}{kyl3SJOTRfmvd5TYYecBPQ}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}]
    at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:294)
    at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59)
    at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:533)
    at org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:48)
    at org.elasticsearch.client.transport.TransportProxyClient.lambda$execute$0(TransportProxyClient.java:60)
    at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:253)
    at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60)
    at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:388)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:391)
    at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1262)
    at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46)
    at org.sonar.server.es.request.ProxyRefreshRequestBuilder.get(ProxyRefreshRequestBuilder.java:42)
... 31 common frames omitted
Caused by: org.elasticsearch.transport.NodeNotConnectedException: [sonarqube][127.0.0.1:9001] Node not connected
    at org.elasticsearch.transport.ConnectionManager.getConnection(ConnectionManager.java:151)
    at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:557)
    at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:529)

ganncamp · August 13, 2019, 3:14pm

Hi,

I’ve code-formatted your logs (``` on the line before and on the line after), which makes them far, far easier to read, thus making it easier for others to help you. If you could handle that in the future it would be great.

I think the source of your problem is this:

The proximity of these two things to each other in the logs makes me suspect you have a bad webhook configured. This could be at either the global or the project level. Could you please investigate this? I would be very interested to know if you do.

Thx,
Ann

upen_raj · August 13, 2019, 5:20pm

Hi Ann,

Thanks for the edits. I will take care of these in the future. This issue is popping up only while running analysis for a project with 230K lines of code. Also the http error i suppose is coming from the analysis using sonar.host.url=http instead of https. The analysis goes fine for every other project.

Regards,
Upen

ganncamp · August 13, 2019, 7:02pm

Hi Upen,

Could you verify that it’s the URL that’s the problem by removing it and running an analysis? Also, could you verify that the URL contains no invisible characters? And if it doesn’t and if your company policies allow could you send me the URL privately, please? I’m at ann dot campbell at sonarsource.com.

Thx!
Ann

upen_raj · August 13, 2019, 9:07pm

Hi Ann,

I tried with the https url. The analysis completes certain times without any error. Few of the instances, it fails during post analysis report processing phase. On the dashboard, we wont be able to drill down inside the report to see the class files.

url - https://sonarapp.corp.*****.com - this is a vip redirecting https to http port 9000. Sonar is http. VIP is doing the redirect.

ganncamp · August 14, 2019, 12:05pm

Hi Upen,

I’m confused by this:

You’re saying you set the webhook up to point to the same URL with an https protocol instead of http and that the https version simply redirects to the original http URL? And that change from http to https makes it succeed some of the time, whereas before it never did?

You’re also saying that you’ve got SonarQube exposed on an http address (but not the one the webhook’s redirect points to, right?). It shouldn’t really matter what SonarQube’s URL is here.

What exactly is the webhook pointing to, and how does it respond (including the http status code) to the webhook?

Thx,
Ann

upen_raj · August 14, 2019, 5:22pm

Hi Ann,

No custom setup has been done on the docker image or kubernetes. Sonar is running on a load balanced ingress on kubernetes which is port 9000. Internally we have a load balancer vip which is https. Users use the https vip from the load balancer at port 443(https) and redirects the traffic to port 9000 on the kubernetes ingress. The setup works for most of the projects, except for the ones with huge code base, the webhook step fails.

Regards.
Upen

ganncamp · August 15, 2019, 12:06pm

Hi Upen,

I’m not sure how the fact that you’re running on Docker is related. Could you tell me what exactly is the webhook pointing to, and how does it (the thing the webhook points to) respond, including the http status code, to the webhook?

Thx,
Ann

upen_raj · August 15, 2019, 10:12pm

Hi Ann,

I have not created any custom webook here - https://sonar.corp.****.com/admin/webhooks. Everything is default.

https://sonar.corp.****.com/api/ce/task?id=AWyXVS2kKiS-NwwpMAhu

 [INFO] Sensor JaCoCo XML Report Importer [jacoco]
 [INFO] Sensor JaCoCo XML Report Importer [jacoco] (done) | time=5ms
 [INFO] Sensor SonarJS [javascript]
 [INFO] 3 source files to be analyzed
 [INFO] Sensor SonarJS [javascript] (done) | time=8151ms
 [INFO] 3/3 source files have been analyzed
 [INFO] Sensor ESLint-based SonarJS [javascript]
 [INFO] 3 source files to be analyzed
 [INFO] Sensor ESLint-based SonarJS [javascript] (done) | time=3357ms
 [INFO] 3/3 source files have been analyzed
 [INFO] ------------- Run sensors on project
 [INFO] Sensor Zero Coverage Sensor
 [INFO] Sensor Zero Coverage Sensor (done) | time=2ms
 [INFO] Calculating CPD for 5 files
 [INFO] CPD calculation finished
 [INFO] Analysis report generated in 144ms, dir size=2 MB
 [INFO] Analysis report compressed in 326ms, zip size=655 KB
 [INFO] Analysis report uploaded in 1067ms
 [INFO] ANALYSIS SUCCESSFUL, you can browse https:/sonar.corp.****.com/dashboard?id=com.gbi.deploy%3ACIDashboard
 [INFO] Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
 [INFO] More about the report processing at https://sonar.corp.****.com/api/ce/task?id=AWyXVS2kKiS-NwwpMAhu
 [INFO] Analysis total time: 31.770 s
 [INFO] ------------------------------------------------------------------------
 [INFO] BUILD SUCCESS
 [INFO] ------------------------------------------------------------------------
 [INFO] Total time: 36.644 s
 [INFO] Finished at: 2019-08-15T15:10:38-07:00
 [INFO] Final Memory: 46M/1336M
 [INFO] ------------------------------------------------------------------------

From logs

 2019.08.15 22:10:38 INFO  ce[][o.s.c.t.CeWorkerImpl] Execute task | project=com.gbi.deploy:CIDashboard | type=REPORT | id=AWyXVS2kKiS-NwwpMAhu
 2019.08.15 22:10:39 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Extract report | status=SUCCESS | time=188ms
 2019.08.15 22:10:39 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Persist scanner context | status=SUCCESS | time=9ms
 2019.08.15 22:10:39 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Propagate analysis warnings from scanner report | status=SUCCESS | time=0ms
 2019.08.15 22:10:39 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Execute DB migrations for current project | status=SUCCESS | time=6ms
 2019.08.15 22:10:39 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Generate analysis UUID | status=SUCCESS | time=0ms
 2019.08.15 22:10:39 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Load analysis metadata | status=SUCCESS | time=104ms
 2019.08.15 22:10:39 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Initialize | status=SUCCESS | time=0ms
 2019.08.15 22:10:39 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Verify billing | status=SUCCESS | time=0ms
 2019.08.15 22:10:39 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Build tree of components | components=72 | status=SUCCESS | time=70ms
 2019.08.15 22:10:39 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Validate project | status=SUCCESS | time=4ms
 2019.08.15 22:10:39 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Load quality profiles | status=SUCCESS | time=601ms
 2019.08.15 22:10:40 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Load Quality gate | status=SUCCESS | time=17ms
 2019.08.15 22:10:40 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Load new code period | status=SUCCESS | time=14ms
 2019.08.15 22:10:40 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Detect file moves | reportFiles=60 | dbFiles=60 | addedFiles=0 | status=SUCCESS | time=12ms
 2019.08.15 22:10:40 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Load duplications | duplications=14 | status=SUCCESS | time=10ms
 2019.08.15 22:10:40 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Compute cross project duplications | status=SUCCESS | time=0ms
 2019.08.15 22:10:40 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Compute size measures | status=SUCCESS | time=40ms
 2019.08.15 22:10:40 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Compute new coverage | status=SUCCESS | time=382ms
 2019.08.15 22:10:40 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Compute coverage measures | status=SUCCESS | time=4ms
 2019.08.15 22:10:40 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Compute comment measures | status=SUCCESS | time=3ms
 2019.08.15 22:10:40 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Copy custom measures | status=SUCCESS | time=4ms
 2019.08.15 22:10:40 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Compute duplication measures | status=SUCCESS | time=3ms
 2019.08.15 22:10:40 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Compute size measures on new code | status=SUCCESS | time=10ms
 2019.08.15 22:10:40 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Compute language distribution | status=SUCCESS | time=15ms
 2019.08.15 22:10:40 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Compute test measures | status=SUCCESS | time=51ms
 2019.08.15 22:10:40 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Compute complexity measures | status=SUCCESS | time=3ms
 2019.08.15 22:10:40 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Load measure computers | status=SUCCESS | time=3ms
 2019.08.15 22:10:40 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Compute Quality Profile status | status=SUCCESS | time=26ms
 2019.08.15 22:10:41 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Execute component visitors | status=SUCCESS | time=1085ms
 2019.08.15 22:10:41 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Checks executed after computation of measures | status=SUCCESS | time=1ms
 2019.08.15 22:10:41 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Compute Quality Gate measures | status=SUCCESS | time=10ms
 2019.08.15 22:10:41 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Compute Quality profile measures | status=SUCCESS | time=3ms
 2019.08.15 22:10:41 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Generate Quality profile events | status=SUCCESS | time=5ms
 2019.08.15 22:10:41 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Generate Quality gate events | status=SUCCESS | time=3ms
 2019.08.15 22:10:41 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Persist components | status=SUCCESS | time=31ms
 2019.08.15 22:10:41 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Persist analysis | status=SUCCESS | time=5ms
 2019.08.15 22:10:41 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Persist analysis properties | status=SUCCESS | time=1ms
 2019.08.15 22:10:41 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Persist measures | inserts=70 | status=SUCCESS | time=20ms
 2019.08.15 22:10:42 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Persist live measures | insertsOrUpdates=2058 | status=SUCCESS | time=650ms
 2019.08.15 22:10:42 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Persist duplication data | insertsOrUpdates=0 | status=SUCCESS | time=12ms
 2019.08.15 22:10:42 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Persist new ad hoc Rules | status=SUCCESS | time=1ms
 2019.08.15 22:10:43 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Persist issues | inserts=0 | updates=14 | merged=0 | untouched=1257 | status=SUCCESS | time=1120ms
 2019.08.15 22:10:43 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Persist project links | status=SUCCESS | time=5ms
 2019.08.15 22:10:43 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Persist events | status=SUCCESS | time=5ms
 2019.08.15 22:10:44 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Persist sources | status=SUCCESS | time=552ms
 2019.08.15 22:10:44 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Persist cross project duplications | status=SUCCESS | time=0ms
 2019.08.15 22:10:44 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Enable analysis | status=SUCCESS | time=4ms
 2019.08.15 22:10:44 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Update last usage date of quality profiles | status=SUCCESS | time=7ms
 2019.08.15 22:10:44 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Purge db | status=SUCCESS | time=82ms
 2019.08.15 22:10:45 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Index analysis | status=SUCCESS | time=1004ms
 2019.08.15 22:10:45 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Send issue notifications | newIssuesNotifs=0 | newIssuesDeliveries=0 | myNewIssuesNotifs=0 | myNewIssuesDeliveries=0 | changesNotifs=0 | changesDeliveries=0 | status=SUCCESS | time=4ms
 2019.08.15 22:10:45 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Publish task results | status=SUCCESS | time=0ms
 2019.08.15 22:10:45 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.s.ComputationStepExecutor] Trigger refresh of Portfolios and Applications | status=SUCCESS | time=0ms
 2019.08.15 22:10:45 INFO  ce[AWyXVS2kKiS-NwwpMAhu][o.s.c.t.CeWorkerImpl] Executed task | project=com.gbi.deploy:CIDashboard | type=REPORT | id=AWyXVS2kKiS-NwwpMAhu | status=SUCCESS | time=6696ms

ganncamp · August 16, 2019, 2:17pm

Hi,

Let’s back up. Your initial post starts with this error:

upen_raj:

2019.08.12 19:43:31 INFO web[][o.a.c.h.Http11Processor] Error parsing HTTP request header 
Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level. 
java.lang.IllegalArgumentException: Invalid character found in the HTTP protocol
    at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:541)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:684)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
...

And then we get the server shutting down:

upen_raj:

2019.08.12 19:43:32 WARN app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 137 
2019.08.12 19:43:32 INFO app[][o.s.a.SchedulerImpl] Process[es] is stopped 
2019.08.12 19:43:33 INFO ce[][o.s.p.ProcessEntryPoint] Hard stopping process 2019.08.12 19:43:33 INFO ce[][o.s.ce.app.CeServer] Compute Engine is stopping... 
2019.08.12 19:43:33 INFO ce[][o.s.c.t.CeProcessingSchedulerImpl] Hard stopping workers... 
2019.08.12 19:43:33 INFO ce[][o.s.c.t.CeProcessingSchedulerImpl] Some in-progress tasks are getting killed.

I believe the DB and ES errors are just side effects of this. I’ve been focusing on this

but in hindsight, it’s probably a side-effect too. Especially since you say

and there are no default webhooks.

…

So what we appear to have is SonarQube (the underlying Tomcat web server) receiving an unparsable request and shutting down.

You’ve said in responses that

And

So I think you need to look to how your load balancer is handling the submission of the analysis reports from the very large projects.

HTH,
Ann

upen_raj · August 22, 2019, 11:36pm

Hi Ann,

I tried increasing the loadbalancer request size etc. Still the same issue. Also sonar is running on http port 9000. Loadbalancer on https 443.

Error Details

 java.lang.IllegalStateException: Fail to execute ES refresh request on indices 'issues'
 	at org.sonar.server.es.request.ProxyRefreshRequestBuilder.get(ProxyRefreshRequestBuilder.java:44)
 	at org.sonar.server.es.request.ProxyRefreshRequestBuilder.get(ProxyRefreshRequestBuilder.java:32)
 	at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:120)
 	at org.sonar.server.issue.index.IssueIndexer.doIndex(IssueIndexer.java:249)
 	at org.sonar.server.issue.index.IssueIndexer.indexOnAnalysis(IssueIndexer.java:103)
 	at org.sonar.ce.task.projectanalysis.step.IndexAnalysisStep.execute(IndexAnalysisStep.java:45)
 	at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:81)
 	at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:72)
 	at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:59)
 	at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
 	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:209)
 	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:191)
 	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:158)
 	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:133)
 	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:85)
 	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
 	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 	at java.base/java.lang.Thread.run(Thread.java:834)
 Caused by: NoNodeAvailableException[None of the configured nodes were available: [{sonarqube}{4TqcgEgITh2ctFgD6VFDvA}{5Rb1cS1BQ6uKFpl0TIGSbg}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}]]; nested: NodeDisconnectedException[[sonarqube][127.0.0.1:9001][indices:admin/refresh] disconnected];
 	at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:294)
 	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59)
 	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1114)
 	at org.elasticsearch.transport.TransportService.lambda$onConnectionClosed$8(TransportService.java:963)
 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
 	... 3 more
 Caused by: NodeDisconnectedException[[sonarqube][127.0.0.1:9001][indices:admin/refresh] disconnected]

upen_raj · August 23, 2019, 12:00am

Hi Ann,

Please see the complete log from analysis start until end. Part (1)

 2019.08.22 23:49:21 INFO  ce[][o.s.c.t.CeWorkerImpl] Execute task | project=CRM-UI:CRM-UI | type=REPORT | id=AWy7vBKcCRIR65Zz25ib
 2019.08.22 23:49:23 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Extract report | status=SUCCESS | time=1499ms
 2019.08.22 23:49:23 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Persist scanner context | status=SUCCESS | time=4ms
 2019.08.22 23:49:23 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Propagate analysis warnings from scanner report | status=SUCCESS | time=0ms
 2019.08.22 23:49:23 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Execute DB migrations for current project | status=SUCCESS | time=2ms
 2019.08.22 23:49:23 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Generate analysis UUID | status=SUCCESS | time=0ms
 2019.08.22 23:49:23 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Load analysis metadata | status=SUCCESS | time=10ms
 2019.08.22 23:49:23 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Initialize | status=SUCCESS | time=0ms
 2019.08.22 23:49:23 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Verify billing | status=SUCCESS | time=0ms
 2019.08.22 23:49:23 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Build tree of components | components=1666 | status=SUCCESS | time=145ms
 2019.08.22 23:49:23 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Validate project | status=SUCCESS | time=2ms
 2019.08.22 23:49:23 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Load quality profiles | status=SUCCESS | time=293ms
 2019.08.22 23:49:23 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Load Quality gate | status=SUCCESS | time=5ms
 2019.08.22 23:49:23 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Load new code period | status=SUCCESS | time=0ms
 2019.08.22 23:49:23 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Detect file moves | status=SUCCESS | time=0ms
 2019.08.22 23:49:24 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Load duplications | duplications=4556 | status=SUCCESS | time=222ms
 2019.08.22 23:49:24 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Compute cross project duplications | status=SUCCESS | time=0ms
 2019.08.22 23:49:24 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Compute size measures | status=SUCCESS | time=179ms
 2019.08.22 23:49:24 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Compute new coverage | status=SUCCESS | time=18ms
 2019.08.22 23:49:24 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Compute coverage measures | status=SUCCESS | time=25ms
 2019.08.22 23:49:24 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Compute comment measures | status=SUCCESS | time=13ms
 2019.08.22 23:49:24 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Copy custom measures | status=SUCCESS | time=2ms
 2019.08.22 23:49:24 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Compute duplication measures | status=SUCCESS | time=101ms
 2019.08.22 23:49:24 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Compute size measures on new code | status=SUCCESS | time=5ms
 2019.08.22 23:49:24 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Compute language distribution | status=SUCCESS | time=58ms
 2019.08.22 23:49:24 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Compute test measures | status=SUCCESS | time=13ms
 2019.08.22 23:49:24 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Compute complexity measures | status=SUCCESS | time=26ms
 2019.08.22 23:49:24 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Load measure computers | status=SUCCESS | time=0ms
 2019.08.22 23:49:24 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Compute Quality Profile status | status=SUCCESS | time=3ms
 2019.08.22 23:49:34 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Execute component visitors | status=SUCCESS | time=10122ms
 2019.08.22 23:49:34 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Checks executed after computation of measures | status=SUCCESS | time=0ms
 2019.08.22 23:49:34 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Compute Quality Gate measures | status=SUCCESS | time=1ms
 2019.08.22 23:49:34 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Compute Quality profile measures | status=SUCCESS | time=3ms
 2019.08.22 23:49:34 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Generate Quality profile events | status=SUCCESS | time=2ms
 2019.08.22 23:49:34 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Generate Quality gate events | status=SUCCESS | time=5ms
 2019.08.22 23:49:35 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Persist components | status=SUCCESS | time=381ms
 2019.08.22 23:49:35 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Persist analysis | status=SUCCESS | time=2ms
 2019.08.22 23:49:35 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Persist analysis properties | status=SUCCESS | time=1ms
 2019.08.22 23:49:35 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Persist measures | inserts=45 | status=SUCCESS | time=6ms
 2019.08.22 23:49:36 INFO  web[][o.a.c.h.Http11Processor] Error parsing HTTP request header
  Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
 java.lang.IllegalArgumentException: Invalid character found in the HTTP protocol
 	at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:541)
 	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:684)
 	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
 	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
 	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
 	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628
 	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
 	at java.base/java.lang.Thread.run(Thread.java:834)
 2019.08.22 23:49:42 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Persist live measures | insertsOrUpdates=46696 | status=SUCCESS | time=7289ms
 2019.08.22 23:49:42 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Persist duplication data | insertsOrUpdates=808 | status=SUCCESS | time=378ms
 2019.08.22 23:49:42 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Persist new ad hoc Rules | status=SUCCESS | time=0ms
 2019.08.22 23:49:58 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Persist issues | inserts=20205 | updates=0 | merged=0 | untouched=0 | status=SUCCESS | time=15464ms
 2019.08.22 23:49:58 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Persist project links | status=SUCCESS | time=5ms
 2019.08.22 23:49:58 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Persist events | status=SUCCESS | time=4ms
 2019.08.22 23:50:05 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 137
 2019.08.22 23:50:05 INFO  app[][o.s.a.SchedulerImpl] Process[es] is stopped
 2019.08.22 23:50:05 INFO  ce[][o.s.p.ProcessEntryPoint] Hard stopping process
 2019.08.22 23:50:05 INFO  ce[][o.s.ce.app.CeServer] Compute Engine is stopping...
 2019.08.22 23:50:05 INFO  ce[][o.s.c.t.CeProcessingSchedulerImpl] Hard stopping workers...
 2019.08.22 23:50:06 INFO  ce[][o.s.c.t.CeProcessingSchedulerImpl] Some in-progress tasks are getting killed.
 2019.08.22 23:50:06 WARN  ce[][o.s.p.ProcessEntryPoint$HardStopperThread] Can not stop in 1000ms
 2019.08.22 23:50:09 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Persist sources | status=SUCCESS | time=11147ms
 2019.08.22 23:50:09 ERROR web[AWy7tbVKHgt8qVv2AASZ][o.s.s.w.WebServiceEngine] Fail to process request http://sonar.***.com/api/components/search_projects?ps=50&facets=reliability_rating%2Csecurity_rating%2Csqale_rating%2Ccoverage%2Cduplicated_lines_density%2Cncloc%2Calert_status%2Clanguages%2Ctags&f=analysisDate%2CleakPeriodDate&s=analysisDate&asc=false
 java.lang.IllegalStateException: Fail to execute ES search request 'SearchRequest{searchType=QUERY_THEN_FETCH, indices=[projectmeasures], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[auth], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, source={"from":0,"size":50,"query":{"bool":{"must":[{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"_source":false,"sort":[{"analysedAt":{"order":"desc"}},{"key":{"order":"asc"}}],"aggregations":{"reliability_rating":{"global":{},"aggregations":{"facet_filter_reliability_rating":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"nested_reliability_rating":{"nested":{"path":"measures"},"aggregations":{"filter_reliability_rating":{"filter":{"terms":{"measures.key":["reliability_rating"],"boost":1.0}},"aggregations":{"reliability_rating":{"filters":{"filters":{"1":{"term":{"measures.value":{"value":1.0,"boost":1.0}}},"2":{"term":{"measures.value":{"value":2.0,"boost":1.0}}},"3":{"term":{"measures.value":{"value":3.0,"boost":1.0}}},"4":{"term":{"measures.value":{"value":4.0,"boost":1.0}}},"5":{"term":{"measures.value":{"value":5.0,"boost":1.0}}}},"other_bucket":false,"other_bucket_key":"_other_"}}}}}}}}}},"security_rating":{"global":{},"aggregations":{"facet_filter_security_rating":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"nested_security_rating":{"nested":{"path":"measures"},"aggregations":{"filter_security_rating":{"filter":{"terms":{"measures.key":["security_rating"],"boost":1.0}},"aggregations":{"security_rating":{"filters":{"filters":{"1":{"term":{"measures.value":{"value":1.0,"boost":1.0}}},"2":{"term":{"measures.value":{"value":2.0,"boost":1.0}}},"3":{"term":{"measures.value":{"value":3.0,"boost":1.0}}},"4":{"term":{"measures.value":{"value":4.0,"boost":1.0}}},"5":{"term":{"measures.value":{"value":5.0,"boost":1.0}}}},"other_bucket":false,"other_bucket_key":"_other_"}}}}}}}}}},"sqale_rating":{"global":{},"aggregations":{"facet_filter_sqale_rating":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"nested_sqale_rating":{"nested":{"path":"measures"},"aggregations":{"filter_sqale_rating":{"filter":{"terms":{"measures.key":["sqale_rating"],"boost":1.0}},"aggregations":{"sqale_rating":{"filters":{"filters":{"1":{"term":{"measures.value":{"value":1.0,"boost":1.0}}},"2":{"term":{"measures.value":{"value":2.0,"boost":1.0}}},"3":{"term":{"measures.value":{"value":3.0,"boost":1.0}}},"4":{"term":{"measures.value":{"value":4.0,"boost":1.0}}},"5":{"term":{"measures.value":{"value":5.0,"boost":1.0}}}},"other_bucket":false,"other_bucket_key":"_other_"}}}}}}}}}},"coverage":{"global":{},"aggregations":{"facet_filter_coverage":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"combined_coverage":{"filter":{"match_all":{"boost":1.0}},"aggregations":{"nested_coverage":{"nested":{"path":"measures"},"aggregations":{"filter_coverage":{"filter":{"terms":{"measures.key":["coverage"],"boost":1.0}},"aggregations":{"coverage":{"range":{"field":"measures.value","ranges":[{"to":30.0},{"from":30.0,"to":50.0},{"from":50.0,"to":70.0},{"from":70.0,"to":80.0},{"from":80.0}],"keyed":false}}}}}},"no_data_coverage":{"filter":{"bool":{"must_not":[{"nested":{"query":{"term":{"measures.key":{"value":"coverage","boost":1.0}}},"path":"measures","ignore_unmapped":false,"score_mode":"avg","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}}}}}}}},"duplicated_lines_density":{"global":{},"aggregations":{"facet_filter_duplicated_lines_density":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"combined_duplicated_lines_density":{"filter":{"match_all":{"boost":1.0}},"aggregations":{"nested_duplicated_lines_density":{"nested":{"path":"measures"},"aggregations":{"filter_duplicated_lines_density":{"filter":{"terms":{"measures.key":["duplicated_lines_density"],"boost":1.0}},"aggregations":{"duplicated_lines_density":{"range":{"field":"measures.value","ranges":[{"to":3.0},{"from":3.0,"to":5.0},{"from":5.0,"to":10.0},{"from":10.0,"to":20.0},{"from":20.0}],"keyed":false}}}}}},"no_data_duplicated_lines_density":{"filter":{"bool":{"must_not":[{"nested":{"query":{"term":{"measures.key":{"value":"duplicated_lines_density","boost":1.0}}},"path":"measures","ignore_unmapped":false,"score_mode":"avg","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}}}}}}}},"ncloc":{"global":{},"aggregations":{"facet_filter_ncloc":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"nested_ncloc":{"nested":{"path":"measures"},"aggregations":{"filter_ncloc":{"filter":{"terms":{"measures.key":["ncloc"],"boost":1.0}},"aggregations":{"ncloc":{"range":{"field":"measures.value","ranges":[{"to":1000.0},{"from":1000.0,"to":10000.0},{"from":10000.0,"to":100000.0},{"from":100000.0,"to":500000.0},{"from":500000.0}],"keyed":false}}}}}}}}}},"alert_status":{"global":{},"aggregations":{"facet_filter_alert_status":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"alert_status":{"filters":{"filters":{"ERROR":{"term":{"qualityGateStatus":{"value":3,"boost":1.0}}},"OK":{"term":{"qualityGateStatus":{"value":1,"boost":1.0}}}},"other_bucket":false,"other_bucket_key":"_other_"}}}}}},"languages":{"global":{},"aggregations":{"languages_filter":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"languages":{"terms":{"field":"languages","size":10,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}}}},"tags":{"global":{},"aggregations":{"tags_filter":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"tags":{"terms":{"field":"tags","size":10,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}}}}}}}' on indices '[projectmeasures]' on types '[auth]'
 	at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:44)
 	at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:32)
 	at org.sonar.server.measure.index.ProjectMeasuresIndex.search(ProjectMeasuresIndex.java:183)
 	at org.sonar.server.component.ws.SearchProjectsAction.searchData(SearchProjectsAction.java:254

Regards,
Upen

upen_raj · August 23, 2019, 12:02am

Part(2)

 Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{F0PCcysmT6iFmmrykhx3Dg}{127.0.0.1}{127.0.0.1:9001}]
 	at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352)
 	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248)
 	at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60)
 	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:388)
 	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
 	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:391)
 	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46)
 	at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:42)
 	... 52 common frames omitted
 2019.08.22 23:50:09 ERROR web[AWy7tbVKHgt8qVv2AASh][o.s.s.w.WebServiceEngine] Fail to process request http://sonar.***.com/api/components/search_projects?ps=50&facets=reliability_rating%2Csecurity_rating%2Csqale_rating%2Ccoverage%2Cduplicated_lines_density%2Cncloc%2Calert_status%2Clanguages%2Ctags&f=analysisDate%2CleakPeriodDate&s=analysisDate&asc=false
 java.lang.IllegalStateException: Fail to execute ES search request 'SearchRequest{searchType=QUERY_THEN_FETCH, indices=[projectmeasures], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[auth], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, source={"from":0,"size":50,"query":{"bool":{"must":[{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"_source":false,"sort":[{"analysedAt":{"order":"desc"}},{"key":{"order":"asc"}}],"aggregations":{"reliability_rating":{"global":{},"aggregations":{"facet_filter_reliability_rating":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"nested_reliability_rating":{"nested":{"path":"measures"},"aggregations":{"filter_reliability_rating":{"filter":{"terms":{"measures.key":["reliability_rating"],"boost":1.0}},"aggregations":{"reliability_rating":{"filters":{"filters":{"1":{"term":{"measures.value":{"value":1.0,"boost":1.0}}},"2":{"term":{"measures.value":{"value":2.0,"boost":1.0}}},"3":{"term":{"measures.value":{"value":3.0,"boost":1.0}}},"4":{"term":{"measures.value":{"value":4.0,"boost":1.0}}},"5":{"term":{"measures.value":{"value":5.0,"boost":1.0}}}},"other_bucket":false,"other_bucket_key":"_other_"}}}}}}}}}},"security_rating":{"global":{},"aggregations":{"facet_filter_security_rating":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"nested_security_rating":{"nested":{"path":"measures"},"aggregations":{"filter_security_rating":{"filter":{"terms":{"measures.key":["security_rating"],"boost":1.0}},"aggregations":{"security_rating":{"filters":{"filters":{"1":{"term":{"measures.value":{"value":1.0,"boost":1.0}}},"2":{"term":{"measures.value":{"value":2.0,"boost":1.0}}},"3":{"term":{"measures.value":{"value":3.0,"boost":1.0}}},"4":{"term":{"measures.value":{"value":4.0,"boost":1.0}}},"5":{"term":{"measures.value":{"value":5.0,"boost":1.0}}}},"other_bucket":false,"other_bucket_key":"_other_"}}}}}}}}}},"sqale_rating":{"global":{},"aggregations":{"facet_filter_sqale_rating":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"nested_sqale_rating":{"nested":{"path":"measures"},"aggregations":{"filter_sqale_rating":{"filter":{"terms":{"measures.key":["sqale_rating"],"boost":1.0}},"aggregations":{"sqale_rating":{"filters":{"filters":{"1":{"term":{"measures.value":{"value":1.0,"boost":1.0}}},"2":{"term":{"measures.value":{"value":2.0,"boost":1.0}}},"3":{"term":{"measures.value":{"value":3.0,"boost":1.0}}},"4":{"term":{"measures.value":{"value":4.0,"boost":1.0}}},"5":{"term":{"measures.value":{"value":5.0,"boost":1.0}}}},"other_bucket":false,"other_bucket_key":"_other_"}}}}}}}}}},"coverage":{"global":{},"aggregations":{"facet_filter_coverage":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"combined_coverage":{"filter":{"match_all":{"boost":1.0}},"aggregations":{"nested_coverage":{"nested":{"path":"measures"},"aggregations":{"filter_coverage":{"filter":{"terms":{"measures.key":["coverage"],"boost":1.0}},"aggregations":{"coverage":{"range":{"field":"measures.value","ranges":[{"to":30.0},{"from":30.0,"to":50.0},{"from":50.0,"to":70.0},{"from":70.0,"to":80.0},{"from":80.0}],"keyed":false}}}}}},"no_data_coverage":{"filter":{"bool":{"must_not":[{"nested":{"query":{"term":{"measures.key":{"value":"coverage","boost":1.0}}},"path":"measures","ignore_unmapped":false,"score_mode":"avg","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}}}}}}}},"duplicated_lines_density":{"global":{},"aggregations":{"facet_filter_duplicated_lines_density":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"combined_duplicated_lines_density":{"filter":{"match_all":{"boost":1.0}},"aggregations":{"nested_duplicated_lines_density":{"nested":{"path":"measures"},"aggregations":{"filter_duplicated_lines_density":{"filter":{"terms":{"measures.key":["duplicated_lines_density"],"boost":1.0}},"aggregations":{"duplicated_lines_density":{"range":{"field":"measures.value","ranges":[{"to":3.0},{"from":3.0,"to":5.0},{"from":5.0,"to":10.0},{"from":10.0,"to":20.0},{"from":20.0}],"keyed":false}}}}}},"no_data_duplicated_lines_density":{"filter":{"bool":{"must_not":[{"nested":{"query":{"term":{"measures.key":{"value":"duplicated_lines_density","boost":1.0}}},"path":"measures","ignore_unmapped":false,"score_mode":"avg","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}}}}}}}},"ncloc":{"global":{},"aggregations":{"facet_filter_ncloc":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"nested_ncloc":{"nested":{"path":"measures"},"aggregations":{"filter_ncloc":{"filter":{"terms":{"measures.key":["ncloc"],"boost":1.0}},"aggregations":{"ncloc":{"range":{"field":"measures.value","ranges":[{"to":1000.0},{"from":1000.0,"to":10000.0},{"from":10000.0,"to":100000.0},{"from":100000.0,"to":500000.0},{"from":500000.0}],"keyed":false}}}}}}}}}},"alert_status":{"global":{},"aggregations":{"facet_filter_alert_status":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"alert_status":{"filters":{"filters":{"ERROR":{"term":{"qualityGateStatus":{"value":3,"boost":1.0}}},"OK":{"term":{"qualityGateStatus":{"value":1,"boost":1.0}}}},"other_bucket":false,"other_bucket_key":"_other_"}}}}}},"languages":{"global":{},"aggregations":{"languages_filter":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"languages":{"terms":{"field":"languages","size":10,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}}}},"tags":{"global":{},"aggregations":{"tags_filter":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"tags":{"terms":{"field":"tags","size":10,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}}}}}}}' on indices '[projectmeasures]' on types '[auth]'
 	at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:44)
 	at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:32)
 Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{F0PCcysmT6iFmmrykhx3Dg}{127.0.0.1}{127.0.0.1:9001}]
 	at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352)
 	... 52 common frames omitted
 2019.08.22 23:50:15 ERROR web[AWy7tbVKHgt8qVv2AASn][o.s.s.w.WebServiceEngine] Fail to process request http://sonar.***.com/api/issues/search?s=FILE_LINE&resolved=false&ps=100&facets=severities%2Ctypes&additionalFields=_all
 java.lang.IllegalStateException: Fail to execute ES search request 'SearchRequest{searchType=QUERY_THEN_FETCH, indices=[issues], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[auth], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, source={"from":0,"size":100,"query":{"bool":{"must":[{"match_all":{"boost":1.0}}],"filter":[{"bool":{"must":[{"term":{"isMainBranch":{"value":"true","boost":1.0}}},{"bool":{"must_not":[{"exists":{"field":"resolution","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},{"term":{"indexType":{"value":"issue","boost":1.0}}},{"bool":{"adjust_pure_negative":true,"boost":1.0}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"_source":false,"sort":[{"project":{"order":"asc","missing":"_first"}},{"filePath":{"order":"asc","missing":"_first"}},{"line":{"order":"asc","missing":"_first"}},{"severityValue":{"order":"desc","missing":"_first"}},{"key":{"order":"asc","missing":"_first"}}],"aggregations":{"types":{"global":{},"aggregations":{"types_filter":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"isMainBranch":{"value":"true","boost":1.0}}},{"bool":{"must_not":[{"exists":{"field":"resolution","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},{"term":{"indexType":{"value":"issue","boost":1.0}}},{"bool":{"adjust_pure_negative":true,"boost":1.0}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"types":{"terms":{"field":"type","size":4,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}}}},"severities":{"global":{},"aggregations":{"severities_filter":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"isMainBranch":{"value":"true","boost":1.0}}},{"bool":{"must_not":[{"exists":{"field":"resolution","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},{"term":{"indexType":{"value":"issue","boost":1.0}}},{"bool":{"adjust_pure_negative":true,"boost":1.0}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"must_not":[{"term":{"type":{"value":"SECURITY_HOTSPOT","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"severities":{"terms":{"field":"severity","size":5,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}}}},"effort":{"sum":{"field":"effort"}}}}}' on indices '[issues]' on types '[auth]'
 	at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:44)
 Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{F0PCcysmT6iFmmrykhx3Dg}{127.0.0.1}{127.0.0.1:9001}]
 	at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352)
 	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248)
 	at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60)
 	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:388)
 	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
 	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:391)
 	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46)
 	at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:42)
 	... 50 common frames omitted
 2019.08.22 23:50:16 WARN  ce[][o.s.c.t.CeProcessingSchedulerExecutorServiceImpl] Pool CeProcessingSchedulerExecutorServiceImpl did not terminate
 2019.08.22 23:50:16 INFO  ce[][o.s.ce.app.CeServer] Compute Engine is stopped
 2019.08.22 23:50:16 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.s.ComputationStepExecutor] Persist cross project duplications | status=FAILED | time=1ms
 2019.08.22 23:50:16 ERROR ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Execution of task class org.sonar.ce.task.projectanalysis.webhook.WebhookPostTask failed
 org.apache.ibatis.exceptions.PersistenceException:
 ### Error querying database.  Cause: java.sql.SQLException: Data source is closed
 ### The error may exist in org.sonar.db.component.ComponentMapper
 ### The error may involve org.sonar.db.component.ComponentMapper.selectByUuid
 ### The error occurred while executing a query
 ### Cause: java.sql.SQLException: Data source is closed
 	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
 	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:149)
 Caused by: java.sql.SQLException: Data source is closed
 	at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:2087)
 	at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
 	... 27 common frames omitted
 2019.08.22 23:50:16 ERROR ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.CeWorkerImpl] Failed to finalize task with uuid 'AWy7vBKcCRIR65Zz25ib' and persist its state to db
 org.apache.ibatis.exceptions.PersistenceException:
 ### Error querying database.  Cause: java.sql.SQLException: Data source is closed
 ### The error may exist in org.sonar.db.ce.CeQueueMapper
 ### The error may involve org.sonar.db.ce.CeQueueMapper.selectByUuid
 ### The error occurred while executing a query
 ### Cause: java.sql.SQLException: Data source is closed
 	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
 	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.finalizeTask(CeWorkerImpl.java:242)
 	at java.base/java.lang.Thread.run(Thread.java:834)
 Caused by: java.sql.SQLException: Data source is closed
 	at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:2087)
 	at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
 	... 21 common frames omitted
 2019.08.22 23:50:16 INFO  ce[AWy7vBKcCRIR65Zz25ib][o.s.c.t.CeWorkerImpl] Executed task | project=CRM-UI:CRM-UI | type=REPORT | id=AWy7vBKcCRIR65Zz25ib | status=CANCELED | time=54973ms
 2019.08.22 23:50:16 ERROR ce[][o.s.c.t.CeWorkerImpl] Call to listener ReportAnalysisFailureNotificationExecutionListener failed.
 org.apache.ibatis.exceptions.PersistenceException:
 ### Error getting a new connection.  Cause: java.sql.SQLException: Data source is closed
 ### Cause: java.sql.SQLException: Data source is closed
org.sonar.db.property.PropertiesDao.hasProjectNotificationSubscribersForDispatchers(PropertiesDao.java:101)
 	at java.base/java.lang.Thread.run(Thread.java:834)
 Caused by: java.sql.SQLException: Data source is closed
org.sonar.db.profiling.NullConnectionInterceptor.getConnection(NullConnectionInterceptor.java:31)
 	at org.sonar.db.profiling.ProfiledDataSource.getConnection(ProfiledDataSource.java:317)
 	at org.apache.ibatis.transaction.jdbc.JdbcTransaction.openConnection(JdbcTransaction.java:138)
 	at org.apache.ibatis.transaction.jdbc.JdbcTransaction.getConnection(JdbcTransaction.java:60)
 	at org.apache.ibatis.session.defaults.DefaultSqlSession.getConnection(DefaultSqlSession.java:297)
 	... 22 common frames omitted
 2019.08.22 23:50:16 INFO  app[][o.s.a.SchedulerImpl] Process[ce] is stopped
 2019.08.22 23:50:17 ERROR web[AWy7tbVKHgt8qVv2AASo][o.s.s.w.WebServiceEngine] Fail to process request http://sonar.***.com/api/components/search_projects?ps=50&facets=reliability_rating%2Csecurity_rating%2Csqale_rating%2Ccoverage%2Cduplicated_lines_density%2Cncloc%2Calert_status%2Clanguages%2Ctags&f=analysisDate%2CleakPeriodDate&s=analysisDate&asc=false
 java.lang.IllegalStateException: Fail to execute ES search request 'SearchRequest{searchType=QUERY_THEN_FETCH, indices=[projectmeasures], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[auth], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, source={"from":0,"size":50,"query":{"bool":{"must":[{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"_source":false,"sort":[{"analysedAt":{"order":"desc"}},{"key":{"order":"asc"}}],"aggregations":{"reliability_rating":{"global":{},"aggregations":{"facet_filter_reliability_rating":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"nested_reliability_rating":{"nested":{"path":"measures"},"aggregations":{"filter_reliability_rating":{"filter":{"terms":{"measures.key":["reliability_rating"],"boost":1.0}},"aggregations":{"reliability_rating":{"filters":{"filters":{"1":{"term":{"measures.value":{"value":1.0,"boost":1.0}}},"2":{"term":{"measures.value":{"value":2.0,"boost":1.0}}},"3":{"term":{"measures.value":{"value":3.0,"boost":1.0}}},"4":{"term":{"measures.value":{"value":4.0,"boost":1.0}}},"5":{"term":{"measures.value":{"value":5.0,"boost":1.0}}}},"other_bucket":false,"other_bucket_key":"_other_"}}}}}}}}}},"security_rating":{"global":{},"aggregations":{"facet_filter_security_rating":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"nested_security_rating":{"nested":{"path":"measures"},"aggregations":{"filter_security_rating":{"filter":{"terms":{"measures.key":["security_rating"],"boost":1.0}},"aggregations":{"security_rating":{"filters":{"filters":{"1":{"term":{"measures.value":{"value":1.0,"boost":1.0}}},"2":{"term":{"measures.value":{"value":2.0,"boost":1.0}}},"3":{"term":{"measures.value":{"value":3.0,"boost":1.0}}},"4":{"term":{"measures.value":{"value":4.0,"boost":1.0}}},"5":{"term":{"measures.value":{"value":5.0,"boost":1.0}}}},"other_bucket":false,"other_bucket_key":"_other_"}}}}}}}}}},"sqale_rating":{"global":{},"aggregations":{"facet_filter_sqale_rating":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"nested_sqale_rating":{"nested":{"path":"measures"},"aggregations":{"filter_sqale_rating":{"filter":{"terms":{"measures.key":["sqale_rating"],"boost":1.0}},"aggregations":{"sqale_rating":{"filters":{"filters":{"1":{"term":{"measures.value":{"value":1.0,"boost":1.0}}},"2":{"term":{"measures.value":{"value":2.0,"boost":1.0}}},"3":{"term":{"measures.value":{"value":3.0,"boost":1.0}}},"4":{"term":{"measures.value":{"value":4.0,"boost":1.0}}},"5":{"term":{"measures.value":{"value":5.0,"boost":1.0}}}},"other_bucket":false,"other_bucket_key":"_other_"}}}}}}}}}},"coverage":{"global":{},"aggregations":{"facet_filter_coverage":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"combined_coverage":{"filter":{"match_all":{"boost":1.0}},"aggregations":{"nested_coverage":{"nested":{"path":"measures"},"aggregations":{"filter_coverage":{"filter":{"terms":{"measures.key":["coverage"],"boost":1.0}},"aggregations":{"coverage":{"range":{"field":"measures.value","ranges":[{"to":30.0},{"from":30.0,"to":50.0},{"from":50.0,"to":70.0},{"from":70.0,"to":80.0},{"from":80.0}],"keyed":false}}}}}},"no_data_coverage":{"filter":{"bool":{"must_not":[{"nested":{"query":{"term":{"measures.key":{"value":"coverage","boost":1.0}}},"path":"measures","ignore_unmapped":false,"score_mode":"avg","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}}}}}}}},"duplicated_lines_density":{"global":{},"aggregations":{"facet_filter_duplicated_lines_density":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"combined_duplicated_lines_density":{"filter":{"match_all":{"boost":1.0}},"aggregations":{"nested_duplicated_lines_density":{"nested":{"path":"measures"},"aggregations":{"filter_duplicated_lines_density":{"filter":{"terms":{"measures.key":["duplicated_lines_density"],"boost":1.0}},"aggregations":{"duplicated_lines_density":{"range":{"field":"measures.value","ranges":[{"to":3.0},{"from":3.0,"to":5.0},{"from":5.0,"to":10.0},{"from":10.0,"to":20.0},{"from":20.0}],"keyed":false}}}}}},"no_data_duplicated_lines_density":{"filter":{"bool":{"must_not":[{"nested":{"query":{"term":{"measures.key":{"value":"duplicated_lines_density","boost":1.0}}},"path":"measures","ignore_unmapped":false,"score_mode":"avg","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}}}}}}}},"ncloc":{"global":{},"aggregations":{"facet_filter_ncloc":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"nested_ncloc":{"nested":{"path":"measures"},"aggregations":{"filter_ncloc":{"filter":{"terms":{"measures.key":["ncloc"],"boost":1.0}},"aggregations":{"ncloc":{"range":{"field":"measures.value","ranges":[{"to":1000.0},{"from":1000.0,"to":10000.0},{"from":10000.0,"to":100000.0},{"from":100000.0,"to":500000.0},{"from":500000.0}],"keyed":false}}}}}}}}}},"alert_status":{"global":{},"aggregations":{"facet_filter_alert_status":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"alert_status":{"filters":{"filters":{"ERROR":{"term":{"qualityGateStatus":{"value":3,"boost":1.0}}},"OK":{"term":{"qualityGateStatus":{"value":1,"boost":1.0}}}},"other_bucket":false,"other_bucket_key":"_other_"}}}}}},"languages":{"global":{},"aggregations":{"languages_filter":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"languages":{"terms":{"field":"languages","size":10,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}}}},"tags":{"global":{},"aggregations":{"tags_filter":{"filter":{"bool":{"must":[{"match_all":{"boost":1.0}},{"term":{"indexType":{"value":"projectmeasure","boost":1.0}}},{"has_parent":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"auth_allowAnyone":{"value":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"parent_type":"auth","score":false,"ignore_unmapped":false,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"tags":{"terms":{"field":"tags","size":10,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}}}}}}}' on indices '[projectmeasures]' on types '[auth]'
 	at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:44)
 	
 	at java.base/java.lang.Thread.run(Thread.java:834)
 Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{F0PCcysmT6iFmmrykhx3Dg}{127.0.0.1}{127.0.0.1:9001}]
 	at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352)
 	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248)

 2019.08.22 23:50:17 INFO  web[][o.s.p.ProcessEntryPoint] Hard stopping process
 2019.08.22 23:50:17 INFO  web[][o.s.s.n.NotificationDaemon] Notification service stopped
 2019.08.22 23:50:17 INFO  app[][o.s.a.SchedulerImpl] Process[web] is stopped
 2019.08.22 23:50:17 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped

ganncamp · August 23, 2019, 12:03pm

Hi,

You’ve got two different things going on here. One is problems with your ES. The other is problems submitting analyses of large projects. For the latter, you still need to look to your load balancer. You should try bypassing the load balancer and submitting those large reports directly to SonarQube. If they succeed that will confirm that the problem you need to diagnose and fix is in your load balancer. And that’s out of the scope of this community.

For the first problem you should really start a new thread.

Ann

ganncamp · August 23, 2019, 12:06pm

Hi again,

BTW, I appreciate that you’ve made an attempt to format your logs. However, you’ve used quote formatting rather than code formatting (add ``` on the line before and on the line after) and quote formatting is no help at all.

I’ve reformatted your logs (again). It would be nice if you’d handle this part in the future.

Ann

upen_raj · August 23, 2019, 5:39pm

Hi Ann,

I got the issue resolved by increasing the memory available for the Kubenetes pod. Thanks so much for your time. I will take care of the code formatting next time. Sorry for the miss.

Regards,
Upen