How to deal with hanging tasks?

We are running 8.3.1 community edition. And we regularly see a task running for a really long time. Most of the time it runs in a matter of seconds. But occasionally one will get stuck running for many many minutes.

Is there a way to kill a long running task?
And how do we diagnose what’s going on?

I got to the logs, and it looks like at some point we start getting timeouts getting to the PostgreSQL server.

It does appear that this has happened a few times. Rebooting the machine always clears it.

Are there any settings we should be looking at to make our connection postgres more reliable?

The error we are seeing is always this:


2020.06.25 03:41:52 ERROR web[AXLpNnurS9Cr3slMAAKC][o.s.s.w.WebServiceEngine] Fail to process request http://sonarqube.bf-app-services-fra02.np.wce.ibm.com/api/settings/values.protobuf
org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
### The error may exist in org.sonar.db.property.InternalPropertiesMapper
### The error may involve org.sonar.db.property.InternalPropertiesMapper.selectAsText-Inline
### The error occurred while setting parameters
### SQL: select       kee as "key",       is_empty as empty,       text_value as value,       created_at as createdAt     from       internal_properties     where       kee in  (  ? )
### Cause: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.

Hi,

This SO answer might help.

 
Ann

We also see frequent hangs in background tasks - which is a newish problem. We use 8.3.1, it started in the 8.x series, I think.

I have not seen any ERROR output, it just hangs. Until we notice and restart the service.

We run it in a docker container with a companion PostgreSQL in its own container.
So I am not sure the suggested SO is relevant (for our problem, anyway).

I enabled trace output before the last hang, if it is something you can use for anything?
I will be happy to spend time testing workaround/fixes.

Hi @jskovjyskebankdk,

We’ve had other reports that you can hit this problem when you run out of file space. I’d be interested to see what errors if any you have in ce.log when this happens.

 
Ann

No errors in ce.log. And 63GB free disk space.

Below the fairly (to my eyes) harmless output at the last hung session.

Can I provide you with some thread/heap dumps the next time it happens?
It see it pretty much daily, or at least weekly. Random projects, it seems.

2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][sql] time=0ms | sql=select i.id, i.kee as kee, i.rule_id as ruleId, i.severity as severity, i.manual_severity as manualSeverity, i.message as message, i.line as line, i.locations as locations, i.gap as gap, i.effort as effort, i.status as status, i.resolution as resolution, i.checksum as checksum, i.assignee as assigneeUuid, i.author_login as authorLogin, i.tags as tagsString, i.issue_attributes as issueAttributes, i.issue_creation_date as issueCreationTime, i.issue_update_date as issueUpdateTime, i.issue_close_date as issueCloseTime, i.created_at as createdAt, i.updated_at as updatedAt, r.is_external as "isExternal", r.plugin_rule_key as ruleKey, r.plugin_name as ruleRepo, r.language as language, p.kee as componentKey, i.component_uuid as componentUuid, p.module_uuid as moduleUuid, p.module_uuid_path as moduleUuidPath, p.path as filePath, root.kee as projectKey, i.project_uuid as projectUuid, i.issue_type as type from issues i inner join rules r on r.id=i.rule_id inner join components p on p.uuid=i.component_uuid inner join components root on root.uuid=i.project_uuid where i.component_uuid = ? and i.status <> 'CLOSED' | params=AXDJDRYMfQL6chJP28Op
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.i.IntegrateIssuesVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/KundeStamKndJBDW.java (done) | time=2ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.MaintainabilityMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/KundeStamKndJBDW.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.MaintainabilityMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/KundeStamKndJBDW.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.NewMaintainabilityMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/KundeStamKndJBDW.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.NewMaintainabilityMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/KundeStamKndJBDW.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.ReliabilityAndSecurityRatingMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/KundeStamKndJBDW.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.ReliabilityAndSecurityRatingMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/KundeStamKndJBDW.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.NewReliabilityAndSecurityRatingMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/KundeStamKndJBDW.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.NewReliabilityAndSecurityRatingMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/KundeStamKndJBDW.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.SecurityReviewMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/KundeStamKndJBDW.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.SecurityReviewMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/KundeStamKndJBDW.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.NewSecurityReviewMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/KundeStamKndJBDW.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.NewSecurityReviewMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/KundeStamKndJBDW.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.s.LastCommitVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/KundeStamKndJBDW.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.s.LastCommitVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/KundeStamKndJBDW.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.m.MeasureComputersVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/KundeStamKndJBDW.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.m.MeasureComputersVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/KundeStamKndJBDW.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.i.IntegrateIssuesVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][sql] time=1ms | sql=select i.id, i.kee as kee, i.rule_id as ruleId, i.severity as severity, i.manual_severity as manualSeverity, i.message as message, i.line as line, i.locations as locations, i.gap as gap, i.effort as effort, i.status as status, i.resolution as resolution, i.checksum as checksum, i.assignee as assigneeUuid, i.author_login as authorLogin, i.tags as tagsString, i.issue_attributes as issueAttributes, i.issue_creation_date as issueCreationTime, i.issue_update_date as issueUpdateTime, i.issue_close_date as issueCloseTime, i.created_at as createdAt, i.updated_at as updatedAt, r.is_external as "isExternal", r.plugin_rule_key as ruleKey, r.plugin_name as ruleRepo, r.language as language, p.kee as componentKey, i.component_uuid as componentUuid, p.module_uuid as moduleUuid, p.module_uuid_path as moduleUuidPath, p.path as filePath, root.kee as projectKey, i.project_uuid as projectUuid, i.issue_type as type from issues i inner join rules r on r.id=i.rule_id inner join components p on p.uuid=i.component_uuid inner join components root on root.uuid=i.project_uuid where i.component_uuid = ? and i.status <> 'CLOSED' | params=AXDJDRYMfQL6chJP28Oq
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.i.IntegrateIssuesVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java (done) | time=2ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.MaintainabilityMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.MaintainabilityMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.NewMaintainabilityMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.NewMaintainabilityMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.ReliabilityAndSecurityRatingMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.ReliabilityAndSecurityRatingMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.NewReliabilityAndSecurityRatingMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.NewReliabilityAndSecurityRatingMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.SecurityReviewMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.SecurityReviewMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.NewSecurityReviewMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.NewSecurityReviewMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.s.LastCommitVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.s.LastCommitVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.m.MeasureComputersVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.m.MeasureComputersVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/AccountJess.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.i.IntegrateIssuesVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/DWAccountId.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][sql] time=1ms | sql=select i.id, i.kee as kee, i.rule_id as ruleId, i.severity as severity, i.manual_severity as manualSeverity, i.message as message, i.line as line, i.locations as locations, i.gap as gap, i.effort as effort, i.status as status, i.resolution as resolution, i.checksum as checksum, i.assignee as assigneeUuid, i.author_login as authorLogin, i.tags as tagsString, i.issue_attributes as issueAttributes, i.issue_creation_date as issueCreationTime, i.issue_update_date as issueUpdateTime, i.issue_close_date as issueCloseTime, i.created_at as createdAt, i.updated_at as updatedAt, r.is_external as "isExternal", r.plugin_rule_key as ruleKey, r.plugin_name as ruleRepo, r.language as language, p.kee as componentKey, i.component_uuid as componentUuid, p.module_uuid as moduleUuid, p.module_uuid_path as moduleUuidPath, p.path as filePath, root.kee as projectKey, i.project_uuid as projectUuid, i.issue_type as type from issues i inner join rules r on r.id=i.rule_id inner join components p on p.uuid=i.component_uuid inner join components root on root.uuid=i.project_uuid where i.component_uuid = ? and i.status <> 'CLOSED' | params=AXDJDRYMfQL6chJP28Os
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.i.IntegrateIssuesVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/DWAccountId.java (done) | time=2ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.MaintainabilityMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/DWAccountId.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.MaintainabilityMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/DWAccountId.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.NewMaintainabilityMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/DWAccountId.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.NewMaintainabilityMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/DWAccountId.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.ReliabilityAndSecurityRatingMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/DWAccountId.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.ReliabilityAndSecurityRatingMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/DWAccountId.java (done) | time=0ms
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.NewReliabilityAndSecurityRatingMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/DWAccountId.java
2020.06.29 13:21:09 TRACE ce[AXL_LrSMoECOuylqj62b][o.s.c.t.p.q.NewReliabilityAndSecurityRatingMeasuresVisitor] Visiting component jb.it.jrm:jb.it.jrm.ejb/src/main/java/dk/jyskebank/jrm/domain/helper/DWAccountId.java (done) | time=0ms
2020.06.29 13:22:44 TRACE ce[][o.s.c.c.CeCleaningSchedulerImpl] Resetting state of tasks with unknown worker UUIDs
2020.06.29 13:31:11 TRACE ce[][o.p.c.QueryExecutorBase]  FE=> Terminate

A thread dump would really help.

@dmeneses OK, will do next time I see a hang. Cheers!